CN115311400A

CN115311400A - Social interaction method, device and equipment

Info

Publication number: CN115311400A
Application number: CN202110497342.9A
Authority: CN
Inventors: 王鹤; 蒋翀宁; 陈波; 刘晨光; 朱欣欣; 潘红; 刘镇伟; 周奇; 何芬; 赵艺; 邬振海; 邢雪源; 佘渡离; 方睿; 程晓龙; 黄学勤
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-05-07
Filing date: 2021-05-07
Publication date: 2022-11-08

Abstract

The embodiment of the application provides a social interaction method, a social interaction device and social interaction equipment, wherein the social interaction method comprises the following steps: acquiring an environment image, and acquiring normal vector information corresponding to each target point on the environment image; determining a connected image area on the environment image according to the normal vector information of each target point; determining a target reference area in the connected image area; and rendering the virtual object according to the target position of the target reference area in the environment image so as to display the virtual object on the environment image. By adopting the method and the device, the image area can be automatically identified in the environment image to display the virtual object.

Description

Social interaction method, device and equipment

Technical Field

The present application relates to the field of computer technologies, and in particular, to a social interaction method, apparatus, device, and medium.

Background

Augmented Reality (AR) technology is also called Augmented Reality technology, and is a technology for skillfully fusing virtual information with the real world. At present, the AR technology is widely applied, can be applied to various production and life, for example, can be applied to social scenes, and can also be applied to multimedia application scenes.

In a social scene, a social application program becomes an indispensable social application by virtue of the characteristic that the social application program supports instant communication or asynchronous communication. In the process of using the social application program, messages to be viewed are often received, and the messages can be virtual article messages besides communication messages such as voice and characters sent among users; for example, gifts to be viewed (e.g., birthday gifts, holiday gifts, etc.) sent by other users are received while using the social application. These virtual good messages may also be presented to the user by way of the AR.

In multimedia application scenarios, such as video recording scenarios, teleconferencing scenarios, etc., there is also a need to display some virtual objects in the environment.

At present, in various related scenes in which virtual objects need to be displayed through an AR technology, the display mode of the virtual objects is not intelligent enough, the display positions of the virtual objects are generally specified positions or fixed at certain positions in a shot picture, that is, the display of the virtual objects needs to be configured in advance, and the automation requirements cannot be met.

Disclosure of Invention

The embodiment of the application provides a social interaction method, a social interaction device, social interaction equipment and social interaction media, which can automatically identify an image area in an environment image to display a virtual object.

In one aspect, an embodiment of the present application provides a social interaction method, where the method includes:

acquiring an environment image, and acquiring normal vector information corresponding to each target point on the environment image;

determining a connected image area on the environment image according to the normal vector information of each target point, wherein the direction corresponding to the normal vector information corresponding to each target point in the connected image area and the target direction meet the consistency condition;

determining a target reference area in the connected image area;

and rendering the virtual object according to the target position of the target reference area in the environment image so as to display the virtual object on the environment image.

On the other hand, the embodiment of the present application provides a social interaction method, including:

displaying a social interaction interface, wherein the social interaction interface comprises a prompt element;

responding to the trigger operation of the prompt element, and displaying the acquired environment image on an image display interface;

displaying a virtual object at a target location of an environment image;

the target position corresponds to a target reference area on the environment image, and a target image area included in the target reference area corresponds to a planar object in the current environment.

In another aspect, an embodiment of the present application provides a social interaction device, including:

the acquisition unit is used for acquiring an environment image and acquiring normal vector information corresponding to each target point on the environment image;

the processing unit is used for determining a connected image area on the environment image according to the normal vector information of each target point, and the direction corresponding to the normal vector information corresponding to each target point in the connected image area and the target direction meet the consistency condition;

the processing unit is also used for determining a target reference area in the connected image area;

and the processing unit is also used for rendering the virtual object according to the target position of the target reference area in the environment image so as to display the virtual object on the environment image.

the display unit is used for displaying a social interaction interface, and the social interaction interface comprises prompt elements;

the processing unit is used for responding to the triggering operation of the prompt element and displaying the acquired environment image on the image display interface;

a processing unit further for displaying a virtual object at a target position of the environment image;

In another aspect, an embodiment of the present application provides an electronic device, which includes:

a storage device for storing a computer application;

and the processor executes the computer application program stored in the storage device to realize the social interaction method.

In another aspect, an embodiment of the present application provides a computer-readable storage medium, where a computer application program is stored, and when the computer application program is executed, the social interaction method described above is implemented.

In another aspect, embodiments of the present application provide a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the social interaction method.

In the embodiment of the present application, whether the environment image includes the target image area or not can be analyzed, and when the environment image includes the target image area, the virtual object can be displayed at the target position of the target image area on the environment image. The virtual object is combined with the AR technology, so that the process of superposing the entity object corresponding to the virtual object on the platform object in the real environment can be simulated, the virtual object can be displayed in a target image area which is contained in the environment image and used for bearing the virtual reality object, the reality sense and the intelligence of the virtual object displayed in the environment image are improved, the display mode of the virtual object is enriched, the interactivity between a target user and the virtual object is improved, and the visual immersion sense of the user is enhanced.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

FIG. 1a is a schematic diagram illustrating an overlay of virtual information with a real environment provided by an exemplary embodiment of the present application;

FIG. 1b is a schematic diagram illustrating another virtual information overlay with a real environment provided by an exemplary embodiment of the present application;

FIG. 2 is a block diagram illustrating an architecture of a social interaction system, according to an exemplary embodiment of the present application;

FIG. 3 illustrates a flow chart of a method of social interaction provided by an exemplary embodiment of the present application;

FIG. 4a is a diagram illustrating a social session interface provided by an exemplary embodiment of the present application;

FIG. 4b illustrates a schematic diagram of another social session interface provided by an exemplary embodiment of the present application;

FIG. 5a is a schematic diagram illustrating an image display interface provided by an exemplary embodiment of the present application;

FIG. 5b illustrates a schematic diagram of an alignment microphone blowing and extinguishing a candle provided by an exemplary embodiment of the present application;

FIG. 6 is a flowchart illustrating a process of updating a display of a virtual object in an environmental image according to an exemplary embodiment of the present application;

FIG. 7 is a diagram illustrating an output of a second hint message provided by an exemplary embodiment of the present application;

fig. 8a is a schematic diagram illustrating a process of sharing multimedia data obtained through screen recording according to an exemplary embodiment of the present application;

fig. 8b is a schematic diagram illustrating another sharing process performed on multimedia data obtained by a screen recording process according to an exemplary embodiment of the present application;

fig. 8c is a diagram illustrating multimedia data obtained by downloading a screen recording process according to an exemplary embodiment of the present application;

FIG. 9a is a diagram illustrating a complimentary virtual object provided by an exemplary embodiment of the present application;

FIG. 9b is a schematic diagram illustrating a candidate object interface provided by an exemplary embodiment of the present application;

FIG. 10 is a flow chart illustrating a method of social interaction provided by an exemplary embodiment of the present application;

FIG. 11 illustrates a schematic diagram of a connected image region provided by an exemplary embodiment of the present application;

FIG. 12 is a flowchart illustrating a process of invoking an image rendering process to render a virtual object according to an exemplary embodiment of the present application;

FIG. 13 illustrates a schematic diagram of a continuous AR effect provided by an exemplary embodiment of the present application;

FIG. 14a is a schematic flow chart illustrating a process for generating a target virtual object according to an exemplary embodiment of the present application;

FIG. 14b is a schematic diagram illustrating a process for viewing a target virtual object according to an exemplary embodiment of the present application;

FIG. 15 is a diagram illustrating an exemplary social interaction device, according to an embodiment of the present disclosure;

FIG. 16 is a schematic diagram illustrating a social interaction device according to an exemplary embodiment of the present disclosure;

fig. 17 shows a schematic structural diagram of an electronic device according to an exemplary embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments of the present application, and not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.

Some terms and concepts related to the embodiments of the present application are described below:

(1) The embodiment of the application relates to an AR technology, an augmented reality technology is also called as an augmented reality technology, the technology is a technology for skillfully fusing virtual information and a real world, various technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, sensing and the like are widely applied, and virtual information generated by a computer is applied to the real world after analog simulation, so that the virtual information and real information of the real world are mutually supplemented, and the real world is enhanced. Specifically, the AR technology may perform analog simulation processing on the virtual information, and superimpose the virtual information after the simulation processing into a real environment in the real world, and after the virtual information and the real environment are superimposed, the virtual information and the real environment can exist in the same environmental image (or environmental picture, space) at the same time; the process of superimposing the real environment and the virtual information can be perceived by human senses, thereby helping users to achieve a sense experience beyond reality. The virtual information can be classified according to different modes; for example, the virtual information may be classified into: entity information (i.e., information that exists in the real world) and fictional information (i.e., fictional information that does not exist in the real world); for another example, classifying the virtual information according to different categories may include: text, image, three-dimensional model, music, video, etc. For convenience of illustration, the following descriptions of the virtual information are all introduced by taking a three-dimensional model as an example; the three-dimensional model can be referred to as a 3D (three-dimensional) model for short, and is a model with three-dimensional data constructed in a virtual three-dimensional space; the 3D model can provide a real three-dimensional sense for people; the three-dimensional model is a three-dimensional model which can be viewed at 360 degrees compared with a two-dimensional model, and the process of viewing a solid object corresponding to a 3D model in a real environment can be simulated by viewing the 3D model in a computer from multiple angles.

It should be noted that, the superposition of the real environment and the virtual information may refer to: superimposing the virtual information on an environment image shot for the real environment; the overlay does not mean that the virtual information completely covers the environment image obtained by shooting the real environment, or that the environment image completely covers the virtual information, but means that the virtual information is overlaid into a target image area which is contained in the environment image and is matched with the virtual information, the target image area corresponds to a platform object in the real environment, the platform object in the real environment can be used for bearing the virtual information or is called as a virtual object, and the virtual object can be a virtual cake, a virtual gift, a virtual car which needs to be displayed in the video recording process, and the like.

An exemplary schematic diagram of superimposing virtual information onto an environment image corresponding to a real environment may be shown in fig. 1a, as shown in fig. 1a, if a real environment includes an open space where trees are planted and the virtual information is a virtual object-vehicle, a vehicle model with three-dimensional data may be first constructed according to the virtual information, and then the vehicle model may be superimposed onto an environment image captured of the real environment, so as to obtain an image with both the real environment and the virtual information, which may satisfy a user's requirements for video recording and the like of an enhanced display scene. In this implementation, since the target image area suitable for carrying the virtual information in the environment image includes an image area occupied by an empty space, the superposition between the real environment and the virtual information may refer to: the car (i.e., the virtual information) is superimposed on the open space, and the car may cover a partial area of the open space in the real environment instead of being displayed on the tree. Another exemplary schematic diagram for superimposing virtual information on an environment image corresponding to a real environment can be seen in fig. 1b, as shown in fig. 1b, assuming that the real environment includes a table on which no object is placed, and the virtual information includes a virtual cake, a cake model with three-dimensional data can be constructed according to the virtual information, and then the cake model is superimposed on the real environment, so as to obtain an image in which the real environment and the virtual information exist simultaneously. In this implementation, since the target image area suitable for bearing the virtual information in the environment image includes an image area occupied by the desktop, the superposition between the real environment and the virtual information may refer to: the virtual cake is overlaid on the desktop, and at the moment, the virtual cake covers partial area of the desktop. In summary, the AR technology can superimpose the virtual information onto the environment image corresponding to the real environment, thereby improving the reality of the virtual information displayed in the environment image.

(2) Embodiments of the present application also relate to applications, which may refer to computer programs designed to perform one or more specific tasks. Classifying the application programs according to different dimensions (such as the running mode, the functions and the like of the application programs), and obtaining the types of the same application program under different dimensions, wherein: (1) classified by the way applications run, applications may include, but are not limited to: a client installed in a terminal, an applet that can be used without download installation, a web application opened through a browser, and the like. (2) Classified by the functional type of the application, the application may include, but is not limited to: IM (Instant Messaging) applications, content interaction applications, and the like; the instant messaging application refers to an application based on internet for instant messaging and social interaction, and the instant messaging application may include but is not limited to: QQ, weChat, business WeChat, map application containing social interaction functionality, gaming application, and the like. The content interaction application is an application capable of realizing content interaction, and may be, for example, an application for investment financing, internet banking, micro-blogging, personal space, news, and the like. It should be noted that the application program mentioned later in this embodiment of the present application may be any one of a plurality of application programs classified according to an operation manner, and may also be any one of a plurality of application programs classified according to a function type; the embodiment of the present application does not limit the type of the application program.

It has been found through practice that in the process of using a social application (such as QQ, wechat, microblog in a small red book, etc.), a target user (i.e., any user using the social application in the current terminal) tends to receive a virtual object (i.e., the aforementioned virtual information, such as the 3D model) sent by another user (i.e., one or more users using the social application in another terminal). At present, the mode of opening the virtual objects sent by other users is single, and only the stage of viewing the virtual objects is stopped, the virtual object display mode is simple, the display effect of the virtual objects is not connected with the real environment, and the frequency of viewing the virtual objects is reduced. Based on this, the embodiment of the present application provides a social interaction scheme, where a virtual object may be superimposed and displayed in an environment image corresponding to a real environment through an AR technology, in other words, virtual information is combined with the AR technology, and a process of superimposing an entity object corresponding to the virtual object on a platform object in the real environment may be simulated; the mode of combining the virtual object with the AR technology enables the virtual object to be displayed at a target position contained in the environment image and used for bearing the virtual reality object, improves the intelligence and the reality of virtual information displayed in the environment image, enriches the display mode of the virtual object, and improves the interactivity between the target user and the virtual object. Of course, the present application also relates to other applications, such as a video application related to video shooting, a video conference application related to video conference, and so on, and in these scenes where virtual object rendering is performed based on AR technology, part or all of the contents in the solution described in the present application can be utilized as required.

The social interaction scheme mentioned above may be performed by an electronic device (or social interaction device). Electronic devices herein may include, but are not limited to: terminal devices such as smartphones, tablets, laptops, desktops, smartphones, tablets, notebooks, desktops, smartphones, smartwatches, etc.; or, the server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms, where the server may be a node server on a block chain. The terminal device and the service device may be directly or indirectly connected through a wired or wireless communication manner, which is not limited herein.

The following provides an exemplary description of a social interaction scenario of a social interaction scheme according to an embodiment of the present application with reference to fig. 2. Referring to fig. 2, fig. 2 is a schematic diagram illustrating an architecture of a social interaction system according to an exemplary embodiment of the present application; it is assumed that the electronic device 201 is a terminal device such as a smartphone, tablet, laptop, desktop, etc., and the electronic device 202 is a service device such as a data processing server, web server, application server, etc. In this case, the electronic device 201 may refer to a terminal device for photographing a real environment to obtain an environment image and for displaying an image in which a virtual object is superimposed on the environment image, and the electronic device 202 may refer to a service device for providing application services and technical support for the electronic device 201. The social interaction scheme mentioned in the embodiment of the present application may be executed by the electronic device 201 or the electronic device 202 alone, or may be executed by both the electronic device 201 and the electronic device 202.

In one implementation, the social interaction scheme may be performed by both electronic device 201 and electronic device 202. In a specific implementation, when the target user receives a virtual object sent by another user, a prompt element may be displayed in the social interaction interface of the electronic device 201; if the prompt element is triggered, a camera of the electronic device 201 is called to collect an environment image, and the environment image is sent to the electronic device 202; after receiving the environment image sent by the electronic device 201, the electronic device 202 obtains normal vector information corresponding to each target point on the environment image, where each target point may refer to each pixel point on the environment image, determines a connected image area on the environment image according to the normal vector information of each target point, and determines a target reference area according to the connected image area; rendering the virtual object according to the target position of the target reference area in the environment image; the electronic device 202 transmits the rendered relevant information of the virtual object (such as display position information in the environment image, parameter information of the virtual object, and the like) to the electronic device 201, so that the electronic device 201 displays the virtual object on the environment image according to the rendered relevant information of the virtual object.

In other implementations, the social interaction scheme may also be performed by the electronic device 201 alone, in particular implementations. When the target user receives the virtual object sent by the other user, a prompt element may be displayed in the social interaction interface of the electronic device 201; if the prompt element is triggered, the electronic device 201 executes the following steps: a camera is called to collect an environment image, normal vector information corresponding to each target point on the environment image is obtained, a connected image area is determined on the environment image according to the normal vector information of each target point, and a target reference area is determined according to the connected image area; rendering the virtual object according to the target position of the target reference area in the environment image; such that the electronic device 201 displays the virtual object on the ambient image.

In summary, the embodiment of the present application can analyze whether the environment image includes the target image region, and when the environment image includes the target image region, display the virtual object at the target position of the target image region on the environment image. The virtual object is combined with the AR technology, so that the process of superposing the entity object corresponding to the virtual object on the platform object in the real environment can be simulated, the virtual object can be displayed in a target image area which is contained in the environment image and used for bearing the virtual reality object, the sense of reality of the virtual object displayed in the environment image is improved, the display mode of the virtual object is enriched, the interactivity between a target user and the virtual object is improved, and the visual immersion of the user is enhanced.

Based on the above described social interaction scheme, the embodiment of the present application provides a more detailed social interaction method, and the social interaction method provided by the embodiment of the present application will be described below with reference to the accompanying drawings.

Referring to fig. 3, fig. 3 is a flow chart illustrating a social interaction method according to an exemplary embodiment of the present application; the social interaction method may be performed by an electronic device (e.g., a terminal device) in the system shown in fig. 2. As shown in FIG. 3, the social interaction method includes, but is not limited to, steps S301-S303.

S301: and displaying a social interaction interface, wherein the social interaction interface comprises a prompt element. The social interaction interface may include, but is not limited to, the following types of interfaces:

the social interaction interface may refer to a social conversation interface between the target user and the other users, such that the prompt element included in the social conversation interface may refer to a conversation message between the target user and the other users, and the conversation message may include one or more of an ARK message (i.e., a structured message similar to a card), a text message, a voice message, and an animation message. An exemplary social session interface may be seen in fig. 4a, as shown in fig. 4a, a social session page 401 is displayed in a terminal screen of the terminal device, and the social session interface 401 is a session interface between the target user and the user a, so that the target user and the user a may implement a social session in the social session interface 401; a prompt element 4011 is included in the social session interface 401, where the prompt element 4011 takes the form of an ARK message, and the prompt element 4011 is associated with a virtual object that user a gives to the target user, in which case user a may be referred to as the donor.

(2) The social interaction interface can refer to any interaction interface provided by the social application program, and the any interaction interface comprises prompt elements corresponding to one or more virtual objects related to the target user. The interface entry of any interactive interface can be arranged in a service interface (such as a personal center interface of a target user, a message list interface of the target user and the like) provided by the social application program, and the any interactive interface can be displayed in a terminal screen by triggering the interface entry in the service interface. Referring to fig. 4b, taking an interface entry disposed in a personal center interface of a target user as an example, as shown in fig. 4b, the personal center interface 402 of the target user includes an interface entry 4021 (e.g., a gift entry), when the interface entry 4021 is triggered, an object service interface 403 is output in a terminal screen, and prompt elements corresponding to all or part of virtual objects related to the target user are displayed in the object service interface 403, such as a prompt element 4031, a prompt element 4032, a prompt element 4033, … …, and the like; in addition, the user identifiers (such as user names, user nicknames, user IDs, and the like) of the donors of the respective prompt elements may also be displayed in the object service interface 403, and specifically, the user identifiers of the donors of the prompt elements may be displayed at the display positions of the prompt elements, so that the target user can see the donor corresponding to each prompt element at a glance.

It should be noted that, because the length of the terminal screen of the terminal device is limited, all the prompt elements related to the target user may not be displayed in the object service interface 403 at one time, and based on this, a sliding shaft is further included in the object service interface 403, and the hidden prompt elements can be displayed in the object service interface 403 by sliding the sliding shaft. Of course, in addition to using the sliding shaft to slide the displayed prompt element, a pull-down icon "" -when the icon "" -is pressed and pulled downward, the hidden prompt element may be displayed in the object service interface 403; or, directly executing a sliding operation in the object service interface 403 to realize sliding display of the prompt element in the object service interface 403; and so on. The embodiment of the present application is described by taking the interface style shown in fig. 4b as an example, and the interface style of the service interface related to the embodiment of the present application is not limited, and is described herein.

S302: and responding to the triggering operation of the prompt element, and displaying the acquired environment image on the image display interface.

S303: the virtual object is displayed at a target position of the environment image.

In steps S302-S303, when a prompt element in the social interaction interface is triggered, indicating that the target user wants to view a virtual object associated with the prompt element, an image display interface is output, a camera (or a camera) is turned on to capture an environment image of the real environment, and the captured environment image is displayed on the image display interface. Referring to fig. 5a, fig. 5a is a schematic diagram illustrating an image display interface provided by an exemplary embodiment of the present application; as shown in fig. 5a, an image display interface 501 is output in the terminal screen, and the environment image collected by the camera can be displayed in real time in the image display interface 501.

It should be noted that, before displaying the virtual object in the environment interface, it is necessary to detect whether a target image area for bearing the virtual object exists in the environment image, and only when the environment image includes the target image area, the virtual object is displayed in the environment image. The presence of the target image region for bearing the virtual object in the environment image may refer to: a platform object that can be used to carry a virtual object exists in the real environment, and an environment image obtained by shooting the real environment at this time may include a platform region (i.e., a target image region) corresponding to the platform object, where the platform region may be used to display the virtual object. For example, if the virtual object is a virtual cake, the target image area included in the environment image for carrying the virtual object may include a plane area corresponding to a plane object (e.g., a table, a stool, etc. with a plane surface) in the current environment, and the plane area may be used for carrying the virtual cake; for another example, if the virtual object is a fish, the target image area included in the environment image and used for bearing the virtual object may include an arc area, where the arc area corresponds to an arc object (e.g., an arc fish tank or a spherical fish tank) in the current environment, and the arc area may be used for bearing the fish; for another example, if the virtual object is a garment, the target image area included in the environment image for carrying the virtual object may include a triangular area, which corresponds to a triangular object (e.g., a triangular hanger) in the current environment, and which can be used for carrying the garment; and so on. For convenience of illustration, the virtual object is taken as a virtual cake, and a target image area for bearing the virtual object included in the environment image includes a plane area, that is, a scene including the plane object in the real environment is taken as an example, and the embodiment of the present application is described in the following.

Displaying the virtual object in the environment image may refer to: and displaying the virtual object at a target position corresponding to a target image area in the environment image, wherein the target position corresponds to a target reference area on the environment image, and the target image area included in the target reference area corresponds to the plane object in the current environment. In a specific implementation, if a prompt element in a social interaction interface is triggered, a first prompt message is displayed in an image display interface in response to a trigger operation on the prompt element, where the first prompt message is used to send a prompt about determining a target image area from an environment image. Referring to fig. 5a again, as shown in fig. 5a, a first prompt message 5011 is displayed in the image display interface 501, where the first prompt message 5011 prompts a user to adjust a position or an orientation of a camera (or a camera, disposed in a terminal device) so that an environment image captured by the camera includes a target image area; the first prompt message 5011 is in text form, but of course the first prompt message may be in other forms, such as an icon 5012 in fig. 5a, and the icon 5012 is used to prompt the user to adjust the shooting direction to find a platform object (here, a plane object) of the type of the icon 5012. When the target image area is detected in the environment image, it indicates that there is a platform object available for bearing a virtual object in the current environment, such as the virtual object is a virtual cake, and indicates that there is a planar object (such as a desktop) available for bearing a virtual cake in the current environment, and at this time, the virtual object, such as the virtual object 5013, can be displayed in the target image area in the environment image. As can be seen from the above description, the way of combining the virtual object with the AR technology can improve the sense of reality of the virtual object displayed in the real environment, create a richer ambience, and improve the interactivity between the user and the virtual object.

It should be noted that (1) in addition to displaying the virtual object in the environment image, other animation effects (such as animation content, voice content, user identification of the presenter, etc.) may be displayed in the environment image during the process of displaying the virtual object in the environment image, so as to enhance the display atmosphere of the virtual object and enrich the display content and manner. Referring to fig. 5a again, when the virtual object 5013 is displayed at the target position in the environment image, animation effects such as a balloon and a blessing word may be displayed in the display area where the virtual object is located, which is helpful to create a more real birthday atmosphere. The user identification of the donor, such as the user name, may also be displayed at the display location where the virtual object is located to help the target user see the donor of the virtual object at a glance. The embodiment of the present application does not limit the form and number of the animation effect displayed in the environment image, for example, a speaker identifier may also be displayed in the environment image, and when the speaker identifier is triggered, the audio or video recorded by the presenter for the target user may be played, which is described herein. (2) In the process of displaying the virtual object in the environment image, the interactive operation between the target user and the virtual object is supported, so that the interactivity between the target user and the virtual object is improved, and the AR experience of the target user is improved. In a specific implementation, prompt information about target control operation can be further displayed on the image display interface, and the virtual object displayed at the target position of the environment image can be dynamically processed in response to the detected target control operation; wherein the target control operation is determined by identifying environment interaction data, and the environment interaction data may include: the method comprises the following steps of acquiring environmental sound data (such as sound data of a target user acquired by a microphone) by a microphone, and calling environmental action data (such as action data of the target user for executing target control operation acquired by a camera) shot by the camera. After the environment interaction data are obtained, the environment interaction data can be subjected to action identification processing to obtain an identification result; and if the identification result comprises the target control operation, responding to the target control operation to perform dynamic adjustment processing on the virtual object. For example, in a social interaction scenario in which the virtual object is a virtual cake, a prompt message is included on the image display interface, and the prompt message is used to prompt the target user to blow out a candle, so that after the target user performs an operation of blowing toward a microphone, if it is detected that an input decibel value meets a condition, the candle on the virtual cake can be turned off, and the prompt message on the image display interface is deleted. An exemplary schematic diagram of blowing air at the microphone and extinguishing the candle can be seen in fig. 5b. The process of blowing out the candle in the simulated real environment improves the AR experience of the target user and creates a more real birthday atmosphere.

Embodiments of the present application also support switching the display or displaying different virtual objects simultaneously in the environment image, which may be from different or the same presenter. For example, the target user receives the virtual objects sent by the donor a, the donor B and the donor C, the virtual objects sent by the donor a, the donor B and the donor C may be displayed in the environment image at the same time, or the virtual object sent by the donor a is displayed in the environment image, and when the switching operation is detected, the virtual objects sent by other donors are switched and displayed in the environment image, such as switching from the virtual object sent by the donor a to the virtual object sent by the donor B.

In specific implementation, a folding option is included in an image display interface, an object display window is output in the image display interface in response to a trigger operation on the folding option, the object display window comprises one or more reference virtual objects, and the one or more reference virtual objects are sent to a target user by other donors and are not triggered to be displayed by the target user; and in response to the selection operation on the object display window, updating and displaying the currently displayed virtual object in the environment image based on the reference virtual object selected by the selection operation. Wherein updating the display process may include: replacing the currently displayed virtual object with a reference virtual object selected by the selected operation in the environment image; or, in the environment image, the reference virtual object selected by the selected operation is displayed at the same time when the virtual object is currently displayed. The one or more reference virtual objects included in the object display window may be displayed in an ordered manner according to the time sequence of adding the reference virtual objects by the presenter, for example, a reference virtual object with an earlier presentation time may be displayed before or on a reference virtual object with a later presentation time; of course, the one or more reference virtual objects may also be displayed in other manners, for example, according to the intimacy between the donor and the target user, if the intimacy between the donor and the target user is higher, the reference virtual object given by the donor may be displayed before or on the reference virtual object added by other donors with lower intimacy; and so on. In addition, if the same presenter presents a plurality of reference virtual objects, the plurality of reference virtual objects can still be displayed in the object display window in the above manner, and the specific display manner is not described herein again.

An exemplary flowchart of the process of updating and displaying the virtual object in the environment image may be seen in fig. 6, as shown in fig. 6, a folding option 601 is included in the image display interface, when there are other non-displayed reference virtual objects related to the target user, a prompt message 6011 may also be displayed at a display position where the folding option 601 is located, where the prompt message 6011 is used to prompt the target user that there are non-displayed reference virtual objects; when the folding option 601 is triggered, outputting an object presentation window 602 in the image display interface, wherein all or part of reference virtual objects related to the target user, such as a reference virtual object 6021, a reference virtual object 6022, a reference virtual object 6023, … …, and the like, are contained in the object presentation window 602; according to the selection operation in the object presentation window 602, when any reference virtual object (such as the reference virtual object 6021) is selected from the object presentation window 602, the virtual object 5013 in the environment image may be displayed instead of the reference virtual object 6021, or the virtual object 5013 and the reference virtual object 6021 may be simultaneously displayed in the environment image. The object display window 602 may be displayed in a separate interface form, except that the object display window is displayed on the image display interface in an overlaid manner as shown in fig. 6; in addition, since the display area of the object display window is limited, all the reference virtual objects related to the target user may not be completely displayed in the object display window, and thus the embodiment of the present application supports sliding display of the hidden reference virtual object in the object display window, and a specific sliding display manner may refer to the foregoing description, which is not described herein again.

It is understood that if the type of the reference virtual object selected by the selection operation is different from the type of the virtual object currently displayed in the environment image, the target image area for displaying the reference virtual object may not be included in the environment image, and a second prompt message may be output to prompt the user to adjust the shooting angle or the shooting position of the camera so that the new environment image to be shot includes the target image area for displaying the reference virtual object. For example, if the reference virtual object is a fish, the platform object corresponding to the target image area for displaying the reference virtual object may include a fish tank; when detecting that the currently acquired environment image does not contain the target image area corresponding to the fish tank in the real environment, a second prompt message can be output to prompt the user to adjust the camera so as to obtain a new environment image containing the target image area corresponding to the fish tank.

In a specific implementation, if the environment image does not contain a reference image area for displaying a reference virtual object selected by the selected operation, outputting a second prompt message, wherein the second prompt message is used for prompting the adjustment of the shooting direction; determining a reference image area in the new environment image acquired after the adjustment, wherein the reference image area corresponds to a platform object (such as a plane object) in the adjusted environment (i.e. the real environment); the reference virtual object is displayed according to the reference virtual area. The above process of outputting the second prompt message can be seen in fig. 7, as shown in fig. 7, where the virtual object 5013 displayed in the current environment image is a virtual cake, and if the reference virtual object 6024 selected by the target user in the object display window, the reference virtual object 6024 is a fish, the reference virtual object 6024 and the virtual object 5013 do not belong to the same type, and the platform object corresponding to the reference virtual object 6024 and the platform object corresponding to the virtual object 5013 are not the same, it may be determined that a reference image region for displaying the fish is not included in the current environment image, and then a second prompt message 701 is output, where the second prompt message 701 is used to prompt the target user to adjust the camera shooting direction; after the camera acquires a new environment image after the camera shooting direction is adjusted, the new environment image comprises a reference image area 702 corresponding to the reference virtual object 6024, and the reference image area 702 corresponds to a fish tank in the real environment, so that the reference virtual object 6024 can be displayed in the reference image area 702 in the new environment image.

The method and the device also support screen recording processing of the image display interface and share the multimedia data obtained after screen recording processing; in other words, the embodiment of the application also supports screen recording processing of the environment image containing the virtual object, and shares multimedia data obtained through screen recording processing. In specific implementation, the image display interface comprises a recording option, the screen recording is carried out in response to the triggering operation of the recording option, and recording progress information is displayed at the display position of the recording option and used for expressing the time length for recording the multimedia data; displaying a screen recording editing interface, wherein the screen recording editing interface comprises a sharing control; and responding to the triggering operation of the sharing control, and sharing the multimedia data obtained by screen recording.

An exemplary process of sharing multimedia data may be seen in fig. 8a, as shown in fig. 8a, a recording option 801 is included in the image display interface, when the recording option 801 is executed with a triggering operation (such as long press, single click, double click, etc.), recording progress information 802 is displayed at a display position of the recording option 801, and the recording progress information 802 may be used to indicate a time duration for recording multimedia data, such as 9 seconds, indicating that multimedia data of 9 seconds has been recorded; if the triggering operation for the recording option 801 is finished, for example, a triggering signal for triggering the recording option 801 is stopped, or the recording option 801 is clicked again, the recording screen editing interface 803 may be displayed, where the recording screen editing interface 803 includes an editing control and a sharing control 8031 for the multimedia data obtained by processing the recording screen; the target user can edit the multimedia data obtained through screen recording processing (such as cutting, adding dynamic effects, beautifying and the like) by triggering the editing control, and when the target user triggers the sharing control 8031, the target user responds to the user operation on the sharing control, obtains the multimedia data obtained through screen recording processing, and shares the multimedia data.

The multimedia data obtained by screen recording can be shared to other users, and can also be shared to a social dynamic interface of a target user; based on this, an exemplary process schematic diagram for sharing multimedia data is provided in the embodiment of the present application, as shown in fig. 8b, if the trigger operation is finished on the recording option 801, the sharing object window 804 is output in the image display interface, where the sharing object window 804 includes one or more sharing identifiers, and the sharing identifiers include: one or more of an object identifier, a social dynamic identifier, and a link identifier of the sharing object, for example, the sharing identifier includes a user identifier 8041 and a social dynamic identifier 8042 of the user a; when the user identifier of any user in the sharing object window 804 is selected according to the identifier selection operation, outputting a social session interface 805 between the target user and the selected user (such as the user a), and displaying a session message 8051 containing multimedia data in the social session interface 805; when the social dynamic identifier 8042 in the sharing object window 804 is selected according to the identifier, which indicates that the target user wants to send a social dynamic (such as a circle of friends), the social dynamic interface 806 of the target user is displayed, and a social dynamic message 8061 related to the multimedia data is displayed in the social dynamic interface 806. It should be noted that the style and the content of each interface shown in fig. 8a and 8b may be changed, for example, the sharing object window in fig. 8b may also be displayed on a separate interface, instead of being displayed on the image display interface in an overlaying manner, and the like.

Of course, the multimedia data obtained after the screen recording process also supports downloading to the local space. As shown in fig. 8c, if the triggering operation on the recording option 801 is finished, before the sharing object window 804 is output in the image display interface, a selection interface 807 may be further output, where the selection interface 807 includes a sharing option 8071 and a downloading option 8072, when the sharing option 8071 is triggered, the sharing object window 804 may be triggered to be displayed, when the downloading option 8072 is triggered, the multimedia data may be downloaded to the local space, and a notification interface 808 is output, and when the downloading is completed, the download progress information is displayed in the notification interface 808, and a notification message notifying the target user that the downloading is completed is output.

The method and the device also support the target user to give the target virtual object to other users, namely the target user can also be used as a donor to give the target virtual object to other users, so that the other users can execute various operations for checking the target virtual object on the terminal equipment used by the other users; the target virtual object may be an object that needs to be paid or an object that is free of charge, which is not limited in this embodiment of the present application. In a specific implementation, a candidate object interface is displayed, where the candidate object interface includes object identifiers of one or more candidate virtual objects (e.g., names, pictures, etc. of the candidate virtual objects); responding to the selection operation on the candidate object interface, and outputting an object detail interface, wherein the object detail interface comprises object information of a target virtual object selected from the candidate object interface according to the selection operation and a confirmation option; after the trigger operation of the confirmation option is detected, outputting a social interaction interface with the sharing object, and displaying a prompt element related to the target virtual object in the social interaction interface with the sharing object.

Describing the process of giving virtual objects in conjunction with fig. 9a, as shown in fig. 9a, a candidate object interface 901 is displayed, where the candidate object interface 901 includes object identifiers of one or more candidate virtual objects, such as an object identifier 9011 of a candidate virtual object 1, object identifiers 9012, … … of a candidate virtual object 1, and the like; in response to a selection operation of an object identifier of any candidate virtual object (such as candidate virtual object 1) in the candidate object interface, outputting an object detail interface 902, where object information (such as an object identifier, dynamic effect information related to the candidate virtual object, and the like) of the target virtual object is included in the object detail interface 902; the object detail interface 902 also includes a confirmation option 9021, and when the confirmation option 9021 is triggered, the social interaction interface 903 with the shared object is displayed, and a prompt element 9031 about the target virtual object is displayed in the social interaction interface 903. The embodiment of the present application further supports the target user to edit in the object detail interface 902, where the editing may include: the embodiment of the present application does not limit the specific editing operation executed by the target user in the object detail interface, and this is described herein.

The described sharing object may be any sharing object selected from the sharing object interface before the candidate object interface is displayed. Referring to the first diagram of fig. 9b, the shared object interface 904 includes object identifiers of a plurality of shared objects, where the plurality of shared objects may be recommended to a target user by the system according to user information (such as birthday information and preference information) of each shared object, and the target user may select any one or more shared objects from one or more shared objects included in the shared object interface to give a virtual object. For example, other users related to the target user include a user a, a user B, a user C, and a user D, where the birthday dates of the user a and the user D are the current month, and the birthday dates of the user B and the user C are not the current month, the user a and the user D may be used as the sharing objects, and the user identifier of the user a and the user identifier of the user D are displayed in the sharing object interface, so that the target user may select any one or more sharing objects to give away the virtual object. In addition, after the confirmation option in the object detail interface is triggered and before the social interaction interface between the object detail interface and the sharing object is output, a waiting interface can be displayed, and the waiting interface is used for prompting the target user that the target virtual object is sent to the sharing object; a schematic diagram of an exemplary wait interface can be seen in the second drawing of fig. 9 b.

The target user may also experience the AR effect of the target virtual object before gifting the target virtual object. Referring to fig. 9a again, the object detail interface 902 further includes an AR experience control 9022, and when the AR experience control 9022 is triggered, the AR experience control indicates that the target user wants to experience a process of viewing the target virtual object in advance; the process of viewing the target virtual object may include a part of the above-described implementation processes shown in steps S301 to S303, such as a process of acquiring an environment image and displaying the virtual object at the target position of the environment image. The method supports the target user to perform AR experience on the target virtual object before presenting the target virtual object to the sharing object, helps the target user to detect whether the set target virtual object meets the self requirement, and can improve the AR experience of the user.

In addition, the target virtual object can be obtained by the target user based on the real environment drawing besides being pushed by the system, so that the target virtual object is displayed in the environment image when the environment image needs to be identified as the image corresponding to the real environment when the target virtual object is drawn in the subsequent AR experience scene of the target virtual object; the image corresponding to the real environment when the environment image is the target virtual object may be: the real environment indicated by the environment image is similar to the real environment when the target virtual object is drawn, that is, the environment image should include a target image region for carrying the target virtual reality. For example, a frame of environment image containing a clothes hanger is acquired, if a target user draws a piece of clothes in an area where the clothes hanger is located in the environment image, it is determined that the drawn target virtual object is the clothes, and when it is detected that a target image area contained in the acquired environment image corresponds to the clothes hanger (i.e., the platform object) in the real environment, the target virtual object is displayed at a target position corresponding to a target reference area on the environment image. The target user self-defines the target virtual object, so that the types of the target virtual object are enriched, and the specificity and uniqueness of the target virtual object are improved.

In the embodiment of the application, whether the environment image includes the target image area or not can be analyzed, and when the environment image includes the target image area, the virtual object can be displayed at the target position of the target image area on the environment image. The virtual information is combined with the AR technology, so that the process of superposing the entity object corresponding to the virtual object on the platform object in the real environment can be simulated, the virtual object can be displayed in a target image area which is contained in the environment image and used for bearing the virtual reality object, the sense of reality of the virtual information displayed in the environment image is improved, the display mode of the virtual object is enriched, the interactivity between a target user and the virtual object is improved, and the visual immersion of the user is enhanced.

The above-mentioned embodiment shown in fig. 3 describes the interface flow of the social interaction method in more detail, and the background technology is described below with reference to fig. 10. Referring to fig. 10, fig. 10 is a flow chart illustrating a social interaction method according to an exemplary embodiment of the present application; the social interaction method may be executed by the electronic device in the system shown in fig. 2, for example: the terminal device and the service device execute independently or the terminal device and the service device execute together. As shown in FIG. 10, the social interaction method includes, but is not limited to, steps S1001-S1004.

S1001: and acquiring an environment image and acquiring normal vector information corresponding to each target point on the environment image. The environment image is obtained by calling a camera (or a camera) of the terminal equipment to shoot a real environment; it can be understood that after the camera is turned on, the image collected by the camera can be displayed in real time in the terminal screen of the terminal.

In specific implementation, a normal vector model can be used to identify image normal vector information of each target point in the environment image in an image coordinate system, and then the image normal vector information corresponding to each target point on the environment image is adjusted according to the orientation of the AR system, so that the image normal vector information corresponding to each target point is converted into a gravity coordinate system, and the normal vector information corresponding to each target point is obtained. Therefore, whether the platform object corresponding to the normal vector information corresponding to the target point is a plane object or not can be judged through the normal vector information corresponding to the target point under the gravity coordinate system, and only when the platform object is the plane object, a virtual cake (namely, a virtual object) can be placed on the plane object, so that the reality sense of the AR effect is improved. The normal vector model may include, but is not limited to, a deep learning monocular normal vector model, which may be obtained by performing supervised learning training on a machine learning model (such as a neural network model) by using a sample environment image set. The sample environment image set comprises a plurality of sample environment images and a label corresponding to each sample environment image, and the label corresponding to any sample environment image comprises: and the real normal vector information of the target point in any sample environment image is obtained by calculating a depth image obtained by shooting the real environment.

S1002: and determining a connected image area on the environment image according to the normal vector information of each target point.

Based on step S1001, normal vector information corresponding to each target point in the environmental image may be obtained, and the normal vector information corresponding to each target point may reflect: and the direction corresponding to the normal vector of the entity object contained in the real environment at the target point. The direction corresponding to the normal vector information corresponding to each target point included in the connected image area in the environment image and the target direction satisfy a consistency condition, where the consistency condition may be: the direction corresponding to the normal vector information corresponding to each target point in the communicated image area is the same as the target direction; or, an angle difference between a direction corresponding to the normal vector information corresponding to each target point included in the connected image area and the target direction is smaller than a preset angle (for example, 10 degrees). As described above, the platform objects for carrying the virtual objects are different according to the types of the virtual objects, and then the target directions are different; for example, if the virtual object is a virtual cake and the platform for placing the virtual cake is a horizontal plane, the target direction is a vertical direction or a gravity direction, that is, the direction corresponding to the normal vector information corresponding to each target point included in the determined connected image region and the gravity direction satisfy the consistency condition.

Specifically, the implementation of determining a connected image area on the environmental image according to the normal vector information of each target point may include: screening N target points which meet the consistency condition with the target direction from the environment image, wherein N is an integer larger than zero; determining M candidate connected image areas according to the N target points, wherein M is a positive integer less than or equal to N, and each candidate connected image area comprises a plurality of adjacent target points; and taking the candidate connected image area with the largest area in the M candidate connected image areas as the connected image area. That is to say, the direction corresponding to the normal vector information may be taken as a target direction as a constraint condition, and a connected domain algorithm is combined to screen out all target points in the environment image, the target directions of which meet a consistency condition, adjacent target points may form a plurality of candidate connected image regions, and a candidate connected image region with the largest area is found from the plurality of candidate connected image regions as a connected image region; when the target direction is a direction perpendicular to the horizontal plane, the connected image region may refer to a connected horizontal plane. An exemplary schematic diagram of a connected image region in an environment image can be seen in fig. 11, as shown in fig. 11, the environment image includes a horizontal table and a plurality of dinner plates placed on the table, and after steps S1001-S1002 are performed on the environment image, normal vector information corresponding to each target point in the environment image can be obtained; then, screening N target points which meet the consistency condition with the direction vertical to the horizontal plane from the environment image, and forming a candidate connected image area by using any adjacent target points to obtain three candidate connected image areas which are a candidate connected image area 1101, a candidate connected image area 1102 and a candidate connected image area 1103 respectively; the candidate connected image region with the largest area among the three candidate connected image regions is taken as a connected image region, for example, the candidate connected image region 1103 is taken as a connected image region.

S1003: a target reference region is determined in the connected image region. The target reference region may include: a region within a target range (e.g., pixel 200 x 200) centered at the virtual click location in the environment image; the target reference area can be understood as a tracking frame, and the motion trend of the camera can be determined in real time through tracking the tracking frame in the continuously shot environment images. The tracking frame (i.e., the target reference area) may include a target image area, the target image area may be used to display a virtual object, and the tracking frame may also be an area formed by target points with obvious image features in the environment image, so that the tracking frame can be better tracked, and a more accurate motion trend of the camera is obtained. In one embodiment, the virtual click location in the environment image may refer to a location of a center point of the connected image region. Initialization of camera position information and initialization of model position information according to the virtual click position, that is, taking the camera position information when the virtual click position is generated as the origin of a camera coordinate system (or world coordinate system), and determining the origin of an AR scene coordinate system according to the position of the virtual click position on an environment image may be performed.

Referring to fig. 11 again, as shown in fig. 11, when the connected image region in the environment image is the candidate connected image region 1103, since the shape of the connected image region is an irregular shape, the connected image region may be preprocessed first to obtain a connected image region with a regular shape; then, determining the position of a central point in the preprocessed connected image area as a virtual click position 11031; the preprocessing of the irregular-shaped connected image region can include operations such as filling and deleting of an irregular boundary of the connected image region, so that the irregular-shaped connected image region is converted into a more regular connected image region.

After the virtual click position is determined in the connected image area based on the steps, the embodiment of the application further generates a simulated click signal according to the virtual click position, namely, the virtual click position is simulated into a click signal of the user on the environment image, and the simulated click signal is output, so that the simulated click signal is triggered to perform AR positioning initialization processing. After the AR system performs AR positioning initialization according to the simulated click signal, a local area may be initialized on the environment image according to the virtual click position, and an area (i.e., a target range) with a certain pixel size around the virtual click position in the local area is taken as a target reference area, and an exemplary schematic diagram of the target reference area in the environment image may be as shown in fig. 11, such as the target reference area 11032.

Wherein, outputting the analog click signal may include: the simulated click signal can be input into the AR system, so that the AR system can perform AR positioning initialization after receiving the simulated click signal; AR positioning initialization may include: initialization of camera position information and initialization of analog position information. The camera position information may include position information and rotation information of the camera, and the initialization of the camera position information may refer to: the position information (such as translation information) and the rotation information (such as inclination angle information) of the camera are initialized, so that after the position information and the rotation information of the subsequent camera are changed, the camera can be effectively tracked based on a camera coordinate system established by initialization. Specifically, position information of the camera may be calculated according to a first frame of environment image acquired by the camera (or the camera), and the position information of the camera at this time may be determined as an origin position of a camera coordinate system, where the camera coordinate system is a three-dimensional coordinate system, and the origin position of the camera position may be represented as (0,0,0); and determining the inclination angle of the camera (or the terminal equipment carried by the camera) when the camera collects the first frame of environment image according to the gravity direction, and taking the inclination angle as the initial rotation angle of the camera. The simulation position information may include position information and rotation information of the virtual object, and the initialization of the simulation position information may refer to: the position information and the rotation information of the virtual object are initialized, so that in the subsequent shooting process, the position information and the rotation information of the virtual object can be kept as the initialized simulated position information, and the changed camera position information of the camera can be used for quickly analyzing the position of the virtual object to be displayed in the environment image. Specifically, the position information of the virtual object may be determined by a ray emitted from a coordinate origin of a camera coordinate system, and in more detail, according to the camera coordinate system and the position of the simulated click signal, a ray may be emitted from the coordinate origin of the camera coordinate system, and according to a preset target distance (e.g., 0.5 m) between the target user in the real environment and the virtual object, the position of the target distance from the coordinate origin on the ray may be determined as the position information of the virtual object.

S1004: and rendering the virtual object according to the target position of the target reference area in the environment image so as to display the virtual object on the environment image. In one embodiment, the rendering process of the virtual object may be implemented by openGL (Open Graphics Library) to display the virtual object on the environment image.

In one embodiment, a feature point set may be first determined in the target reference region, where the feature point set includes a plurality of image feature points extracted from the target reference region, and the image feature points may be used to indicate attribute information (e.g., grayscale information, brightness information, etc.) of pixel points corresponding to the image feature points.

And secondly, performing feature point matching and tracking processing according to the determined feature point set and the obtained environment updating image, and obtaining camera position information according to a tracking processing result, wherein the camera position information can be used for representing any one or two of position change and orientation change of the camera in a world coordinate system in the process from shooting the environment image to the environment updating image. In detail, sparse optical flow processing may be performed according to the feature point set, so that a tracking algorithm (e.g., a KLT tracking algorithm) may be used to track the optical flow, and a tracking processing result between two previous and next frames of environment images (i.e., a first frame of environment image and a second frame of environment image (which may be referred to as an environment update image)) may be obtained, where the tracking processing result may be embodied as a matching relationship between the two previous and next frames of environment images, and the matching relationship may reflect position information of the same optical flow in the two frames of environment images, so that one or more of a position change or an orientation change of the camera in a process from capturing the environment image to capturing the environment update image may be inferred according to the matching relationship between two adjacent frames of environment images.

Finally, rendering the virtual object according to the camera position information so as to display the virtual object in the environment updating image; that is, the motion trend of the camera in the process of shooting the environment image to the environment updating image can be estimated according to the camera position information, and then the display position of the virtual object in the environment updating image is determined according to the feature point set, so that the display positions of the rendered virtual object in the environment image and the environment updating image are matched. It should be noted that, if the acquired environment update image and the environment update image are the same, that is, the camera does not move during the process of capturing the environment image and the environment update image, the virtual object displayed in the terminal screen is always displayed at the same display position.

In the above description, the implementation manner of rendering the virtual object according to the camera position information may include: and calling an image rendering process to render the virtual object to obtain the virtual object rendered in the environment image (or the environment updating image), wherein the image rendering process can be a process designed based on openGL. And the image rendering process renders the virtual object according to the initialized model position information, the initialized camera position information and the initialized projection matrix. The image rendering process can comprise an MVP matrix, and the MVP matrix comprises an M matrix, a V matrix and a P matrix under the rendering system; wherein the M matrix is a matrix of model position information (model matrix), the V matrix is a matrix of camera position information (camera view matrix) and the P matrix is a projection matrix (project matrix); the virtual object can be rendered in the environment image according to the MVP matrix.

Referring to fig. 12, a flow of calling an image rendering process to perform rendering processing on a virtual object may be shown, and as shown in fig. 12, first, position information of the virtual object in an AR coordinate system may be converted into simulated position information in a world coordinate system; secondly, calculating updated camera position information according to the tracking processing result and a PNP algorithm (a method for solving Point pair motion of 2D → 3D or 3D → 2D); finally, in the subsequent tracking process according to the feature point set, the simulated position information of the virtual object is unchanged, but the camera position information is changed, after the changed camera position information is acquired, the camera position information corresponding to the environment image is updated to the camera position information corresponding to the environment updating image in the rendering system, the position information of the virtual object in the image coordinate system can be obtained according to the simulated position information of the virtual object, the changed camera position information and the projection matrix, and then the virtual object in the environment updating image is obtained according to the position information rendering of the virtual object in the image coordinate system; after a new environment image is acquired, the process is executed on the new environment image, so that the continuous AR display effect of the virtual object can be obtained in the adjacent multi-frame environment images.

An exemplary continuous AR presentation effect can be seen in fig. 13, as shown in fig. 13, in which a target reference area in an environment image taken by a camera is located at a first display position 1501, and then a virtual object can be displayed at the first display position 1501; according to the tracking processing result of the target reference region, when the target reference region moves to the second display position 1502 in the environment updating image, the camera position information in the rendering system is modified according to the rendering process, and the virtual object in the environment updating image can be rendered. As can be seen from fig. 13, although the environment image is not the same as the environment updated image, the target reference areas in the environment image and the environment updated image have a matching relationship, so that the virtual cakes are all placed at the same position in the two frames of images, thereby realizing that the virtual objects can be continuously displayed at the corresponding target positions in the continuous multi-frame environment images, and improving the sense of reality of the virtual objects in the environment image.

In addition, just as the embodiment of the present application described in the embodiment of fig. 3 also supports the target user to give away the target virtual object to other users, the process of performing the generation of the target virtual object in the background may refer to fig. 14a, and as shown in fig. 14a, the process of generating the target virtual object may include:

firstly, a target user can request material information of a plurality of candidate virtual objects from a server through a terminal device, specifically, the material information of the plurality of candidate virtual objects is requested from a material resource module contained in the server, and the material resource module can be used for storing the material information of the candidate virtual objects. And after receiving the material information request sent by the terminal, the material resource module can return the material information of a plurality of candidate virtual objects to the terminal.

Secondly, the terminal device obtains the object material of each candidate virtual object according to the material information of each candidate virtual object, specifically, the terminal sends a request for obtaining the object materials of a plurality of candidate virtual objects to a server (such as a material background module contained in the server), and receives the object material of each candidate virtual object returned by the server, so that the terminal device displays the object material of each candidate virtual object in a candidate object interface. And when the target user performs the selection operation in the candidate object interface, the terminal can receive the selection operation of the object materials of the target virtual object in the candidate object interface.

Finally, the terminal equipment generates a target virtual object based on the object material of the selected target virtual object and the acquired user interaction information of the target user; specifically, the terminal may employ a shadow module (e.g., a shadow sdk module) to generate the target virtual object based on the object material of the target virtual object and the acquired user interaction information of the target user. The user interaction information may include a user identifier of a target user (e.g., information such as user gender, nickname, age, etc.), and editing information for the target virtual object, where the editing information is generated for the target user, and the editing information includes: one or more items of voice information, animation information and character information; therefore, the target virtual object generated based on the user interaction information can carry the relevant information (such as user identification and editing information) of the target user, and the specificity of the virtual object is improved. Of course, the editing information may further include a decibel value and user information of the user to be presented (i.e., the user receiving the target virtual object), where the decibel value may be used as a condition for the subsequent user to be presented to perform the target control operation, and if the target virtual object is a virtual cake, it is determined that the candle on the virtual cake is turned off only when the decibel value of the user to be presented when blowing air toward the microphone reaches the decibel value set by the target user. The target virtual object generated based on the process can carry target user-defined information and user information of the target user, so that the target user can be helped to define the target virtual object, the specificity of the target virtual object is improved, and the user experience is improved.

In addition, the server also comprises a basic background module, so that when or after the terminal requests the material information of the candidate virtual objects from the material resource module, the terminal can also send a constellation information request to the basic background module, wherein the constellation information request is used for requesting the basic background module to return constellation information of a target user or a user to be presented, so that the terminal can display the constellation information of the target user in a shared object interface or an object detail interface, the target user can conveniently know own constellation information when configuring the target virtual object, and a more appropriate target virtual object can be selected for the user to be presented according to the constellation information of the user to be presented.

Fig. 14a mainly introduces the background implementation in which the target user selects the target virtual object from the multiple candidate virtual objects, so that after the target user selects the target virtual object, the selected target virtual object may be given to another user (e.g., a candidate to be given) so that the other user can view the target virtual object in the terminal used by the other user.

In one embodiment, the flow of the target user selecting the target virtual object and giving the target virtual object to other users, and the other users viewing the target virtual object in the terminal used by the other users, is described with reference to fig. 14 b; as shown in fig. 14b, assuming that the terminal used by the target user is terminal 1 and the terminals used by other users are terminals 2, the process of selecting, rendering and displaying virtual objects such as gifts includes:

firstly, a target user can select a target virtual object from a plurality of candidate virtual objects through the terminal 1, specifically, the target virtual object is selected in the object mall module, and at the moment, the object mall module can send related information of the target virtual object to the material resource module so that the material resource module stores the related information of the target virtual object; the related information of the target virtual object may include material information of the target virtual object and information such as a user identifier of the target user. It should be noted that, the specific implementation process of the target user selecting the target virtual object through the terminal 1 can refer to the specific implementation process shown in fig. 14 a.

Secondly, the target user can send a prompt element to other users (i.e. the persons to be presented) through the terminal 1, wherein the prompt element is related to the target virtual object and can be sent through corresponding applications and application interfaces mentioned in other embodiments of the application; after other users trigger the prompt element, the terminal 2 can send an object material request about the target virtual object to the basic background module; after receiving the object material request sent by the terminal 2, the basic background module requests the material resource module to acquire the material information of the target virtual object and the user identifier of the target user, so that the material resource module returns the material information of the target virtual object and the user identifier of the target user to the basic background module; the basic background module requests the material background module for the object material of the target virtual object according to the material information of the target virtual object; and after receiving the object material of the target virtual object returned by the material background module, the basic background module returns the object material of the target virtual object to the terminal 2.

Finally, the terminal 2 receives the object material of the target virtual object returned by the basic background module and the information such as the user identifier of the target user, and executes the step of generating the target virtual object based on the object material of the target virtual object and the acquired user interaction information of the target user, where the user interaction information may include the user identifier of the target user.

The above describes the background implementation of viewing the target virtual object by interacting between the terminal 2 and the basic background module, but it can be understood that the terminal 2 can also directly perform information interaction with other functional modules (such as the material background module and the material resource module) to view the target virtual object. Similarly, the terminal 1 may further send a request for obtaining the material information of the plurality of candidate virtual objects to the basic background module, and the basic background module obtains the material information of the plurality of candidate virtual objects; and so on. In summary, the embodiment of the present application does not limit the data flow direction between the background function modules, and is described herein.

In the embodiment of the application, after the environment image is obtained, normal vector information corresponding to each target point contained in the environment image can be analyzed, and a target position which can be used for bearing a virtual object in the environment image is determined according to the normal vector information of each target point. In the process, the virtual object is displayed at the target position which is contained in the environment image and used for bearing the virtual reality object, so that the sense of reality of virtual information displayed in the environment image can be improved, the display mode of the virtual object is enriched, the interactivity between the target user and the virtual object is improved, and the visual immersion of the user is enhanced.

Fig. 15 is a schematic structural diagram illustrating a social interaction device according to an embodiment of the present disclosure; the social interaction device is arranged in a terminal or a server. In some embodiments, the social interaction device may be a client running in a server (e.g., an application having a function of identifying a stamp); the social interaction device may include units that are realized as described in the foregoing embodiments. Referring to fig. 15, the social interaction apparatus according to the embodiment of the present disclosure includes the following units:

an obtaining unit 1501, configured to obtain an environment image, and obtain normal vector information corresponding to each target point on the environment image;

the processing unit 1502 is configured to determine a connected image region on the environmental image according to the normal vector information of each target point, where a direction corresponding to the normal vector information corresponding to each target point included in the connected image region and a target direction meet a consistency condition;

the processing unit 1502 is further configured to determine a target reference area in the connected image area;

the processing unit 1502 is further configured to perform rendering processing on the virtual object according to the target position of the target reference area in the environment image, so as to display the virtual object on the environment image.

In an implementation manner, when determining a connected image area on an environment image according to the normal vector information of each target point, the processing unit 1502 is specifically configured to:

screening N target points meeting the consistency condition with the target direction from the environment image, wherein N is an integer larger than zero;

determining M candidate connected image areas according to the N target points, wherein M is a positive integer less than or equal to N, and each candidate connected image area comprises a plurality of adjacent target points;

and taking the candidate connected image area with the largest area in the M candidate connected image areas as the connected image area.

In one implementation, the processing unit 1502 is further configured to:

determining a virtual click position in the connected image area;

generating a simulated click signal according to the virtual click position, and outputting the simulated click signal so as to trigger AR positioning initialization processing through the simulated click signal;

wherein, AR location initialization processing includes: initializing camera position information and model position information; the target reference region includes: a region within the target range centered on the virtual click position in the environment image.

In one implementation, when the processing unit 1502 is configured to perform rendering processing on a virtual object according to a target position of a target reference area in an environment image, the processing unit is specifically configured to:

determining a feature point set in a target reference region, wherein the feature point set comprises a plurality of image feature points;

performing characteristic point matching tracking processing according to the characteristic point set and the obtained environment updating image;

obtaining camera position information according to the tracking processing result, wherein the camera position information is used for indicating any one or two of position change and orientation change of the camera in a world coordinate system in the process from shooting an environment image to an environment updating image;

and rendering the virtual object according to the camera position information so as to display the virtual object in the environment updating image.

In an implementation manner, when the processing unit 1502 is configured to perform rendering processing on a virtual object according to the camera position information, specifically, to:

calling an image rendering process to render the virtual object;

and the image rendering process is used for rendering the virtual object according to the initialized model position information, the initialized camera position information and the initialized projection matrix.

In one implementation, the processing unit 1502 is further configured to:

acquiring material information of a plurality of candidate virtual objects;

acquiring the object material of each candidate virtual object according to the material information of each candidate virtual object so as to display the object material of each candidate virtual object on a candidate object interface;

receiving selection operation of object materials of a target virtual object in a candidate object interface;

and generating a target virtual object based on the object material selected by the selection operation and the acquired user interaction information of the target user.

In one implementation manner, when the processing unit 1502 is configured to generate the target virtual object based on the object material selected by the selection operation and the acquired user interaction information of the target user, specifically:

acquiring user interaction information of a target user, wherein the user interaction information comprises: any one or combination of the user identification of the target user and the editing information for the target virtual object, the editing information comprising: generating one or more items of voice information, animation information and text information for the target user;

and generating a target virtual object based on the user interaction information and the object material selected by the selection operation.

In one implementation, the processing unit 1502 is further configured to:

displaying a screen recording editing interface, wherein the screen recording editing interface comprises an editing control and a sharing control of multimedia data obtained by screen recording processing;

and responding to the user operation of the sharing control, acquiring the multimedia data obtained by screen recording processing, and sharing the multimedia data.

In one implementation, the processing unit 1502 is further configured to:

acquiring environment interaction data, wherein the environment interaction data comprises one or a combination of two of environment sound data and environment action data;

performing action recognition processing on the environment interaction data to obtain a recognition result;

and if the identification result comprises the target control operation, responding to the target control operation to perform dynamic adjustment processing on the virtual object.

According to an embodiment of the present application, each unit in the social interaction method shown in fig. 15 may be respectively or entirely combined into one or several other units to form the social interaction method, or some unit(s) may be further split into multiple units with smaller functions to form the social interaction method, which may implement the same operation without affecting implementation of technical effects of the embodiment of the present application. The units are divided based on logic functions, and in practical application, the functions of one unit can be realized by a plurality of units, or the functions of a plurality of units can be realized by one unit. In other embodiments of the present application, the social interaction apparatus may also include other units, and in practical applications, these functions may also be implemented by being assisted by other units, and may be implemented by cooperation of multiple units. According to another embodiment of the present application, the social interaction apparatus as shown in fig. 15 may be constructed by running a computer program (including program codes) capable of executing the steps involved in the corresponding method as shown in fig. 10 on a general-purpose computing device such as a computer including a central processing unit (C PU), a random access storage medium (RAM), a read-only storage medium (ROM), and the like, and a storage element, and the social interaction method of the embodiment of the present application may be implemented. The computer program may be recorded on a computer-readable recording medium, for example, and loaded and executed in the above-described computing apparatus via the computer-readable recording medium.

In the embodiment of the present application, the acquisition unit 1501 may acquire the environment image, and the processing unit 1502 can display the virtual object at the target position of the target image area on the environment image by analyzing whether the environment image includes the target image area or not, and when the target image area is included. The virtual information is combined with the AR technology, so that the process of superposing the entity object corresponding to the virtual object on the platform object in the real environment can be simulated, the virtual object can be displayed in a target image area which is contained in the environment image and used for bearing the virtual reality object, the sense of reality of the virtual information displayed in the environment image is improved, the display mode of the virtual object is enriched, the interactivity between a target user and the virtual object is improved, and the visual immersion of the user is enhanced.

Fig. 16 is a schematic structural diagram illustrating a social interaction device according to an embodiment of the present disclosure; the social interaction device is arranged in a terminal or a server. In some embodiments, the social interaction device may be a client running in a server (e.g., an application having a function of identifying a stamp); the social interaction device may include units that are realized as described in the foregoing embodiments. Referring to fig. 16, the social interaction apparatus according to the embodiment of the present disclosure includes the following units:

the display unit 1601 is configured to display a social interaction interface, where the social interaction interface includes a prompt element;

a processing unit 1602, configured to display the acquired environment image on an image display interface in response to a trigger operation on the prompt element;

a processing unit 1602, further configured to display a virtual object at a target position of the environment image;

In one implementation, the processing unit 1602 is further configured to:

and responding to the triggering operation of the prompting element, and displaying a first prompting message on the image display interface, wherein the first prompting message is used for sending out a prompt about determining a target image area from the environment image.

In one implementation, the image display interface includes a fold option; a processing unit 1602, further configured to:

responding to the triggering operation of the folding option, and outputting an object display window in the image display interface, wherein the object display window comprises one or more reference virtual objects;

and in response to the selection operation on the object display window, updating and displaying the currently displayed virtual object in the environment image based on the reference virtual object selected by the selection operation.

In an implementation manner, the processing unit 1602 is configured to, when performing update display processing on a currently displayed virtual object based on a reference virtual object selected by a selection operation in an environment image, specifically:

replacing the currently displayed virtual object with a reference virtual object selected by the selected operation in the environment image;

or, in the environment image, the reference virtual object selected by the selected operation is displayed at the same time when the virtual object is currently displayed.

In one implementation, the processing unit 1602 is further configured to:

if the environment image does not contain a reference image area for displaying the reference virtual object selected by the selected operation, outputting a second prompt message, wherein the second prompt message is used for prompting the adjustment of the shooting direction;

determining a reference image area in a new environment image acquired after adjustment, wherein the reference image area corresponds to a plane object in the adjusted environment;

the reference virtual object is displayed according to the reference image area.

In one implementation, the image display interface includes a recording option; a processing unit 1602, further configured to:

responding to the triggering operation of the recording option, performing screen recording processing, and displaying recording progress information at the display position of the recording option, wherein the recording progress information is used for expressing the time length for recording the multimedia data;

displaying a screen recording editing interface, wherein the screen recording editing interface comprises a sharing control;

and responding to the triggering operation of the sharing control, and sharing the multimedia data obtained by screen recording.

In one implementation, the processing unit 1602 is further configured to:

displaying a candidate object interface, wherein the candidate object interface comprises object identifications of one or more candidate virtual objects;

responding to the selection operation on the candidate object interface, and outputting an object detail interface, wherein the object detail interface comprises object information of a target virtual object selected from the candidate object interface according to the selection operation and a confirmation option;

after the trigger operation of the confirmation option is detected, outputting a social interaction interface with the sharing object;

and displaying a prompt element related to the target virtual object in a social interaction interface with the sharing object.

In one implementation, the processing unit 1602 is further configured to:

displaying prompt information about target control operation on an image display interface;

dynamically processing a virtual object displayed at a target position of the environment image in response to the detected target control operation;

the target control operation is determined by identifying environment interaction data, the environment interaction data comprising: and one or two of the environmental sound data acquired by the microphone and the environmental action data shot by the calling camera are combined.

According to an embodiment of the present application, each unit in the social interaction method shown in fig. 16 may be respectively or entirely combined into one or several other units to form the social interaction method, or some unit(s) may be further split into multiple units with smaller functions to form the social interaction method, which may achieve the same operation without affecting implementation of technical effects of the embodiment of the present application. The units are divided based on logic functions, and in practical application, the functions of one unit can be realized by a plurality of units, or the functions of a plurality of units can be realized by one unit. In other embodiments of the present application, the social interaction apparatus may also include other units, and in practical applications, these functions may also be implemented by being assisted by other units, and may be implemented by cooperation of multiple units. According to another embodiment of the present application, the social interaction apparatus as shown in fig. 16 may be constructed by running a computer program (including program codes) capable of executing steps involved in the corresponding method as shown in fig. 3 on a general-purpose computing device such as a computer including a Central Processing Unit (CPU), a random access storage medium (RAM), a read-only storage medium (ROM), and the like, and a memory element, and the social interaction method of the embodiment of the present application may be implemented. The computer program may be recorded on a computer-readable recording medium, for example, and loaded and executed in the above-described computing apparatus via the computer-readable recording medium.

In this embodiment, the display unit 1601 may display a social interaction interface, and the processing unit 1502 may be configured to display the virtual object at a target position of the target image area on the environment image by analyzing whether the environment image includes the target image area or not, and when the environment image includes the target image area. The virtual information is combined with the AR technology, so that the process of superposing the entity object corresponding to the virtual object on the platform object in the real environment can be simulated, the virtual object can be displayed in a target image area which is contained in the environment image and used for bearing the virtual reality object, the sense of reality of the virtual information displayed in the environment image is improved, the display mode of the virtual object is enriched, the interactivity between a target user and the virtual object is improved, and the visual immersion of the user is enhanced.

Fig. 17 shows a schematic structural diagram of an electronic device according to an exemplary embodiment of the present application. Referring to fig. 17, the electronic device includes a storage 1701 and a processor 1702, and in this embodiment, the electronic device further includes a network interface 1703 and a user interface 1704. The electronic device may be a smart phone, a tablet computer, a smart wearable device, etc., and may access the internet through the network interface 1703 to communicate and interact with a server and other electronic devices. The user interface 1704 may include a touch display screen or the like, and may be capable of receiving user operations, and may display various interfaces to the user so as to receive user operations.

The storage 1701 may include a volatile memory (volatile memory), such as a random-access memory (RAM); the storage device 1701 may also include a non-volatile memory (non-volatile memory), such as a flash memory (flash memory), a solid-state drive (SSD), or the like; the memory device 1701 may also include a combination of the above-described types of memories.

The processor 1702 may be a Central Processing Unit (CPU). The processor 1702 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a Programmable Logic Device (PLD), or the like. The PLD may be a field-programmable gate array (FPGA), a General Array Logic (GAL), or the like.

The storage device 1701 in the embodiment of the present application stores a computer program, and the processor 1702 calls the computer program in the storage device, and when the computer program is executed, the processor 1702 may be configured to implement the method described above, such as the embodiments corresponding to fig. 3 and fig. 10.

In one embodiment, the electronic device may correspond to a terminal or a server; the storage device 1701 stores a computer program; the computer program is loaded and executed by the processor 1702 to implement the corresponding steps in the above embodiments of the social interaction method; specifically, the processor 1702 is configured to perform the following steps:

determining a connected image area on the environment image according to the normal vector information of each target point, wherein the direction corresponding to the normal vector information corresponding to each target point included in the connected image area and the target direction meet the consistency condition;

determining a target reference area in the connected image area;

In one implementation, when the processor 1702 determines a connected image region on the environment image according to the normal vector information of each target point, the following steps are specifically performed:

screening N target points which meet the consistency condition with the target direction from the environment image, wherein N is an integer larger than zero;

In one implementation, the processor 1702 is further configured to perform the following steps:

determining a virtual click position in the connected image area;

generating an analog click signal according to the virtual click position, and outputting the analog click signal so as to trigger AR positioning initialization processing through the analog click signal;

wherein, AR location initialization processing includes: initializing camera position information and model position information; the target reference area includes: a region within the target range centered on the virtual click position in the environment image.

In one implementation, the processor 1702 is specifically configured to perform the following steps when performing rendering processing on a virtual object according to a target position of a target reference region in an environment image:

In one implementation, the processor 1702 is specifically configured to perform the following steps when performing rendering processing on a virtual object according to the camera position information:

calling an image rendering process to render the virtual object;

acquiring material information of a plurality of candidate virtual objects;

In one implementation, when the processor 1702 generates the target virtual object based on the object material selected by the selection operation and the acquired user interaction information of the target user, specifically, the following steps are performed:

and responding to the user operation of the sharing control, acquiring the multimedia data obtained by the screen recording processing, and sharing the multimedia data.

In another embodiment, the electronic device may correspond to a terminal or a server; the storage device 1701 stores a computer program; the computer program is loaded and executed by the processor 1702 to implement the corresponding steps in the above embodiments of the social interaction method; specifically, the processor 1702 is configured to perform the following steps:

displaying a virtual object at a target location of an environment image;

and in response to the triggering operation of the prompting element, displaying a first prompting message on the image display interface, wherein the first prompting message is used for sending out a prompt about determining the target image area from the environment image.

In one implementation, the image display interface comprises a folding option; the processor 1702 is further configured to perform the following steps:

In one implementation, when the processor 1702 performs the update display processing on the currently displayed virtual object based on the reference virtual object selected by the selection operation in the environment image, the following steps are specifically performed:

In one implementation, the image display interface includes a recording option; the processor 1702 is further configured to perform the steps of:

and displaying a prompt element about the target virtual object in a social interaction interface with the sharing object.

the target control operation is determined by identifying environment interaction data, wherein the environment interaction data comprises: and one or two of the environmental sound data acquired by the microphone and the environmental action data shot by the calling camera are combined.

Embodiments of the present application also provide a computer-readable storage medium (Memory), which is a Memory device in an electronic device and is used for storing programs and data. It is understood that the computer readable storage medium herein may include both a built-in storage medium in the electronic device and, of course, an extended storage medium supported by the electronic device. The computer readable storage medium provides a memory space that stores a processing system of the electronic device. Also stored in this memory space are computer programs (including program code) adapted to be loaded and executed by processor 1702. It should be noted that the computer-readable storage medium may be a high-speed RAM memory, or may be a non-volatile memory (non-volatile memory), such as at least one disk memory; and optionally at least one computer readable storage medium located remotely from the aforementioned processor.

According to an aspect of the application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, so that the electronic device executes the social interaction method provided in the above-mentioned various alternatives.

Those of ordinary skill in the art would appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the technical solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.

The above disclosure is only for the purpose of illustrating the preferred embodiments of the present invention, and it is therefore to be understood that the invention is not limited by the scope of the appended claims.

Claims

1. A method of social interaction, comprising:

determining a target reference area in the connected image area;

rendering a virtual object according to the target position of the target reference area in the environment image so as to display the virtual object on the environment image.

2. The method as claimed in claim 1, wherein said determining a connected image area on the environment image according to the normal vector information of the target points comprises:

3. The method of claim 1, wherein the method further comprises:

determining a virtual click position in the connected image area;

wherein the AR positioning initialization processing comprises: initializing camera position information and model position information; the target reference region includes: a region within a target range centered on the virtual click position in the environment image.

4. The method of claim 1, wherein the rendering the virtual object according to the target position of the target reference region in the environment image comprises:

determining a feature point set in the target reference region, wherein the feature point set comprises a plurality of image feature points;

obtaining camera position information according to a tracking processing result, wherein the camera position information is used for indicating any one or two of position change and orientation change of a camera in a world coordinate system in the process from shooting of the environment image to the environment updating image;

rendering the virtual object according to the camera position information so as to display the virtual object in the environment updating image.

5. The method of claim 4, wherein said rendering the virtual object according to the camera position information comprises:

calling an image rendering process to render the virtual object;

and the image rendering process is used for rendering the virtual object according to the initialized model position information, the camera position information and the projection matrix.

6. The method of any one of claims 1-5, further comprising:

acquiring material information of a plurality of candidate virtual objects;

according to the material information of each candidate virtual object, acquiring an object material of each candidate virtual object so as to display the object material of each candidate virtual object on a candidate object interface;

receiving selection operation of object materials in the candidate object interface;

7. The method of claim 6, wherein generating the target virtual object based on the object material selected by the selection operation and the acquired user interaction information of the target user comprises:

acquiring user interaction information of the target user, wherein the user interaction information comprises: any one or a combination of the user identification of the target user and the edit information for the target virtual object, the edit information including: generating one or more items of voice information, animation information and text information for the target user;

and generating the target virtual object based on the user interaction information and the object material selected by the selection operation.

8. The method of any one of claims 1-5, further comprising:

9. The method of claim 1, wherein the method further comprises:

and if the identification result comprises a target control operation, responding to the target control operation to perform dynamic adjustment processing on the virtual object.

10. A social interaction method, comprising:

displaying a virtual object at a target location of the environmental image;

the target position corresponds to a target reference area on the environment image, and a target image area included in the target reference area corresponds to a plane object in the current environment.

11. The method of claim 10, wherein the method further comprises:

and responding to the triggering operation of the prompting element, and displaying a first prompting message on an image display interface, wherein the first prompting message is used for sending out a prompt about determining a target image area from the environment image.

12. The method of claim 10, wherein the image display interface includes a fold option; the method further comprises the following steps:

in response to the selection operation on the object display window, updating and displaying the currently displayed virtual object in the environment image based on the reference virtual object selected by the selection operation.

13. The method according to claim 12, wherein the updating, in the environment image, the display processing of the virtual object currently displayed based on the reference virtual object selected by the selection operation includes:

or, in the environment image, the reference virtual object selected by the selection operation is displayed while the virtual object is currently displayed.

14. The method of claim 12, wherein the method further comprises:

if the environment image does not contain a reference image area for displaying the reference virtual object selected by the selection operation, outputting a second prompt message, wherein the second prompt message is used for prompting the adjustment of the shooting direction;

and displaying the reference virtual object according to the reference image area.

15. The method of claim 10, wherein the image display interface includes a recording option; the method further comprises the following steps:

responding to the triggering operation of the recording option, performing screen recording processing, and displaying recording progress information at the display position of the recording option, wherein the recording progress information is used for expressing the time length for recording multimedia data;

responding to the triggering operation of the sharing control, and sharing the multimedia data obtained by screen recording.

16. The method of claim 10, wherein the method further comprises:

responding to the selection operation on the candidate object interface, and outputting an object detail interface which contains object information of a target virtual object selected from the candidate object interface according to the selection operation and a confirmation option;

and displaying a prompt element related to the target virtual object in a social interaction interface between the target virtual object and the sharing object.

17. The method of claim 10, wherein the method further comprises:

displaying prompt information about target control operation on the image display interface;

18. A social interaction device, comprising:

the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring an environment image and acquiring normal vector information corresponding to each target point on the environment image;

the processing unit is further used for determining a target reference area in the connected image area;

the processing unit is further configured to perform rendering processing on a virtual object according to a target position of the target reference area in the environment image, so as to display the virtual object on the environment image.

19. A social interaction device, comprising:

the display unit is used for displaying a social interaction interface, and the social interaction interface comprises a prompt element;

the processing unit is used for responding to the triggering operation of the prompt element and displaying the acquired environment image on an image display interface;

the processing unit to display a virtual object at a target location of the environment image;

20. An electronic device, comprising: a storage device and a processor;

the storage device stores a computer program therein;

a processor for executing a computer program stored in the storage device to implement the social interaction method according to any one of claims 1 to 9, or to implement the social interaction method according to any one of claims 10 to 17.