WO2019119815A1

WO2019119815A1 - Ar service processing method, apparatus, device and computer readable storage medium

Info

Publication number: WO2019119815A1
Application number: PCT/CN2018/098145
Authority: WO
Inventors: 查俊莉
Original assignee: 广州市动景计算机科技有限公司
Priority date: 2017-12-18
Filing date: 2018-08-01
Publication date: 2019-06-27
Also published as: CN108134945B; CN108134945A

Abstract

The embodiments of the present application provide an AR service processing method, an apparatus, a device and a computer readable storage medium. The AR service processing method comprises: acquiring, according to a provided identifier, a first image acquired by a first terminal and a second image acquired by a second terminal; acquiring a target from the second image, and drawing the target into the first image; and displaying, in the first terminal, the drawn first image. In the embodiments of the present invention, a user may provide an appropriate AR service usage scenario according to his/her own preferences, effectively solving the problem of the existing live broadcast picture being monotonous, not being able to satisfy users' personalized requirements on live broadcast.

Description

AR service processing method, device, device and computer readable storage medium

Technical field

The embodiments of the present invention relate to the field of computer technologies, and in particular, to an AR service processing method, apparatus, device, and computer readable storage medium.

Background technique

AR (Augmented Reality) technology is a new technology that integrates real world information and virtual world information "seamlessly", which makes it difficult to experience information within a certain period of time in the real world (eg, Visual information, sound information, etc.), superimposed on the real information after the simulation, the real environment and the virtual object are superimposed on the same picture or in the same time in real time.

With the development of AR technology, AR technology has been applied to all aspects of live broadcast, for example, sending gifts to the anchor through AR technology, red packets to fans on the anchor, and so on. However, the application of the existing AR technology in the live broadcast only enhances the interaction between the anchor and the fans, and improves the interest of the live broadcast, but the interaction screen between the anchor and the fans is relatively simple, and each live broadcast is a monolithic anchor interface. Can not meet the personalized needs of the live broadcast.

Summary of the invention

In view of this, the embodiments of the present invention provide an AR service processing method, apparatus, device, and computer readable storage medium, so as to solve the problem that the existing live broadcast picture is single and cannot meet the user's personalized demand for live broadcast.

According to a first aspect of the present invention, an AR service processing method is provided, including: acquiring a first image collected by a first terminal and a second image collected by a second terminal according to a setting identifier; Obtaining a target object, drawing the target object into the first image; displaying the drawn first image in the first terminal.

According to a second aspect of the embodiments of the present invention, an AR service processing apparatus is provided, including: a first acquiring module, configured to acquire, according to a setting identifier, a first image collected by a first terminal and a second image collected by a second terminal a drawing module, configured to acquire a target object from the second image, and draw the target object into the first image; a display module, configured to display the drawn image in the first terminal An image.

According to a third aspect of the embodiments of the present invention, an augmented reality AR service processing device is provided, including:

Memory

Processor;

Computer program;

Wherein the computer program is stored in the memory and configured to be executed by the processor to implement the AR service processing method as described in the first aspect.

According to a fourth aspect of the embodiments of the present invention, there is provided a computer readable storage medium having stored thereon a computer program,

The computer program is executed by a processor to implement the AR service processing method as described in the first aspect.

According to the solution provided by the embodiment of the present invention, when the AR service is implemented, if the AR service is implemented in the live broadcast scenario, the image collected by the first terminal (such as the fan terminal) in real time (ie, the first image) is obtained according to the setting identifier. The second terminal (such as the anchor end) collects the image in real time (ie, the second image); and further, extracts the target object (such as the image of the anchor) from the image acquired by the second terminal, and draws it in the image of the first terminal. Based on the solution, the first terminal may set a private scene (such as a fan's living room scene) that implements the AR service, and project the target object in the private scene, and the private scene may be visible only to the first terminal. Therefore, the user of the first terminal can set an appropriate AR service usage scenario according to its own preference, which effectively solves the problem that the existing live broadcast screen is single and cannot meet the user's personalized demand for the live broadcast.

In addition, the setting identifier is related to the AR service processing of the first terminal and the AR service processing of the second terminal. Therefore, when the setting identifier is triggered, the image collected by the first terminal and the second terminal may be acquired. image. By setting the identifier, the terminal information can be quickly obtained and determined, and the AR service speed can be improved and the user experience can be improved.

DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below. Obviously, the drawings in the following description are only It is a few embodiments described in the embodiments of the present invention, and other drawings can be obtained according to the drawings for those skilled in the art.

1 is a flow chart showing the steps of an AR service processing method according to Embodiment 1 of the present invention;

2 is a flow chart showing the steps of an AR service processing method according to Embodiment 2 of the present invention;

3 is a structural block diagram of an AR service processing apparatus according to Embodiment 3 of the present invention;

4 is a structural block diagram of an AR service processing apparatus according to Embodiment 4 of the present invention;

FIG. 5 is a schematic structural diagram of a terminal according to Embodiment 5 of the present invention; FIG.

FIG. 6 is a structural diagram of an augmented reality AR service processing apparatus according to an exemplary embodiment of the present invention.

Detailed ways

For a better understanding of the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. The embodiments are only a part of the embodiments of the embodiments of the invention, and not all of the embodiments. All other embodiments obtained by those skilled in the art should be within the scope of protection of the embodiments of the present invention based on the embodiments in the embodiments of the present invention.

The specific implementation of the embodiments of the present invention is further described below in conjunction with the accompanying drawings.

Embodiment 1

Referring to FIG. 1, a flow chart of steps of an AR service processing method according to a first embodiment of the present invention is shown.

The AR service processing method in this embodiment includes the following steps:

Step S102: Acquire a first image collected by the first terminal and a second image collected by the second terminal according to the setting identifier.

The setting identifier is related to the AR service processing of the first terminal and the AR service processing of the second terminal. When the setting identifier is triggered, the image acquired by the first terminal is acquired, and the image collected by the second terminal is acquired. The specific implementation form of the setting identifier can be appropriately set by a person skilled in the art according to actual needs, and the embodiment of the present invention does not limit this. For example, the setting identifier can be a two-dimensional code.

The first image collected by the first terminal and the second image collected by the second terminal are real-time images, and the first terminal and the second terminal may be mobile terminals at the same time, or may be non-mobile terminals at the same time, or one mobile terminal. One is a non-mobile terminal.

In a live broadcast scenario, the first terminal may be a terminal of the fan party, and the second terminal may be a terminal of the host party. The first terminal collects the scene image of the fan side in real time, and the second terminal collects the scene image of the anchor party in real time.

Step S104: Acquire a target object from the second image, and draw the target object into the first image.

In the embodiment of the present invention, the target object may be an image object corresponding to the subject having the action and the emotional expression, for example, the anchor image of the anchor in the live broadcast in the video.

The specific implementation of obtaining the target object from the second image may be implemented by any suitable means according to actual conditions by a person skilled in the art, for example, by means of a map, or by means of feature extraction, and the like.

After the target object is acquired, the target object may be drawn into the first image. For example, the acquired two-dimensional target object may be directly superimposed into the first image, or the three-dimensional reconstruction may be used to create a three-dimensional image for the target object. After the model, the 3D target object is drawn into the first image. The specific drawing mode can be implemented by any suitable method according to the actual needs of the person skilled in the art, for example, the OpenGL mode, etc., which is not limited by the embodiment of the present invention.

For example, in the live broadcast, the image of the anchor is extracted from the anchor screen and then drawn into the scene image of the fan side to realize the personalized live broadcast of the anchor in the scene set by the fan side; or, according to the image extracted from the anchor screen. The image of the anchor image (such as feature information), generate a three-dimensional model of the anchor, and then draw the three-dimensional model of the anchor into the scene image of the fan side, as if the anchor image is projected to the setting scene of the fan home, such as the living room, thereby Realize the personalized live broadcast of the fans.

Step S106: Display the first image after drawing in the first terminal.

It should be noted that, if there is no further setting, the first image after the default drawing will be displayed only in the first terminal, and the other terminals including the second terminal are not displayed, so as to implement the private live broadcast scene of the first terminal user. . However, the first terminal may also share the first image with the second terminal in real time, but the terminal other than the first terminal and the second terminal will not be displayed, and the first terminal user is also implemented. Live scene.

For example, in a live scene, the image of the anchor is projected into the fan's living room, and the anchor can only be seen live in the living room through the first terminal of the fan party, or only the fan party and the anchor party see the anchor in the fan home. Live in the living room. Thereby, the user experience is further improved.

In this embodiment, when the AR service is implemented, if the AR service is implemented in the live broadcast scenario, the image collected by the first terminal in real time (that is, the first image) and the image collected by the second terminal in real time are obtained according to the setting identifier (ie, Second image); further, the target object is extracted from the image acquired by the second terminal and drawn in the image of the first terminal. Based on the solution, the first terminal may set a private scenario for implementing the AR service, and project the target object in the private scenario, and the private scenario may be visible only to the first terminal, or may be visible only to the first terminal and the second terminal. . Therefore, the user of the first terminal can set an appropriate AR service usage scenario according to its own preference, which effectively solves the problem that the existing live broadcast screen is single and cannot meet the user's personalized demand for the live broadcast.

The AR service processing method in this embodiment may be performed by any suitable terminal device having data processing capability, including but not limited to: a mobile terminal, such as a tablet computer, a mobile phone, a desktop computer, or the like.

Embodiment 2

Referring to FIG. 2, a flow chart of steps of an AR service processing method according to Embodiment 2 of the present invention is shown.

Step S202: Identify the setting identifier.

The setting identifier may be generated according to the user identifier of the user of the first terminal and the user identifier of the user of the second terminal. Optionally, the setting identifier may be a two-dimensional code. However, it is not limited to the form of the two-dimensional code, and other suitable forms are also applicable.

By identifying the setting identifier, the first terminal and the second terminal can perform corresponding data access and processing to implement the AR service according to the data in the two terminals.

For example, in the live broadcast, each fan can have its own unique QR code (the set identifier), generated according to the ID of the fan and the anchor, and the exclusive QR code enters the entrance of the live room set by the fan. . Each fan logs in to the live stream, has the fan's own platform ID information, and each anchor also has platform ID information. When the fan clicks into the live room belonging to the fan for the first time, the unique QR code of the fan entering the live room is generated by the QRID generation algorithm through the platform ID information of the fan and the anchor, and the exclusive two is generated. The dimension code is saved to the playlist. Fans enter the live room set by the fans themselves through the exclusive QR code entry. When a fan does not enter the set live broadcast room for the first time, it can directly enter through the generated unique QR code. All the information in the live broadcast room is the private information of the saved fan.

Step S204: Receive an AR scene setting instruction, collect a scene image according to the setting instruction, and/or set a virtual object; store the scene image and/or the set virtual object corresponding to the setting identifier.

When the user of the first terminal uses the AR service for the first time, it can set its own dedicated scene and/or virtual object to generate a usage scenario of the AR service that conforms to its own preference. Further, the usage scenario may be stored corresponding to the setting identifier, so that the usage scenario is directly used when the AR service is performed again.

For example, in a live broadcast scenario, when a fan enters a set live broadcast room for the first time through a dedicated QR code, the live broadcast environment of the anchor must first be set. Through AR technology, the anchor can be projected in a corner of the fan's own home, and the name of the live broadcast scene, that is, the name of the set live broadcast room. After the setup is complete, the anchor's image will appear in the fan home to complete the live broadcast. The fan side terminal will save the live broadcast scene settings.

In the normal live broadcast, fans can arrange a warm small room for the anchor in the live scene set by themselves. For example, virtual dolls, virtual puppies, virtual Christmas hats, virtual cookware, etc. are added through AR technology. Through these settings, the fan and the anchor's personalized scene interaction, such as: the anchor to help fans to walk the dog, the anchor to help fans cook.

Each time the fan exits from the set live room, the specific settings (such as the ornament image and location) of the live room are saved as templates. Among them, the arrangement of the set virtual objects (such as decorations) and the usage scenarios of the AR business (such as the fan's living room) are separately set. If the fan changes the usage scene, you can choose to restore the virtual object settings such as the decoration layout, or you can choose to abandon the previous virtual object settings such as the decoration layout.

By using the scene settings, each fan has its own live broadcast scene, and can use AR technology to dynamically decorate and save the anchor image.

It should be noted that this step is an optional step. It is executed when the usage scenario of the AR service needs to be set. After the usage scenario setting is completed, if the reset command is not received, it will be used.

Step S206: triggering the first terminal to collect the scene image according to the setting identifier; determining whether the collected scene image is consistent with the stored scene image; if yes, executing step S208; if not, performing step S210.

When the user of the first terminal sets the usage scenario of the AR service, each time the AR service is used, the setting identifier is recognized and then the usage scenario is entered, the image collection of the usage scene is performed, and the collected scene image is stored before. The scene images are compared to determine whether the scenes are the same.

For example, the first terminal does not enter the set live broadcast room for the first time, and the first terminal actively prompts the matching scene to perform live broadcast according to the scene name. If you want to replace the live scene, you can discard the match and reset the usage scene. After that, you can enter the normal live broadcast.

The image of the captured scene is consistent with the stored scene image by any suitable image similarity matching algorithm, such as SIFT (Scale-invariant feature transform), optical flow method, and the like.

Step S208: If the collected scene image is consistent with the stored scene image, the first terminal is instructed to continue to collect the scene image, and the captured scene image is used as the first image, and the second image collected by the second terminal is acquired. Then, step S212 is performed.

The first image collected by the first terminal is a real-time image of the scene where the first terminal is located, and the second image collected by the second terminal is a real-time image of the scene where the second terminal is located.

Generally, the first video stream collected by the first terminal and the second video stream collected by the second terminal in real time are obtained according to the setting identifier; the first image is obtained from the first video stream, and the second video stream is obtained from the second video stream. Second image.

For the terminal device, after the image acquisition device such as the camera is activated, the image is continuously acquired and transmitted to the target end, that is, the video stream. That is, the video stream is composed of consecutive images, and therefore, each frame of the video stream can be acquired and processed, and of course, it can be acquired and processed separately.

Step S210: Send a prompt message to the first terminal to prompt whether to perform AR scene resetting; if yes, go to step S204; if no, go to step S212.

For example, if the first terminal does not enter the set live broadcast room for the first time, the first terminal will actively match the usage scenario according to the scene name. If the match is unsuccessful, the prompt message is used to access whether the fan replaces the live broadcast scenario. If you want to replace the live scene, reset the live scene. After that, you can enter the normal live broadcast. If you do not change the live scene, you can prompt an error.

Step S212: Acquire a target object from the second image, and draw the target object into the first image.

After the target object is acquired, the two-dimensional image of the target object may be drawn into the first image, or the three-dimensional image of the target object may be drawn into the first image after the target object is three-dimensionally reconstructed.

It should be noted that, if the first terminal performs the usage scenario setting of the AR service and the virtual object is set in the usage scenario, before acquiring the target object from the second image, the information of the set virtual object needs to be acquired; According to the information of the virtual object, the virtual object is acquired and the virtual object is drawn onto the first image. Based on this, when the target object is acquired from the second image, and the target object is drawn into the first image, the target object can be acquired from the second image, and the target object is drawn into the first image in which the virtual object is drawn.

Optionally, when the target object is acquired from the second image, in a feasible manner, the target image may be detected by the first terminal on the second image, and the target object is acquired according to the detection result. The method for detecting the target object can be implemented by any suitable method according to the actual situation in the art, which is not limited by the embodiment of the present invention. In another possible manner, the first terminal may acquire data of the target object from the server, and acquire the target object according to the data of the received target object. That is, in this manner, the detection and extraction of the target object is completed by the server, and the first terminal obtains the detected and extracted data from the server to obtain the target object.

After the target object is acquired, the target object is drawn into the first image. As described above, if the fan of the first terminal further sets the virtual object in the scene through the scene setting, both the virtual object and the target object in the scene are Need to draw into the first image.

In a specific manner, in a feasible manner, the display position of the target object may be determined according to the position information of the virtual object in the first image; the target object is obtained from the second image, and the target object is drawn into the first image. Placement. In this case, the virtual object in the first image needs to be detected in position to obtain the location information thereof, and then the display position of the target object is determined according to the location information. For example, if the virtual object in the first image is a virtual puppy, after determining the position of the virtual puppy in the first image, it may be determined that the image of the target object, such as the anchor, is drawn at any appropriate position around the virtual puppy. and many more.

In order to further improve the user experience, the triggering operation of the virtual object in the first image may be received, and an operation instruction is sent to the second terminal according to the triggering operation, and the user corresponding to the target object is instructed to perform the operation instruction. Corresponding operation. For example, taking the virtual object as a virtual puppy as an example, assuming that the virtual puppy is located on the sofa of the fan's living room in the current image frame, when the fan clicks on the virtual puppy, the first terminal sends the virtual puppy to the second terminal. The triggered instruction, the instruction may carry operation information indicating that the anchor performs corresponding operations, such as sitting and stroking the puppy; the instruction may also notify the anchor only that the virtual puppy is triggered, and the second terminal where the anchor is located may The server or the operation corresponding to the virtual puppy stored locally from the second terminal selects an operation for the virtual puppy. After receiving the image frame sent by the second terminal, the first terminal of the fan side extracts the anchor image and detects it, thereby determining the display position of the anchor image, and rendering the anchor image to the display in the current image. At the location, the anchor is sitting on the sofa in the fan's living room and stroking the image of the virtual puppy. However, it is not limited to the manner of sending an operation instruction to the second terminal according to the triggering operation. In actual use, the fan may also invite the anchor to perform corresponding interactive operations by voice or text.

Optionally, the target object may also be acquired from the second image, and the target object is subjected to motion and/or expression detection; the virtual object is updated according to the detection result; and the target object and the updated virtual object are drawn into the first image. The update to the virtual object may include at least one of the following: an image of the virtual object, a placement of the virtual object, and a presentation level of the virtual object. By performing motion and/or expression detection on the target object, the virtual object is adjusted, so that the interaction between the target object and the virtual object is more diverse, thereby further improving the user experience.

For example, the anchor is still operated by the anchor, and the second terminal of the anchor party collects the image of the action of the anchor to perform the sitting and touching, and transmits the image to the first terminal of the fan side; A terminal extracts the image of the anchor from the image and detects the motion and expression of the anchor in the image; and detects that the motion of the anchor is a stroke motion, and the expression is a smile, correspondingly, the image of the virtual puppy is lying down Updated to squat and look up at the anchor's image.

For another example, if the action of the anchor is changed from sitting to standing, after detecting the action, the virtual puppy is updated from sitting on the sofa to jump to the front of the sofa, that is, The image of the virtual puppy is updated from the squat image to the standing image, and the virtual puppy's display position is updated from the corresponding position on the original sofa to the position in front of the sofa.

For another example, if the action of the anchor changes from standing to moving toward the virtual puppy, after detecting the action, the display level of the virtual puppy is updated from the same level as the anchor image to the image below the anchor image. At the level, the part in which the virtual puppy is blocked by the anchor image is not displayed, and the image that is still not displayed in the unoccluded part is closer to the actual living scene.

As mentioned earlier, when a continuous image frame is composed of a video stream, an interactive AR is generated for the anchor to help the dog.

In addition, optionally, the first terminal may reconstruct the three-dimensional image of the anchor by using a dynamic character three-dimensional reconstruction technology, and then the three-dimensional image is drawn into the first image. Three-dimensional reconstruction is a mathematical process and computer technology that uses three-dimensional images to recover three-dimensional information (shapes, etc.) of objects, including image acquisition, preprocessing, point cloud stitching, and feature analysis. Through the three-dimensional reconstruction, the same live broadcast situation as the anchor in the live broadcast scene can be seen on the first terminal of the fan side.

For example, the second terminal of the anchor party collects the real-time video image frame of the anchor, and the captured video image frame is extracted by video compression technology to extract key information and spread efficiently in the network, so that the information transmitted in the network is rarely And it's fast. When the transmission frame rate reaches 60fps or more, the network transmission delay is not perceived by the user, and the cardon and delay are not felt, and the normal live state of the anchor can be seen in real time.

In addition, the interaction between the anchor and the fans can also be initiated by the anchor. The anchor prompts the fans to do the action of walking the dog. The fans can add the virtual puppy's 3D rendering animation to the specified location through the AR in the scene interface to achieve the anchor. Interactive operation.

Further, in the live broadcast process, the real-time interactive scene of the AR may be captured by the first terminal screenshot for sharing.

In order to make the motion and/or the expression of the anchor more in place, the real-time image of the first terminal may also be displayed in the live interface of the second terminal of the anchor. For example, the real-time of the first terminal may be displayed in the live interface of the second terminal. The small window of the image may also be a full-screen display or the like, which is not limited in the embodiment of the present invention.

It can be seen that when the solution of the embodiment is applied to the live broadcast scenario, the live broadcast portal of the fan party is a dedicated two-dimensional code, and other people cannot enter, and the privacy information of the fan is protected; the live broadcast room set by the fan is exclusive (other fans) Only the live broadcast screen of the anchor can be shared; the live broadcast room set by the fans can be used to arbitrarily distribute the anchor image through AR technology, and can be dynamically decorated and permanently saved (for example, each time the anchor is placed in a corner of the fan home); The live room set by the fans has dynamic variability, and the scene can be flexibly set as needed, and will be saved as a template for fans to use again for more flexibility. As a result, the individual needs of the live broadcast of the fans are greatly satisfied, and the interaction is better.

Embodiment 3

Referring to FIG. 3, a block diagram of a structure of an AR service processing apparatus according to Embodiment 3 of the present invention is shown.

The AR service processing apparatus of this embodiment includes: a first obtaining module 302, configured to acquire a first image collected by the first terminal and a second image collected by the second terminal according to the setting identifier; and a drawing module 304, configured to be used by the second The target object is obtained in the image, and the target object is drawn into the first image; the display module 306 is configured to display the drawn first image in the first terminal.

The AR service processing apparatus in this embodiment is used to implement the corresponding AR service processing method in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, and details are not described herein again.

Embodiment 4

Referring to FIG. 4, a block diagram of a structure of an AR service processing apparatus according to Embodiment 4 of the present invention is shown.

The AR service processing apparatus of this embodiment includes: a first acquiring module 402, configured to acquire a first image collected by the first terminal and a second image collected by the second terminal according to the setting identifier; and a drawing module 404, configured to be used from the second The target object is obtained in the image, and the target object is drawn into the first image. The display module 406 is configured to display the first image after the drawing in the first terminal.

Optionally, the setting identifier is generated according to a user identifier of a user of the first terminal and a user identifier of a user of the second terminal.

Optionally, the setting identifier is a two-dimensional code.

Optionally, the AR service processing apparatus of this embodiment further includes: a second obtaining module 408, configured to acquire information of the set virtual object before the drawing module 404 acquires the target object from the second image; and according to the information of the virtual object Obtaining a virtual object and drawing the virtual object onto the first image; the drawing module 404 is configured to acquire the target object from the second image and draw the target object into the first image in which the virtual object is drawn.

Optionally, the drawing module 404 is configured to determine a display position of the target object according to the position information of the virtual object in the first image; acquire the target object from the second image, and draw the target object to the display position in the first image .

Optionally, the AR service processing apparatus of the embodiment further includes: an instruction module 410, configured to receive a trigger operation on the virtual object, send an operation instruction to the second terminal according to the triggering operation, and indicate that the user corresponding to the target object performs the operation instruction Corresponding operation.

Optionally, the drawing module 404 includes: a detecting module 4042, configured to acquire a target object from the second image, and perform an action and/or an expression detection on the target object; and an updating module 4044, configured to update the virtual object according to the detection result; The post rendering module 4046 is configured to draw the target object and the updated virtual object into the first image.

Optionally, the updating module 4044 is configured to perform at least one of the following updates on the virtual object according to the detection result: an image of the virtual object, a display position of the virtual object, and a display level of the virtual object.

Optionally, the AR service processing apparatus of this embodiment further includes: a setting module 412, configured to: before the first acquiring module 402 acquires the first image collected by the first terminal and the second image collected by the second terminal according to the setting identifier Receiving an AR scene setting instruction, collecting a scene image and/or setting a virtual object according to the setting instruction, and storing the scene image and/or the set virtual object corresponding to the setting identifier.

Optionally, the first acquiring module 402 is configured to trigger the first terminal to collect the scene image according to the setting identifier, and determine whether the collected scene image is consistent with the stored scene image; if they are consistent, the first terminal is instructed to continue collecting the scene image. And acquiring the captured scene image as the first image, and acquiring the second image collected by the second terminal.

Optionally, the drawing module 404 is configured to perform target object detection on the second image, acquire the target object according to the detection result, or acquire the target object according to the received data of the target object; and draw the target object into the first image.

Optionally, the first obtaining module 402 is configured to acquire, according to the setting identifier, the first video stream collected by the first terminal in real time and the second video stream collected by the second terminal in real time; and acquiring the first image from the first video stream, The second image is acquired in the second video stream.

Embodiment 5

Referring to FIG. 5, a schematic structural diagram of a terminal according to Embodiment 5 of the present invention is shown. The specific implementation of the present invention does not limit the specific implementation of the terminal.

As shown in FIG. 5, the terminal may include a processor 502, a communications interface 504, a memory 506, and a communication bus 508.

among them:

Processor 502, communication interface 504, and memory 506 complete communication with one another via communication bus 508.

The communication interface 504 is configured to communicate with other terminals or servers.

The processor 502 is configured to execute the program 510, and specifically, the related steps in the foregoing AR service processing method embodiment.

In particular, program 510 can include program code, the program code including computer operating instructions.

The processor 502 may be a central processing unit CPU, or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in the terminal may be the same type of processor, such as one or more CPUs; or may be different types of processors, such as one or more CPUs and one or more ASICs.

The memory 506 is configured to store the program 510. Memory 506 may include high speed RAM memory and may also include non-volatile memory, such as at least one disk memory.

The program 510 may be specifically configured to: the processor 502: obtain the first image collected by the first terminal and the second image collected by the second terminal according to the setting identifier; acquire the target object from the second image, and draw the target object Up to the first image; the first image after the rendering is displayed in the first terminal.

In an optional implementation manner, the setting identifier is generated according to a user identifier of a user of the first terminal and a user identifier of a user of the second terminal.

In an optional implementation manner, the setting identifier is a two-dimensional code.

In an optional implementation manner, the program 510 is further configured to: when the processor 502 acquires the target object from the second image, acquire information of the set virtual object; according to the information of the virtual object, acquire the virtual object and The object is drawn onto the first image; the target object is obtained from the second image, and the target object is drawn into the first image in which the virtual object is drawn.

In an optional implementation, the program 510 is further configured to cause the processor 502 to acquire the target object from the second image and draw the target object into the first image in which the virtual object is drawn, according to the first image. The position information of the virtual object in the determination, the display position of the target object; the target object is obtained from the second image, and the target object is drawn to the placement position in the first image.

In an optional implementation, the program 510 is further configured to: the processor 502 receives a triggering operation on the virtual object, and sends an operation instruction to the second terminal according to the triggering operation, indicating that the user corresponding to the target object performs the operation instruction. Operation.

In an optional implementation, the program 510 is further configured to cause the processor 502 to acquire the target object from the second image and draw the target object into the first image in which the virtual object is drawn, from the second image. Obtaining the target object, and performing motion and/or expression detection on the target object; updating the virtual object according to the detection result; and drawing the target object and the updated virtual object into the first image.

In an optional implementation manner, the program 510 is further configured to: when the processor 502 updates the virtual object according to the detection result, perform at least one of the following updates on the virtual object: an image of the virtual object, a placement of the virtual object, and a virtual The presentation level of the object.

In an optional implementation, the program 510 is further configured to: when the processor 502 acquires the first image collected by the first terminal and the second image collected by the second terminal according to the setting identifier, the AR scenario setting instruction is received, where Acquiring a scene image and/or setting a virtual object according to the setting instruction; storing the scene image and/or the set virtual object corresponding to the setting identifier.

In an optional implementation, the program 510 is further configured to: when the processor 502 acquires the first image collected by the first terminal and the second image collected by the second terminal according to the setting identifier, according to the setting identifier Triggering the first terminal to collect the scene image; determining whether the collected scene image is consistent with the stored scene image; if they are consistent, instructing the first terminal to continue collecting the scene image, using the collected scene image as the first image, and acquiring the second terminal. The second image.

In an optional implementation manner, the program 510 is further configured to: when the processor 502 acquires the target object from the second image, perform target object detection on the second image, acquire the target object according to the detection result; or, according to the receiving The target object's data is obtained from the target object.

In an optional implementation, the program 510 is further configured to: when the processor 502 acquires the first image collected by the first terminal and the second image collected by the second terminal according to the setting identifier, acquiring the first The first video stream collected by the terminal in real time and the second video stream collected by the second terminal in real time; the first image is obtained from the first video stream, and the second image is obtained from the second video stream.

For a specific implementation of the steps in the program 510, reference may be made to the corresponding steps in the foregoing embodiment of the AR service processing method and the corresponding description in the unit, and details are not described herein. A person skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the device and the module described above may be referred to the corresponding process description in the foregoing method embodiment, and details are not described herein again.

It can be seen that the terminal in this embodiment can be used as the first terminal.

In this embodiment, when the AR service is implemented, if the AR service is implemented in the live broadcast scenario, the image (ie, the first image) and the second terminal that are collected by the first terminal (such as the fan) in real time are obtained according to the setting identifier. (such as the anchor end) the image acquired in real time (ie, the second image); further, the target object (such as the image of the anchor) is extracted from the image acquired by the second terminal and drawn in the image of the first terminal. Based on the solution, the first terminal may set a private scene (such as a fan's living room scene) that implements the AR service, and project the target object in the private scene, and the private scene may be visible only to the first terminal. Therefore, the user of the first terminal can set an appropriate AR service usage scenario according to its own preference, which effectively solves the problem that the existing live broadcast screen is single and cannot meet the user's personalized demand for the live broadcast.

As shown in FIG. 6, the augmented reality AR service processing device provided in this embodiment includes:

Memory 61;

Processor 62;

Computer program;

Wherein the computer program is stored in the memory and configured to be executed by the processor to implement any of the augmented reality AR service processing methods described above.

The embodiment further provides a computer readable storage medium on which a computer program is stored.

The computer program is executed by a processor to implement any of the augmented reality AR service processing methods described above.

It should be noted that the various components/steps described in the embodiments of the present invention may be split into more components/steps according to the needs of the implementation, or two or more components/steps or partial operations of the components/steps may be combined into one. New components/steps to achieve the objectives of embodiments of the present invention.

The above method according to an embodiment of the present invention may be implemented in hardware, firmware, or implemented as software or computer code that may be stored in a recording medium such as a CD ROM, a RAM, a floppy disk, a hard disk, or a magneto-optical disk, or implemented by The network downloads computer code originally stored in a remote recording medium or non-transitory machine readable medium and stored in a local recording medium so that the methods described herein can be stored using a general purpose computer, a dedicated processor or programmable Such software processing on a recording medium of dedicated hardware such as an ASIC or an FPGA. It will be understood that a computer, processor, microprocessor controller or programmable hardware includes storage components (eg, RAM, ROM, flash memory, etc.) that can store or receive software or computer code, when the software or computer code is The AR service processing method described herein is implemented when the processor or hardware accesses and executes. Moreover, when a general purpose computer accesses code for implementing the AR business processing method shown herein, execution of the code converts the general purpose computer into a special purpose computer for executing the AR business processing method shown herein.

Those of ordinary skill in the art will appreciate that the elements and method steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the solution. A person skilled in the art can use different methods to implement the described functions for each particular application, but such implementation should not be considered to be beyond the scope of the embodiments of the present invention.

The above embodiments are only used to illustrate the embodiments of the present invention, and are not intended to limit the embodiments of the present invention, and those skilled in the art can also make various kinds without departing from the spirit and scope of the embodiments of the present invention. Variations and modifications, therefore, all equivalent technical solutions are also within the scope of the embodiments of the present invention, and the scope of patent protection of the embodiments of the present invention should be defined by the claims.

Claims

An augmented reality AR service processing method, comprising:

Obtaining, by the setting identifier, the first image collected by the first terminal and the second image collected by the second terminal;

Obtaining a target object from the second image, and drawing the target object into the first image;

The drawn first image is displayed in the first terminal.
The method according to claim 1, wherein the setting identifier is generated according to a user identifier of a user of the first terminal and a user identifier of a user of the second terminal.
The method of claim 2 wherein said setting identifier is a two-dimensional code.
A method according to any one of claims 1 to 3, wherein

Before acquiring the target object from the second image, the method further includes: acquiring information of the set virtual object; acquiring the virtual object according to the information of the virtual object, and drawing the virtual object to the first On the image;

Obtaining the target object from the second image, and drawing the target object into the first image comprises: acquiring a target object from the second image, and drawing the target object to a drawing In the first image of the virtual object.
The method of claim 4, wherein acquiring the target object from the second image and rendering the target object into the first image in which the virtual object is drawn comprises:

Determining a display position of the target object according to location information of the virtual object in the first image;

A target object is obtained from the second image and the target object is drawn to the display location in the first image.
The method of claim 4 wherein the method further comprises:

Receiving a triggering operation on the virtual object, and sending an operation instruction to the second terminal according to the triggering operation, instructing a user corresponding to the target object to perform an operation corresponding to the operation instruction.
The method of claim 4, wherein acquiring the target object from the second image and rendering the target object into the first image in which the virtual object is drawn comprises:

Obtaining a target object from the second image, and performing an action and/or a feeling detection on the target object;

Updating the virtual object according to the detection result;

The target object and the updated virtual object are drawn into the first image.
The method of claim 7, wherein the updating the virtual object according to the detection result comprises:

Performing at least one of the following updates on the virtual object: an image of the virtual object, a display location of the virtual object, and a display hierarchy of the virtual object.
The method according to any one of claims 1-3, wherein before the acquiring the first image collected by the first terminal and the second image collected by the second terminal according to the setting identifier, the method further includes:

Receiving an AR scene setting instruction, collecting a scene image and/or setting a virtual object according to the setting instruction;

The scene image and/or the set virtual object are stored corresponding to the setting identifier.
The method of claim 9, wherein the acquiring the first image collected by the first terminal and the second image collected by the second terminal according to the setting identifier comprises:

And triggering, by the setting identifier, the first terminal to collect a scene image;

Determining whether the collected scene image is consistent with the stored scene image;

If the information is consistent, the first terminal continues to collect the scene image, and the captured scene image is used as the first image, and the second image collected by the second terminal is acquired.
The method of any one of claims 1-3, wherein the obtaining the target object from the second image comprises:

Performing target object detection on the second image, and acquiring the target object according to the detection result;

or,

The target object is acquired according to data of the received target object.
The method according to any one of claims 1-3, wherein the acquiring the first image collected by the first terminal and the second image collected by the second terminal according to the setting identifier comprises:

Obtaining, by the setting identifier, the first video stream collected by the first terminal in real time and the second video stream collected by the second terminal in real time;

Acquiring the first image from the first video stream and acquiring the second image from the second video stream.
An augmented reality AR service processing device, comprising:

a first acquiring module, configured to acquire, according to the setting identifier, the first image collected by the first terminal and the second image collected by the second terminal;

a drawing module, configured to acquire a target object from the second image, and draw the target object into the first image;

And a display module, configured to display the drawn first image in the first terminal.
The apparatus according to claim 13, wherein the setting identifier is generated according to a user identifier of a user of the first terminal and a user identifier of a user of the second terminal.
The apparatus of claim 14, wherein the setting identifier is a two-dimensional code.
A device according to any of claims 13-15, wherein

The device further includes: a second obtaining module, configured to acquire information of the set virtual object before the drawing module acquires the target object from the second image; and acquire the virtual object according to the information of the virtual object Drawing the virtual object onto the first image;

The drawing module is configured to acquire a target object from the second image, and draw the target object into a first image in which the virtual object is drawn.
The apparatus according to claim 16, wherein the drawing module is configured to determine a display position of the target object according to position information of the virtual object in the first image; from the second image A target object is obtained and the target object is drawn to the display location in the first image.
The apparatus of claim 16 wherein said apparatus further comprises:

The instruction module is configured to receive a triggering operation on the virtual object, and send an operation instruction to the second terminal according to the triggering operation, and instruct the user corresponding to the target object to perform an operation corresponding to the operation instruction.
The apparatus of claim 16 wherein said rendering module comprises:

a detecting module, configured to acquire a target object from the second image, and perform an action and/or an expression detection on the target object;

An update module, configured to update the virtual object according to the detection result;

And an update drawing module, configured to draw the target object and the updated virtual object into the first image.
The device according to claim 19, wherein the updating module is configured to perform at least one of the following updates on the virtual object according to the detection result: an image of the virtual object, a display position of the virtual object, the The presentation level of the virtual object.
The device of any of claims 13-15, wherein the device further comprises:

a setting module, configured to receive an AR scene setting instruction, and collect a scene image and/or a setting according to the setting instruction, before acquiring the first image collected by the first terminal and the second image collected by the second terminal according to the setting identifier a virtual object; storing the scene image and/or the set virtual object corresponding to the setting identifier.
The device according to claim 21, wherein the first acquiring module is configured to trigger the first terminal to collect a scene image according to the setting identifier; and determine whether the collected scene image is consistent with the stored scene image; If the image is consistent, the first terminal continues to collect the scene image, and the captured scene image is used as the first image, and the second image collected by the second terminal is acquired.
The apparatus according to any one of claims 13 to 15, wherein the drawing module is configured to perform target object detection on the second image, acquire the target object according to the detection result; or, according to the received target object Data, the target object is obtained; the target object is drawn into the first image.
The device according to any one of claims 13-15, wherein the first obtaining module is configured to acquire, according to the setting identifier, a first video stream collected by the first terminal in real time and a second video captured by the second terminal in real time. Streaming; acquiring the first image from the first video stream and acquiring the second image from the second video stream.
An augmented reality AR service processing device, comprising:

Memory

Processor;

Computer program;

Wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method of any of claims 1-12.
A computer readable storage medium having stored thereon a computer program

The computer program is executed by a processor to implement the method of any of claims 1-12.