CN113382020A

CN113382020A - Interaction control method and device, electronic equipment and computer readable storage medium

Info

Publication number: CN113382020A
Application number: CN202110767625.0A
Authority: CN
Inventors: 苏伟; 孙林; 刘磊
Original assignee: Beijing Sensetime Technology Development Co Ltd
Current assignee: Beijing Sensetime Technology Development Co Ltd
Priority date: 2021-07-07
Filing date: 2021-07-07
Publication date: 2021-09-10

Abstract

The embodiment of the disclosure discloses an interaction control method, an interaction control device, electronic equipment and a computer-readable storage medium. The method comprises the following steps: receiving interactive information provided by a real user; the interaction information indicates an interaction between the real user and the virtual object; under the condition that the interactive information meets the assistance condition, sending a remote assistance request to a real customer service terminal; the remote assistance request represents a virtual object terminal to request a real customer service terminal to provide assistance information; under the condition that a feedback message aiming at the remote assistance request is received, sending first communication data of a real user to a real customer service terminal; the first communication data is interactive request data which is provided by a real user to a virtual object; and under the condition of receiving the assistance information which is sent by the real customer service terminal and corresponds to the first communication data, adopting the assistance information to interact with the real user. Through the method and the device, the intelligence of the virtual object terminal is improved.

Description

Interaction control method and device, electronic equipment and computer readable storage medium

Technical Field

The present disclosure relates to computer technologies, and in particular, to an interaction control method and apparatus, an electronic device, and a computer-readable storage medium.

Background

The digital human service is a novel human-computer interaction system, and a real user interacts with the digital human service in computer vision, voice conversation, screen touch and other modes by displaying a virtual digital human image; the digital human service is integrated with modules such as voice recognition, semantic understanding, voice synthesis and the like, and can automatically answer questions provided by real users and carry out conversation with the real users. In the process of interacting with the real user, the digital human service mainly depends on a semantic understanding module to automatically generate answers to interact with the real user.

In the related art, digital people have defects when dealing with complex problems, and the intelligence is not high.

Disclosure of Invention

The embodiment of the disclosure provides an interaction control method and device, electronic equipment and a computer-readable storage medium, which can improve the intelligence of a terminal.

The technical scheme of the embodiment of the disclosure is realized as follows:

the embodiment of the disclosure provides an interaction control method, which includes: receiving interactive information provided by a real user; the interaction information is used for indicating the interaction between the real user and the virtual object; detecting that the interaction information meets assistance conditions, and sending a remote assistance request to a real customer service terminal; the remote assistance request representation virtual object terminal requests the real customer service terminal to provide assistance information; receiving a feedback message aiming at the remote assistance request, and sending first communication data of the real user to the real customer service terminal; the first communication data is interaction request data which is made by the real user to the virtual object; and receiving the assistance information which is sent by the real customer service terminal and corresponds to the first communication data, and adopting the assistance information to interact with the real user.

The embodiment of the present disclosure further provides an interaction control method, including: receiving a remote assistance request sent by a virtual object terminal; the remote assistance request represents that the virtual object terminal requests the real customer service terminal to provide assistance information; the assistance information is used for interaction between the virtual object and a real user; sending a request acceptance message for the remote assistance request to a server, so that the server sends a feedback message to the virtual object terminal based on the request acceptance message; receiving first communication data of the real user sent by the virtual object terminal based on the feedback message; and generating the assistance information corresponding to the first communication data and sending the assistance information to the virtual object terminal.

The embodiment of the present disclosure further provides an interaction control method, including: receiving a remote assistance request sent by a virtual object terminal, and forwarding the remote assistance request to a real customer service terminal; the remote assistance request represents that the virtual object terminal requests the real customer service terminal to provide assistance information; the assistance information is used for interaction between the virtual object and a real user; receiving a request acceptance message sent by the real customer service terminal; sending a feedback message to the virtual object terminal based on the request acceptance message; receiving first communication data of a real user sent by the virtual object terminal based on the feedback message, and forwarding the first communication data to the real customer service terminal; and receiving the assistance information which is sent by the real customer service terminal and corresponds to the first communication data, and forwarding the assistance information to the virtual object terminal.

The disclosed embodiment provides an interactive control device, including: the first receiving unit is used for receiving interactive information provided by a real user; the interaction information is used for indicating the interaction between the real user and the virtual object;

the first sending unit is used for sending a remote assistance request to a real customer service terminal when detecting that the interaction information meets the assistance condition; the remote assistance request representation virtual object terminal requests the real customer service terminal to provide assistance information; receiving a feedback message aiming at the remote assistance request, and sending first communication data of the real user to the real customer service terminal; the first communication data is interaction request data which is made by the real user to the virtual object; and the interaction unit is further used for receiving the assistance information which is sent by the real customer service terminal and corresponds to the communication data, and adopting the assistance information to interact with the real user.

The embodiment of the present disclosure further provides an interactive control device, which includes: the second receiving unit is used for receiving the remote assistance request sent by the virtual object terminal; the remote assistance request represents that the virtual object terminal requests the real customer service terminal to provide assistance information; the assistance information is used for interaction between the virtual object and a real user; a second sending unit, configured to send a request acceptance message for the remote assistance request to a server, so that the server sends a feedback message to the virtual object terminal based on the request acceptance message; the first receiving unit is further configured to receive first communication data of the real user, which is sent by the virtual object terminal based on the feedback message; an information generating unit further configured to generate the assistance information corresponding to the first communication data; the second sending unit is further configured to send the assistance information corresponding to the first communication data to the virtual object terminal.

The embodiment of the present disclosure further provides an interactive control device, which includes: the information forwarding unit is used for receiving a remote assistance request sent by the virtual object terminal and forwarding the remote assistance request to the real customer service terminal; the remote assistance request represents that the virtual object terminal requests the real customer service terminal to provide assistance information; the assistance information is used for interaction between the virtual object and a real user; a third receiving unit, configured to receive a request acceptance message sent by the real customer service terminal; a third sending unit, configured to send a feedback message to the virtual object terminal based on the request acceptance message; the information forwarding unit is further configured to receive first communication data of a real user sent by the virtual object terminal based on the feedback message, and forward the first communication data to the real customer service terminal; and receiving the assistance information which is sent by the real customer service terminal and corresponds to the first communication data, and forwarding the assistance information to the virtual object terminal.

An embodiment of the present disclosure provides an electronic device, including: a memory for storing an executable computer program; and a processor for implementing the above-described interactive control method when executing the executable computer program stored in the memory.

The embodiment of the present disclosure provides a computer-readable storage medium, which stores a computer program for causing a processor to execute the method for controlling interaction described above.

The interaction control method, the device, the electronic device and the computer readable storage medium provided by the embodiments of the present disclosure, the virtual customer service terminal receives interaction information provided by a real user and used for indicating interaction between the real user and a virtual object, sends a remote assistance request indicating that the virtual object terminal requests the real customer service terminal to provide assistance information to the real customer service terminal when it is detected that the interaction information satisfies assistance conditions, sends interaction request data provided by the real user to the virtual object to the real customer service terminal when a feedback message for the remote assistance request is received, and interacts with the real user using the assistance information when assistance information corresponding to the interaction request data sent by the real customer service terminal is received, so that a response coverage of the virtual object terminal to the interaction request data of the real user is improved, and the correctness of the response information provided for the real user, thereby increasing the functional diversity of the virtual object terminal and improving the intelligence of the virtual object terminal.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.

Fig. 1 is a schematic structural diagram of an interactive control system according to an embodiment of the present disclosure;

fig. 2 is an alternative flow chart of an interaction control method provided in the embodiment of the present disclosure;

fig. 3 is an alternative flowchart of an interaction control method provided in the embodiment of the present disclosure;

fig. 4 is an alternative flowchart of an interaction control method provided in the embodiment of the present disclosure;

fig. 5 is an alternative flowchart of an interaction control method provided in the embodiment of the present disclosure;

fig. 6 is an alternative flowchart of an interaction control method provided in the embodiment of the present disclosure;

fig. 7 is an alternative flowchart of an interaction control method provided in the embodiment of the present disclosure;

fig. 8 is an alternative flowchart of an interaction control method provided in the embodiment of the present disclosure;

fig. 9 is an alternative flowchart of an interaction control method provided in the embodiment of the present disclosure;

FIG. 10 is a schematic diagram of an exemplary landing status presentation page provided by an embodiment of the present disclosure;

FIG. 11 is a schematic diagram of an exemplary request presentation page provided by an embodiment of the present disclosure;

FIG. 12 is an alternative flowchart of an interactive control method provided by an embodiment of the present disclosure;

fig. 13 is an alternative flowchart of an interaction control method provided in the embodiment of the present disclosure;

FIG. 14 is a schematic view of an exemplary first assistance page provided by embodiments of the present disclosure;

FIG. 15 is another schematic diagram of an exemplary first assistance page provided by an embodiment of the present disclosure;

FIG. 16 is an alternative flow chart of the interaction control method provided by the embodiment of the present disclosure;

FIG. 17A is a schematic view of an exemplary second assistance page provided by embodiments of the present disclosure;

FIG. 17B is another illustration of an exemplary second assistance page provided by an embodiment of the disclosure;

FIG. 18 is yet another illustration of an exemplary second assistance page provided by an embodiment of the disclosure;

FIG. 19A is yet another illustration of an exemplary first assistance page provided by an embodiment of the disclosure;

FIG. 19B is yet another illustration of an exemplary second assistance page provided by an embodiment of the disclosure;

FIG. 20 is an alternative flow chart of the interaction control method provided by the embodiment of the present disclosure;

fig. 21 is a schematic view of an interaction flow among a virtual object terminal, a real client server terminal, and a server according to an embodiment of the present disclosure;

fig. 22 is a schematic structural diagram of an interaction control device according to an embodiment of the present disclosure;

fig. 23 is another schematic structural diagram of an interaction control device provided in an embodiment of the present disclosure;

FIG. 24 is a schematic view of another structure of an interactive control device according to an embodiment of the present disclosure;

fig. 25 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.

Detailed Description

For the purpose of making the purpose, technical solutions and advantages of the present disclosure clearer, the present disclosure will be described in further detail with reference to the accompanying drawings, the described embodiments should not be construed as limiting the present disclosure, and all other embodiments obtained by a person of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present disclosure.

The digital human service is a novel human-computer interaction system, and a real user interacts with the digital human service in computer vision, voice conversation, screen touch and other modes by displaying a virtual digital human image; the digital human service is integrated with modules such as voice recognition, semantic understanding, voice synthesis and the like, and can automatically answer questions provided by real users and carry out conversation with the real users. In the process of interaction with a real user, the digital human service mainly depends on a semantic understanding module to automatically generate answers and interacts with the real user, but the digital human service has the defects of processing complex problems, failing to answer the problems beyond the range, failing to process emergency situations and having low intelligence.

The embodiment of the disclosure provides an interaction control method, which can improve the intelligence of equipment. The interaction control method provided by the embodiment of the disclosure is applied to electronic equipment. The electronic device provided by the embodiment of the present disclosure may be implemented as various types of real user terminals (hereinafter referred to as terminals) such as AR glasses, a notebook computer, a tablet computer, a desktop computer, a set-top box, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, and a portable game device), and may also be implemented as a server.

In the following, description is made in connection with a scenario in which a digital person interacts with a real user, and interaction control between terminals is realized by server interaction between a virtual object terminal and a real customer service terminal. Illustratively, fig. 1 is an architecture diagram of an interactive control system provided by an embodiment of the present disclosure; as shown in fig. 1, the interaction control system 100 includes at least one virtual object terminal 200 (exemplarily, a terminal 200A is shown in fig. 1), at least one real customer service terminal 300 (exemplarily, a terminal 300A is shown in fig. 1), and at least one server (exemplarily, a server 400A is shown in fig. 1). As shown in fig. 1, in the case that the virtual object terminal 200A runs the digital person service, a virtual digital person 201 is displayed on a display interface; the real customer service terminal 300A has a remote control service running thereon for providing information assistance to the virtual object terminal through an operation of the real customer service (e.g., a real person using the remote control service) for the remote control service.

The virtual object terminal 200A receives interaction information sent by the real user and used for indicating interaction between the real user and the virtual digital person 201, and when the virtual object terminal 200A determines that the interaction information does not satisfy the assistance condition (for example, the virtual object terminal 200A may respond to question information of the real user, etc.), the virtual object terminal 200A generates response information corresponding to the interaction information to respond to the real user. In a case where the interaction information of the virtual object 200A satisfies the assistance condition (for example, the virtual object terminal 200A cannot respond to the problem information of the real user, or the real user request interacts with the real customer service, etc.), the virtual object terminal 200A sends a remote assistance request to the server 400A through the digital human service; the server 400A forwards the received remote assistance request to the real customer service terminal 300A. The real customer service terminal 300A sends a request acceptance message to the server 400A when receiving a request acceptance operation sent by the real customer service; upon receiving the request acceptance message, the server 400A transmits a feedback message to the virtual object terminal 200A. When receiving the feedback message for the remote assistance request, the virtual object terminal 200A sends first communication data such as text data, voice data, or video data of the real user to the server 400A through the digital human service (the first communication data is the above-mentioned interaction information, or the first communication data is data different from the interaction information, for example, the received data sent by the real user may be data after receiving the interaction information); the server 400A forwards the received first communication data to the real customer service terminal 300A. When receiving the first communication data, the real customer service terminal 300A generates assistance information such as text information, voice information, or video information corresponding to the first communication data through information such as text information, voice information, or video information received by the remote control service and input by the real customer service, and sends the assistance information to the server 400A; the server 400A forwards the received assistance information sent by the real customer service terminal 300A to the virtual object terminal 200A. The virtual object terminal 200A interacts with the real user by using the received assistance information through the virtual digital person 201.

The interaction control method provided by the embodiment of the present disclosure will be described in detail below, wherein the electronic device may be implemented as a terminal. Fig. 2 is an alternative flowchart of an interaction control method provided by the embodiment of the present disclosure, which will be described with reference to the steps shown in fig. 2.

S101, receiving interactive information provided by a real user; the interaction information is used for indicating interaction between the real user and the virtual object.

In the embodiment of the disclosure, a virtual object is displayed on the virtual object terminal, and the virtual object terminal can receive interaction information which is sent to the virtual object terminal by a real user and is used for indicating interaction between the real user and the virtual object; for example, the interaction information may be question information input by a real user on the virtual object terminal, so that the virtual object terminal can solve through the virtual object.

In some embodiments, the virtual object terminal may display a virtual object (e.g., a virtual character) on the display screen and may interact with a real user through the virtual object in the case of running the digital human service, for example, the virtual object may be a virtual customer service, a virtual service person, or the like.

In some embodiments, the virtual object terminal may collect the interaction information of the user through a camera, a microphone, and other collecting components. For example, the virtual object terminal may receive voice interaction information sent by a real user through a microphone, and may also collect video interaction information of the user by using a camera.

S102, when the interactive information meets the assistance condition, sending a remote assistance request to a real customer service terminal; the remote assistance request represents that the virtual object terminal requests the real customer service terminal to provide assistance information.

In the embodiment of the present disclosure, the virtual object terminal may send a remote assistance request to the real customer service terminal when it is determined that the interactive information satisfies the assistance condition, for example, when it is determined that the problem information provided by the user cannot be responded to autonomously, and request the real customer service terminal to provide assistance information such as text information, voice information, or video information through the remote assistance request.

In some embodiments, the interaction information satisfies the assistance condition and satisfies the assistance condition, and the interaction information satisfies the assistance condition and includes any one of the following three conditions: (1) determining that the interaction state between the real user and the user meets a preset condition based on the interaction information; (2) detecting that a preset keyword appears in second communication data contained in the interactive information; (3) and detecting the interaction information as the assistance operation.

For the case (1) above, in some embodiments, the virtual object terminal may determine that the interaction state with the real user satisfies the preset condition when detecting that the failure time of generating the state of the response information corresponding to the interaction information reaches a first preset time.

Here, during the interaction between the virtual object and the real user through the virtual object, the virtual object terminal may detect an interaction state between the virtual object and the real user in real time, for example, whether normal interaction is performed or whether a question of the real user cannot be answered, and detect that the response information corresponding to the interaction information fails to be generated, that is, the question of the real user cannot be answered, and determine that the interaction state between the virtual object terminal and the real user satisfies a preset condition when the number of failure times reaches a first preset number of times.

Illustratively, the number of times of failure of detecting the state of generating the response information corresponding to the interactive information is a first preset number of times, which may be the number of times of the same real user; or may be the number of times accumulated for a plurality of real users, which is not limited in the embodiment of the present disclosure.

For the situation of the above (1), in some embodiments, the virtual object terminal may further determine that the interaction state with the real user satisfies the preset condition when detecting that the number of times of the preset feedback keyword corresponding to the response information in the interaction information reaches a second preset number of times; the response information is generated aiming at the interactive information; the preset feedback keyword represents that the real user requests to provide correct response information again.

Exemplarily, in the process of interacting with the real user through the virtual object, the virtual object terminal may perform text recognition on the feedback information of the real user for the response information by using the semantic understanding software module after the virtual object provides the response information to the real user, or perform voice recognition on the feedback information of the real user for the response information by using the voice recognition software module; and determining that the interaction state between the real user and the user meets the preset condition under the condition that the preset feedback keyword is identified from the feedback information and the number of times of identifying the preset keyword reaches a second preset number of times.

The preset feedback keywords are identified from the feedback information, and the times of identifying the preset keywords reach a second preset time, which can be times of aiming at the same real user; the number of times accumulated for a plurality of real users may also be used, and the embodiment of the present disclosure also does not limit this.

The preset feedback keywords may be set according to actual needs, for example, the preset feedback keywords may be "dissatisfied", "incorrect", and the like, which is not limited in the embodiment of the present disclosure; the numerical values of the first preset number and the second preset number can be set according to actual needs, which is not limited in the embodiment of the disclosure.

In the case of the above (2), the virtual object terminal may perform keyword recognition on the second communication data, such as text information or voice information, included in the received interactive information, for example, perform text recognition on the text information, perform voice recognition on the voice information, and send a remote assistance request to the real customer service terminal when a preset keyword is recognized from the text information or the voice information.

Illustratively, the virtual object terminal can perform keyword recognition on the character information input by the real user through digital human service; and carrying out voice recognition on voice information input by a real user, and sending a remote assistance request to the real customer service terminal under the condition of recognizing the preset keyword.

For the case (3) above, the virtual object terminal may send a remote assistance request to the real customer service terminal when detecting that the interaction information is an assistance operation of the real user.

In some embodiments, the virtual object terminal may display a virtual key while displaying the virtual object, or the virtual object terminal is further provided with a physical key, the real user may send an assistance operation to the real customer service terminal by clicking the virtual key or the physical key, and the real customer service terminal sends a remote assistance request to the virtual object terminal when receiving the assistance operation.

S103, receiving a feedback message aiming at the remote assistance request, and sending first communication data of a real user to a real customer service terminal; the first communication data is interactive request data made by a real user to the virtual object.

In the embodiment of the present disclosure, when the virtual object terminal receives the feedback message for the remote assistance request, the virtual object terminal may send the first communication data of the real user to the real customer service terminal, so that the real customer service terminal generates the assistance information by referring to the first communication data.

In some embodiments, the first communication data may be the above-mentioned mutual information; for example, when the interactive information is question information that the virtual object terminal cannot answer, and when the real customer service terminal receives the remote assistance request, the virtual object terminal forwards the question information to the real customer service terminal. In some embodiments, the first communication data may be data different from the interaction information, for example, in a case where the real user sends a request for interacting with the real customer service to the virtual object terminal, the interaction information is the request, and the first communication data is data which is input to the virtual object terminal by the real user and interacts with the real customer service in a case where the real customer service terminal accepts the remote assistance request.

Illustratively, a feedback message that the real customer service terminal has accepted the remote assistance request is received at the virtual object terminal, or a feedback message that a session channel for transmitting session data is established (successfully established) is received at the virtual object terminal; and under the condition that the virtual object terminal and the real user are carrying out voice interaction, the virtual object terminal can forward the collected voice data of the real user to the real customer service terminal so that the real customer service terminal can generate corresponding assistance information according to the voice data of the real user.

Illustratively, when a feedback message that the real customer service terminal has accepted the remote assistance request is received, or a feedback message that a session channel for transmitting session data is established and sent by a server is received at the virtual object terminal; and under the condition that the virtual object terminal is carrying out video interaction with the real user through the virtual object, the virtual object terminal can forward the video data of the real user to the real customer service terminal so that the real customer service terminal can generate corresponding assistance information according to the video data of the real user. It should be noted that, in the embodiment of the present disclosure, the virtual object terminal performs voice interaction with the real user, may represent that the real user is sending a request for voice interaction to the virtual object terminal, and may also represent that the virtual object terminal has accepted the request for voice interaction of the real user and is performing interaction through voice communication; similarly, the virtual object terminal and the real user in the embodiment of the present disclosure perform video interaction, which may represent that the real user is sending a video interaction request to the virtual object terminal, and may also represent that the virtual object terminal has accepted the video interaction request of the real user and is performing interaction through video communication, which is not limited in this embodiment of the present disclosure. It should be noted that the virtual object terminal performs video interaction with the real user through the virtual object, the representation virtual object terminal displays the real-time image of the virtual object on the display screen of the virtual object terminal, and the virtual object terminal acquires the video image of the real user in real time through the camera and simultaneously acquires the voice information of the real user in real time through the microphone.

And S104, receiving the assistance information which is sent by the real customer service terminal and corresponds to the first communication data, and interacting with the real user by adopting the assistance information.

In the embodiment of the disclosure, the virtual object terminal may use assistance information such as text information, voice information, or video information corresponding to the first communication data sent by the real customer service terminal to interact with the real user when receiving the assistance information.

For example, in a case where the digital human service is running on the virtual object terminal and the virtual object is displayed on the display screen of the virtual object terminal, the virtual object terminal may interact with the real user according to the received assistance information through the virtual object, for example, the virtual object terminal may answer a question of the real user with the received assistance information through the virtual object.

In the disclosed embodiment, the virtual customer service terminal receives interactive information provided by a real user and used for indicating the real user to interact with a virtual object, sends a remote assistance request representing that the virtual object terminal requests the real customer service terminal to provide assistance information to the real customer service terminal when detecting that the interactive information meets assistance conditions, sends interactive request data provided by the real user to the virtual object to the real customer service terminal when receiving feedback information aiming at the remote assistance request, and interacts with the real user by adopting the assistance information when receiving the assistance information corresponding to the interactive request data sent by the real customer service terminal, so that the response coverage of the virtual object terminal on the interactive request data of the real user and the correctness of response information provided to the real user are improved, therefore, the function diversity of the virtual object terminal is increased, and the intelligence of the virtual object terminal is improved.

In some embodiments of the present disclosure, the method further includes S105, which will be described below with reference to the steps shown in fig. 3.

And S105, generating response information corresponding to the interactive information for responding when the interactive information is detected not to meet the assistance condition.

In the embodiment of the disclosure, the virtual object terminal may generate the response information corresponding to the interaction information and provide the response information to the real user when determining that the received interaction information of the real user does not satisfy the assistance condition, so as to implement the autonomous response to the interaction information.

In some embodiments, in a case where the interactive information is question information provided by the user and the virtual object terminal determines that the question information can be solved autonomously, the virtual object terminal determines that the interactive information does not satisfy the assistance condition; in some embodiments, in a case that the interactive information is an interactive request with a low degree of urgency, for example, in a case that the interactive request does not relate to urgent recourse, property, or the like, the virtual object terminal determines that the interactive information does not satisfy the assistance condition; and in some embodiments, when the virtual object performs voice or text recognition on the interactive information, and the virtual object terminal determines that the interactive information does not satisfy the assistance condition under the condition that the preset keyword is not recognized.

In some embodiments of the present disclosure, the above S102 may be implemented by S1021 or S1022, which will be described below in conjunction with the steps shown in fig. 4.

S1021, detecting that the first type of keywords appear in the second communication data in the interactive information, and sending a takeover request to the real customer service terminal; the preset keywords include: a first type of preset keyword; the remote assistance request includes: the request is taken over.

In the embodiment of the present disclosure, the second communication data may be text information or voice information in the interaction information, and the virtual object terminal may send a takeover request to the real customer service terminal when the first type of keyword is detected from the received text information or voice information of the real user.

In some embodiments, the takeover request and the video assistance request represent the assistance information provided by the request as two different types of information; in other embodiments, the takeover request is a request with low urgency, and the video assistance request is a request with high urgency, which is not limited by the embodiments of the present disclosure.

S1022, when a second type of keywords appear in second communication data in the interactive information, sending a video assistance request to a real customer service terminal; the preset keywords include: presetting keywords in a second type; the remote assistance request includes: and requesting video assistance.

In the embodiment of the present disclosure, the virtual object terminal may send a takeover request to the real customer service terminal when detecting the second type of keyword from the received text information or voice information of the real user.

The first type of keywords and the second type of keywords represent different semantics and can be set according to actual needs; for example, the first category of keywords may be "dumb", "unable to solve", and the second category of keywords may be "alarm", "urgent", which is not limited by the embodiments of the present disclosure.

In some embodiments of the present disclosure, the above S102 may also be implemented by S1023 or S1024, which will be described below in conjunction with the steps shown in fig. 5.

S1023, sending a takeover request to the real customer service terminal under the condition that the interaction information is detected to be the takeover assisting operation; the assisting operation comprises: the video assists the operation.

S1024, sending a video assistance request to a real customer service terminal under the condition that the interactive information is detected to be video assistance operation; the assisting operation comprises: the video assists the operation.

In the embodiment of the present disclosure, the virtual object terminal may send a takeover request corresponding to the takeover assistance operation to the real customer service terminal when detecting that the interaction information of the real user is the takeover assistance operation, and send a video assistance request corresponding to the video assistance operation to the real customer service terminal when detecting that the interaction information of the real user is the video assistance operation.

Illustratively, the virtual object terminal may simultaneously display different virtual buttons, for example, a first virtual button and a second virtual button, on a page on which the virtual object is displayed during interaction with a real user through the virtual object; alternatively, different physical buttons, for example, a first physical button and a second physical button, may be provided on the virtual object terminal. Under the condition that the touch operation of a real user for a first virtual button or a first physical button is received, determining that the takeover assisting operation is received, and sending a takeover request to a real customer service terminal; or, in the case of receiving a touch operation of the real user on the second virtual button or the second physical button, determining that the video assistance operation is received, and sending a video assistance request to the real customer service terminal.

Exemplarily, the virtual object terminal may further acquire an image of the real user through the camera, identify a limb action of the real user in the image, determine to receive the takeover assistance operation and send a takeover request to the real customer service terminal when the limb action is a first preset action, and determine to receive the video assistance operation and send the video assistance request to the real customer service terminal when the limb action is a second preset action.

In some embodiments of the present disclosure, S103 may be implemented by S1031 or S1032, which will be described below with reference to the steps shown in fig. 6.

S1031, under the condition that a feedback message aiming at the remote assistance request is received and the remote assistance request is a takeover request, sending the collected voice data or video data of the real user to the real customer service terminal; the first communication data includes: audio data or video data; the remote assistance request includes: the request is taken over.

In the embodiment of the present disclosure, when the remote assistance request sent to the real customer service terminal is a takeover request, the virtual object terminal may send the collected voice data or video data of the real user to the real customer service terminal, so that the real customer service terminal generates corresponding assistance information according to the voice data or video data.

For example, in a case where the remote assistance request is a takeover request and the virtual object displayed on the virtual object terminal performs voice interaction with the real user, the virtual object terminal may send voice data of the real user to the real customer service terminal in real time.

It should be noted that the voice data may be voice data acquired by the virtual object terminal through the microphone in real time, or may also be data corresponding to a segment of recording; and, the video data may include: the virtual object terminal acquires a video image of a real user in real time through a camera, and simultaneously acquires voice information of the real user in real time through a microphone of the virtual object terminal; the data may also be data corresponding to a segment of recorded video, which is not limited in the embodiment of the present disclosure.

S1032, sending the collected video data of the real user to the real customer service terminal under the condition that a feedback message aiming at the remote assistance request is received and the remote assistance request is a video assistance request; the first communication data includes: video data; the remote assistance request includes: and requesting video assistance.

In the embodiment of the disclosure, when the remote assistance request sent to the real customer service terminal is a video assistance request, the virtual object terminal may send the collected video data of the real user to the real customer service terminal, so that the real customer service terminal generates corresponding assistance information according to the video data.

For example, in a case where the remote assistance request is a video assistance request and the virtual object displayed on the virtual object terminal is performing video interaction with the real user, the virtual object terminal may send video interaction data of the real user to the real customer service terminal in real time.

In some embodiments of the present disclosure, the method further includes S106, and it should be noted that S106 may be executed after S103.

And S106, under the condition that the first communication data comprise voice data, performing voice recognition on the voice data to obtain text information, and sending the text information to the real customer service terminal.

In the embodiment of the disclosure, when determining that the first communication data of the real user includes the voice data, the virtual object terminal may perform voice data recognition on the voice data in real time by using a voice recognition device or recognition software to obtain corresponding text information, and send the text information to the real customer service terminal.

In the embodiment of the disclosure, the virtual object terminal performs voice recognition on voice data and sends the obtained text information to the real customer service terminal, so that the real customer service terminal can provide assistance information according to the text information, and the efficiency of providing the assistance information by the real customer service terminal is improved.

In some embodiments of the present disclosure, the step S104 may be implemented by any step of S1041-S1043, which is described in conjunction with fig. 7.

S1041, when the remote assistance request is a takeover request and the text assistance information is received, adopting the text assistance information to interact with a real user through a virtual object; the assistance information includes: any one of text assistance information, voice assistance information, and video assistance information.

In the embodiment of the disclosure, when the virtual object terminal sends the takeover request to the real customer service terminal and receives the text assistance information provided by the real customer service terminal, the virtual object terminal interacts with the real user by using the text information.

In some embodiments, the virtual object terminal may convert the received text assistance information into voice information through the virtual object, and feed back the text assistance information to the real user through the virtual object in a voice broadcast manner.

S1042, under the condition that the remote assistance request is a takeover request and the voice assistance information is received, recognizing the voice assistance information to obtain text assistance information, and interacting with a real user by adopting the obtained text assistance information through a virtual object.

In the embodiment of the disclosure, when the virtual object terminal sends the takeover request to the real customer service terminal and receives the voice assistance information provided by the real customer service terminal, the virtual object terminal obtains the text assistance information corresponding to the voice assistance information in a voice recognition mode, and sends the text assistance information to the real user through the virtual object.

In some embodiments, the virtual object terminal may convert the received voice assistance information into voice information, and interact with the real user by using the obtained text assistance information through the virtual object.

And S1043, sending the video assistance information to a real user for interaction under the condition that the remote assistance request is a video assistance request and the video assistance information is received.

In the embodiment of the disclosure, when the virtual object terminal sends the video assistance request to the real customer service terminal and receives the video assistance information provided by the real customer service terminal, the virtual object terminal can forward the video assistance information to the real user, so as to realize interaction with the real user.

In some embodiments, the real customer service terminal may send real-time video data generated by the real customer service terminal in the video call to the virtual object terminal when receiving the video call request of the virtual object terminal and switching on the video call request with the real customer service terminal, so that the real customer service terminal may perform a video call with a real user corresponding to the virtual object terminal to provide video assistance information through the video call with the real user.

In some embodiments, the above method further comprises S107, and S107 may be performed before or after S101:

s107, sending the first login state of the virtual object terminal to a server; the first login state represents whether the virtual object terminal is in an online state or an offline state.

In the embodiment of the disclosure, the virtual object terminal may send the first login state to the server to inform that the server is in an online state or an offline state.

In some embodiments, the virtual object terminal may send the presence status to the server to notify of the presence status in case of being online; and, the virtual object terminal may transmit the offline state to the server in case of being offline, to notify the offline state.

In some embodiments, the virtual object terminal may further send the real-time login status to the server when receiving a status acquisition request of the server; or the virtual object terminal can send the login state to the server at regular time, so as to realize the regular reporting of the state.

Fig. 8 is an alternative flowchart of an interaction control method provided by the embodiment of the present disclosure, which will be described with reference to the steps shown in fig. 8.

S401, receiving a remote assistance request sent by a virtual object terminal; the remote assistance request represents a virtual object terminal to request a real customer service terminal to provide assistance information; the assistance information is used for interaction between the virtual object terminal and the real user.

S402, sending a request acceptance message aiming at the remote assistance request to the server, so that the server sends a feedback message to the virtual object terminal based on the request acceptance message.

In the embodiment of the present disclosure, the real customer service terminal may send, to the server, a request acceptance message for the remote assistance request when receiving the remote assistance request sent by the virtual object terminal, so that the server sends, to the virtual object terminal, a feedback message for the remote assistance request based on the request acceptance message.

In some embodiments, a remote control service is run on the real customer service terminal, and when the real customer service terminal receives a remote assistance request of the virtual object terminal through the remote control service and receives a request receiving operation of the real customer service for the remote assistance request, the real customer service terminal sends a request receiving message to the server to represent that the real customer service terminal can provide assistance information for the virtual object terminal; it should be noted that the real customer service may be a real person operating the remote control service.

In some embodiments, the real customer service terminal may display the virtual key through a display page of the remote control service, or the real customer service terminal may be provided with a physical key, and determine that a request acceptance operation for the real customer service is received and send a request acceptance message for the remote assistance request to the server, in a case that a touch operation for the virtual key or the physical key by the real customer service is received.

S403, receiving first communication data of the real user sent by the virtual object terminal based on the feedback message.

In the embodiment of the present disclosure, after the real customer service terminal sends the request acceptance message to the server, the first communication data of the real user sent by the virtual object terminal based on the feedback message may be received from the virtual object terminal.

And S404, generating assistance information corresponding to the first communication data and sending the assistance information to the virtual object terminal.

In the embodiment of the present disclosure, when receiving the first communication data sent by the virtual object terminal, the real customer service terminal may generate corresponding assistance information according to the first communication data, and send the assistance information to the virtual object terminal.

In some embodiments, the real customer service terminal runs a remote control service, and the terminal may generate assistance information corresponding to the first communication data according to information such as voice information, text information, or video information input by the real customer service through the remote control service.

In some embodiments, prior to S401 described above, S405-S406 may be performed, as will be described below in conjunction with fig. 9.

S405, acquiring a first login state of the virtual object terminal; the first login state represents whether the virtual object terminal is in an online state or an offline state.

In the embodiment of the disclosure, the real customer service terminal may obtain the first login state of the virtual object terminal from the server, so as to know whether the virtual object terminal is in an online state.

S406, displaying a login state display page according to the first login state and a second login state of the real customer service terminal; the second login state represents whether the real customer service terminal is in an online state or an offline state; wherein, the landing state display page comprises: the first login state and the identification information of the virtual object terminal, and the second login state and the identification information of the real customer service terminal.

In the embodiment of the disclosure, the real customer service terminal can acquire the second login state under the condition of acquiring the first login state of the virtual object terminal, and generates and displays the login state display page according to the first login state and the second login state, and displays the first login state and the identification information of the virtual object terminal and the second login state and the identification information of the real customer service terminal by displaying the login state display page. The identification information of the virtual object terminal may be at least one of account information, avatar information, and nickname information of the virtual object; and the identification information of the real customer service terminal may be at least one of account information, avatar information and nickname information of the real customer service.

In some embodiments, the real customer service terminal is in an idle state when not receiving the remote assistance request, and displays the login state display page according to the first login state of the virtual object terminal and the second login state of the real customer service terminal in the idle state.

Illustratively, fig. 10 is a schematic diagram of an exemplary login status presentation page provided by an embodiment of the present disclosure. As shown in fig. 10, in the case that the virtual object is a virtual customer service and the real customer service is a real person customer service, the left part of the login status display page displays the head portrait of the virtual customer service, the account information "02899" and the nickname information "digital person", and the head portrait of the virtual customer service is gray to represent that the virtual customer service terminal is currently in an off-line state; the head portrait of the real person customer service, the account information 'service _ 01' and the nickname information 'Zhang III' are displayed on the right part of the login state display page, and the head portrait of the real person customer service is bright and represents that the real person customer service terminal is in an online state at present.

In some embodiments, before the above S402, the method further comprises S407:

s407, displaying a request display page based on the remote assistance request; the request display page comprises a request receiving inlet; the request accepting inlet is used for receiving an accepting operation aiming at the remote assistance request.

In the embodiment of the present disclosure, when receiving the remote assistance request, the real customer service terminal may display a request presentation page according to the remote assistance request, and display a request acceptance entry by displaying the request presentation page, so as to receive a request acceptance operation of the real customer service through the request acceptance entry.

It should be noted that the "entry" in the embodiment of the present disclosure may be a virtual control, or may be in other forms, which is not limited in the embodiment of the present disclosure.

In some embodiments, the real customer service terminal changes from the idle state to the called state when receiving the remote assistance request, and displays the request presentation page when being in the called state.

In some embodiments, the request presentation page may further display a request animation, where the request animation may be a flashing avatar of the virtual customer service, may also be a timing picture of a timer, or may also include both the flashing avatar of the virtual customer service and the timing picture of the timer, where the timer is used to count the call time of the remote assistance request, and when the call time reaches a preset time, the real customer service terminal jumps from the request presentation page to the login status presentation page, thereby changing from the called status to the idle status.

Exemplarily, fig. 11 is a schematic diagram of an exemplary request presentation page provided by the embodiment of the present disclosure. As shown in fig. 11, the request presentation page displays a flashing avatar of the virtual customer service, displays a timing screen of a timer, and counts up to 59S currently, and displays an "answer" control (the request acceptance entry mentioned above) and a "hang-up" control (the request rejection entry) for rejecting the remote assistance request; and under the condition that the request refusing operation of the real customer service for the request refusing entrance is received, the real customer service terminal jumps to the display page in the login state from the request display page, so that the called state is changed into the idle state.

In some embodiments, S402 may be implemented by S4021 or S4022, and will be described with reference to fig. 12 by taking fig. 8 as an example.

S4021, receiving a request acceptance operation for the request acceptance entry, and transmitting a request acceptance message for the remote assistance request to the server.

In the embodiment of the disclosure, when receiving an operation of receiving a request for a request receiving entry by a real customer service, a real customer service terminal determines that the real customer service agrees to provide assistance information to a virtual object terminal, and sends a request receiving message to a server according to the request receiving operation, so that the server sends a feedback message for a remote assistance request to the virtual object terminal according to the request receiving message.

Exemplarily, with respect to fig. 11, in a case where the real customer service terminal receives a touch operation for the "answer" control, a request acceptance message for the remote assistance request is sent to the server.

S4022, the real customer service terminal is in idle state, and sends a request acceptance message for the remote assistance request to the server; the idle state represents that the real customer service terminal does not provide assistance information.

In the embodiment of the present disclosure, when determining that no assistance information is provided to any terminal, the real customer service terminal may send a request acceptance message to the server, so that the server sends a feedback message for the remote assistance request to the virtual object terminal according to the request acceptance message.

In some embodiments, S404 described above may be implemented by S4041:

s4041, generating an interactive page for displaying the interactive data, generating assistance information corresponding to the first communication data based on the interactive page, and sending the assistance information to the virtual object terminal.

In the embodiment of the disclosure, the real customer service terminal may generate an interaction page for displaying the interaction data when receiving the first communication data sent by the virtual object terminal, and generate assistance information corresponding to the first communication data according to the interaction page, and send the assistance information to the virtual object terminal.

In some embodiments, S4041 may be implemented by S501 or S502, that is, S404 in fig. 8 may be implemented by S501 or S502, which will be described in conjunction with fig. 13.

S501, under the condition that the remote assistance request is a takeover request, generating a first assistance page, generating assistance information corresponding to first communication data based on the first assistance page, and sending the assistance information to the virtual object terminal; the interactive page includes: a first assistance page; the remote assistance request includes: the request is taken over.

In the embodiment of the present disclosure, the real customer service terminal generates the first assistance page when the remote assistance request is the takeover request, and the real customer service terminal may generate the first assistance page corresponding to the takeover request when receiving the takeover request sent by the virtual object terminal and receiving the takeover request.

In some embodiments, the real customer service terminal may jump from the request presentation page to the first assistance page when receiving the takeover request sent by the virtual object terminal, so as to change the called state to the takeover state.

Illustratively, fig. 14 is a schematic diagram of an exemplary first assistance page provided by an embodiment of the present disclosure. As shown in fig. 14, the first assistance page may display chat information of the real customer service with the virtual object, an avatar of the virtual object, and an assistance information input tool for use by the real customer service, e.g., a "voice access" control in fig. 14.

In some embodiments, where the interaction page is a first assistance page, the displayed interaction data includes at least one of: the video data of the real user, the video data of the virtual object, the text assistance information and the text information sent by the virtual object terminal; the text information is obtained by performing voice recognition on voice data in the communication data.

Illustratively, fig. 15 is another schematic diagram of an exemplary first assistance page provided by an embodiment of the present disclosure. As shown in fig. 15, the first assistance page displays video data 151 of the real user, video data 152 of the virtual object, text assistance information 153, and text information 154 transmitted from the virtual object terminal.

In some embodiments, S4041 may be implemented by S601 or S602, that is, S404 in fig. 8 may also be implemented by S601 or S602, which will be described in conjunction with fig. 16.

S601, generating a first assistance page under the condition that the remote assistance request is a takeover request, generating text assistance information or voice assistance information corresponding to communication data under the condition that a first input operation aiming at the first assistance page is received, and sending the text assistance information or the voice assistance information to the virtual object terminal; the interactive page includes: a first assistance page; the remote assistance request includes: the request is taken over.

In the embodiment of the disclosure, the real customer service terminal generates and displays the first assistance page when the remote assistance request is the takeover request, and the real customer service terminal can generate text assistance information or voice assistance information corresponding to the first communication data and send the text assistance information or the voice assistance information to the virtual object terminal when receiving the first input operation of the real customer service for the first assistance page.

Exemplarily, in a case that the real customer service terminal receives a touch operation of the real customer service for the "voice access" control in fig. 14, the real customer service terminal may generate the voice assistance information corresponding to the first communication data according to the voice data input by the real customer service through the touch operation; or, the text assistance information corresponding to the first communication data may be generated according to the voice data input by the real customer service through the touch operation.

Illustratively, a virtual keyboard may be further displayed on the first assistance page, and the real customer service terminal may generate text assistance information corresponding to text data according to text data input by the real customer service through a touch operation when receiving the touch operation of the real customer service on the virtual keyboard.

Continuing with the description of S502 in fig. 13: s502, under the condition that the remote assistance request is a video assistance request, generating a second assistance page, generating assistance information corresponding to the communication data based on the second assistance page, and sending the assistance information to the virtual object terminal; the interactive page includes: a second assistance page; the remote assistance request includes: and requesting video assistance.

In the embodiment of the present disclosure, when receiving a video assistance request sent by a virtual object terminal and receiving the video assistance request, a real customer service terminal may generate and display a second assistance page corresponding to the video assistance request, generate assistance information corresponding to first communication data through the second assistance page, and send the assistance information to the virtual object terminal.

In some embodiments, in the case where the interaction page is a second assistance page, the displayed interaction data includes at least one of: video data of real users and video data of real customer services.

In some embodiments, the real customer service terminal may jump from the request presentation page to the second assistance page to realize the transition from the called state to the video assistance state, when receiving the video assistance request sent by the virtual object terminal.

Illustratively, fig. 17A is a schematic diagram of an exemplary second assistance page provided by an embodiment of the present disclosure. As shown in fig. 17A, a video image 171 of the real user, a video image 172 of the real customer service, and a "video" control may be displayed on the second assistance page, and a request assistance state of the real customer service for the virtual object, such as the avatar of the virtual object, the account information "02899", and "listening" shown in fig. 17A, is displayed, which indicates that the remote assistance request of the real customer service for the digital person 02899 is being listened to. The video control is used for receiving touch operation, the real customer service terminal starts a microphone and a camera of the real customer service terminal under the condition that the real customer service terminal receives the touch operation aiming at the video control, the video image of the real customer service collected by the camera is displayed in 172, and the voice data of the real customer service collected by the microphone is sent to the virtual object terminal; it should be noted that, under the condition that the real customer service terminal displays the second assistance page, the real customer service terminal can receive the video image and the voice data of the real user at the same time, so as to realize the audio-video interaction with the real user under the condition that the touch operation of the real customer service for the "video" control is received.

Continuing with the description of S602 in fig. 16: s602, generating a second assistance page under the condition that the remote assistance request is a video assistance request, generating video assistance information corresponding to the first communication data under the condition that a second input operation aiming at the second assistance page is received, and sending the video assistance information to the virtual object terminal; the interactive page includes: a second assistance page; the remote assistance request includes: and requesting video assistance.

For example, as described above, when the real customer service terminal receives the touch operation of the real customer service on the "video" control, in 172 of the second assistance page, the video image of the real customer service collected by the camera is displayed for the real user to browse, for example, as shown in fig. 17B; and sending the voice data of the real customer service collected by the microphone to the virtual object terminal for the real user corresponding to the virtual object terminal to listen to, thereby realizing the generation of the video assistance information corresponding to the video data of the real user.

In some embodiments, the real customer service terminal may further generate video assistance information corresponding to the first communication data directly based on the second assistance page.

In some embodiments, the real customer service terminal displays the second assistance page and simultaneously displays the video image of the real customer service through the second assistance page, and simultaneously sends the voice information of the real customer service to the virtual object terminal, and simultaneously receives the video image and the voice information of the real user sent by the virtual object terminal. Illustratively, fig. 18 is yet another schematic diagram of an exemplary second assistance page provided by an embodiment of the present disclosure. As shown in fig. 18, the second assistance page displays a video image of the real user, a video image of the real customer service, and a request assistance state of the real customer service for the virtual object, such as the avatar of the virtual object, the account information "02899" and "listening" shown in fig. 18, indicating that the remote assistance request of the real customer service for the digital person 02899 is being listened to; a mute control is also displayed and is used for stopping sending the voice data of the real customer service to the virtual object terminal under the condition of receiving the touch operation of the real customer service to the control; and, also displayed is "talk time: 00:18 ", representing the video call duration of the real customer service and the real user.

In some embodiments, after the above S407, S408 may be further included;

s408, under the condition that the operation for closing the entrance is received, finishing the remote assistance to the virtual object terminal; the interactive page includes: the inlet is closed.

In the embodiment of the present disclosure, the interactive page further includes a closed entry, and the real customer service terminal may end the remote assistance to the virtual object terminal when receiving the situation that the real customer service is directed to the closed entry.

Illustratively, FIG. 19A is a first assistance page containing a close entry, and the close entry is an "end takeover" control; FIG. 19B is a second assistance page that includes a close entry, the close entry being a "hang up" control.

Fig. 20 is an alternative flowchart of an interaction control method provided by the embodiment of the present disclosure, which will be described with reference to the steps shown in fig. 20.

S701, receiving a remote assistance request sent by a virtual object terminal, and forwarding the remote assistance request to a real customer service terminal; the remote assistance request represents a virtual object terminal to request a real customer service terminal to provide assistance information; the assistance information is used for the virtual object to interact with the real user.

In the embodiment of the disclosure, the server may forward the received remote assistance request sent by the virtual object terminal to the real customer service terminal.

S702, receiving a request acceptance message sent by the real customer service terminal.

In the embodiment of the disclosure, after forwarding the received remote assistance request sent by the virtual object terminal to the real customer service terminal, the server may receive a request acceptance message for the remote assistance request sent by the real customer service terminal.

And S703, sending a feedback message to the virtual object terminal based on the request acceptance message.

In the embodiment of the present disclosure, after receiving the request acceptance message, the server may send a feedback message for the remote assistance request to the virtual object terminal.

In some embodiments, the server may send the received request acceptance message to the virtual object terminal as a feedback message for the remote assistance request; in some embodiments, the remote assistance request further carries a number of a communication channel, and the server may initialize the communication channel after receiving the request acceptance message, add both the virtual object terminal and the real customer service terminal to the communication channel, thereby establishing a session channel between the virtual object terminal and the real customer service terminal and used for transmitting communication data and assistance information, and send a feedback message that the session channel is successfully established to the virtual object terminal after the session channel is established.

S704, first communication data of the real user sent by the virtual object terminal based on the feedback message is received, and the first communication data is forwarded to the real customer service terminal.

In the embodiment of the disclosure, the server may forward the communication data to the real customer service terminal when receiving the communication data of the real user, which is sent by the virtual object terminal based on the feedback message and corresponds to the remote assistance request, so that the real customer service terminal generates the assistance information according to the communication information.

S705, receiving assistance information which is sent by the real customer service terminal and corresponds to the first communication data, and forwarding the assistance information to the virtual object terminal.

In the embodiment of the disclosure, the server may send the assistance information to the virtual object terminal when receiving the assistance information which is sent by the real customer service terminal and corresponds to the first communication data, so that the virtual object terminal interacts with the real user based on the assistance information.

In some embodiments, the method further includes S706, it should be noted that S706 may be executed before S701 or after S701.

S706, sending the received first login state sent by the virtual object terminal to the real customer service terminal; the first login state represents whether the virtual object terminal is in an online state or an offline state.

In some embodiments, the server may forward the first login state to the real customer service terminal upon receiving the first login state sent by the virtual object terminal; and in some embodiments, the server can also query the first login state of the virtual object terminal at regular time and send the first login state to the real customer service terminal at regular time.

In other embodiments, the server may further forward the received first login state sent by the virtual object terminal to the real customer service terminal when receiving the state acquisition request of the real customer service terminal; in other embodiments, the server may further obtain the first login state from the virtual object terminal only when receiving the state obtaining request of the real customer service terminal, and forward the received first login state sent by the virtual object terminal to the real customer service terminal.

The interactive control method provided by the embodiment of the present disclosure will be described below through an application of the interactive control method in an actual scene. Fig. 21 is a schematic view of an interaction flow between a virtual object terminal, a real client server terminal and a server according to an embodiment of the present disclosure, which will be described with reference to the steps shown in fig. 21.

S1, the virtual object terminal receives the interaction information which is provided by the real user and used for indicating the interaction between the real user and the virtual customer service;

s2, the virtual object terminal detects that the interaction state between the virtual customer service and the real user meets the preset condition, or detects a first type of key words from the voice information or text information input by the real user interacting with the virtual customer service, or receives the takeover assisting operation of the real user and sends a takeover request to the real customer service terminal; or the virtual object terminal detects that the interaction state between the virtual customer service and the real user meets a preset condition, or detects a second type of key words from voice information or text information input by the real user, or receives video assistance operation of the real user and sends a video assistance request to the real customer service terminal;

s3, the server forwards the takeover request or the video assistance request to the real customer service terminal;

s4, the real customer service terminal obtains the first login state of the virtual customer service terminal from the server side, and displays a login state display page according to the first login state and the second login state of the real customer service terminal;

s5, the real customer service terminal jumps to a request display page from a landing state display page under the condition of receiving a takeover request or a video assistance request, and the request display page comprises a request receiving inlet;

s6, the real customer service terminal sends a request acceptance message to the server when receiving the request acceptance operation of the real customer service for the request acceptance entrance; meanwhile, under the condition that a takeover request is received, jumping to a first assistance page from a request display page, or under the condition that a video assistance request is received, jumping to a second assistance page from the request display page;

s7, the server initializes the communication channel according to the request receiving information, establishes the conversation channel between the virtual object terminal and the real customer service terminal for transmitting the communication data and the interactive information, and sends the successful channel establishment information to the virtual object terminal after the conversation channel is established successfully;

s8, the virtual object terminal sends the first communication data of the real user to the real customer service terminal by adopting the session channel based on the channel establishment success message;

s9, the server forwards the first communication data of the real user to the real customer service terminal;

s10, generating text assistance information or voice assistance information corresponding to the first communication data by the real customer service terminal under the condition of receiving touch operation aiming at the voice access control or the virtual keyboard on the first assistance page, and sending the text assistance information or the voice assistance information to the server; or the real customer service terminal establishes a video call with the virtual object terminal under the condition of jumping to the second assistance page, displays the video image of the real customer service through the second assistance page, and sends the video data of the real customer service to the virtual object terminal;

and S11, the server sends the text assistance information, the voice assistance information or the video information sent by the real customer service terminal to the virtual object terminal based on the session channel.

S12, the virtual object terminal broadcasts the text assistance information to the real user by voice, or sends the voice assistance information to the real user; or the virtual object terminal forwards the video data of the real person customer service sent by the real customer service terminal to the real user; and meanwhile, forwarding the video data of the real user to a real customer service terminal to realize the video interaction between the real customer service and the real user.

The present disclosure further provides an interactive control device, and fig. 22 is a schematic structural diagram of an interactive control device provided in an embodiment of the present disclosure; as shown in fig. 22, the interactive control device 1 includes: a first receiving unit 11, configured to receive interaction information provided by a real user; the interaction information is used for indicating the interaction between the real user and the virtual object; the first sending unit 12 is configured to send a remote assistance request to a real customer service terminal when detecting that the interaction information satisfies an assistance condition; the remote assistance request representation virtual object terminal requests the real customer service terminal to provide assistance information; receiving a feedback message aiming at the remote assistance request, and sending first communication data of the real user to the real customer service terminal; the first communication data is interaction request data which is made by the real user to the virtual object; and the interaction unit 13 is further configured to receive the assistance information which is sent by the real customer service terminal and corresponds to the first communication data, and interact with the real user by using the assistance information.

In some embodiments of the present disclosure, the interaction control apparatus 1 further includes a detection unit, configured to determine, based on the interaction information, that an interaction state with the real user satisfies a preset condition; or detecting that a preset keyword appears in second communication data contained in the interactive information; or detecting that the interaction information is assistance operation.

In some embodiments of the present disclosure, the first communication data comprises: voice data or video data; the remote assistance request comprises: taking over the request; the first sending unit 12 is further configured to send the acquired voice data or the acquired video data of the real user to the real customer service terminal when the remote assistance request is the takeover request.

In some embodiments of the present disclosure, the first communication data comprises: video data; the remote assistance request comprises: a video assistance request; the first sending unit 12 is further configured to send the acquired video data of the real user to the real customer service terminal when the remote assistance request is the video assistance request.

In some embodiments of the present disclosure, the first sending unit 12 is further configured to, when the first communication data includes voice data, perform voice recognition on the voice data to obtain text information, and send the text information to the real customer service terminal.

In some embodiments of the present disclosure, the preset keyword includes: a first type of preset keyword; the remote assistance request comprises: taking over the request; the first sending unit 12 is further configured to detect that the first type of keyword occurs in the second communication data in the interaction information, and send the takeover request to the real customer service terminal.

In some embodiments of the present disclosure, the preset keyword includes: presetting keywords in a second type; the remote assistance request comprises: a video assistance request; the first sending unit 12 is further configured to detect that the second type of keyword occurs in the second communication data in the interaction information, and send the video assistance request to the real customer service terminal.

In some embodiments of the present disclosure, the assistance information includes: any one of text assistance information, voice assistance information, and video assistance information; the interaction unit 13 is further configured to interact with the real user by using the text assistance information through the virtual object when the remote assistance request is a takeover request and the text assistance information is received; or, in the case that the remote assistance request is the takeover request and the voice assistance information is received, recognizing the voice assistance information to obtain text assistance information, and interacting with the real user by using the obtained text assistance information through the virtual object; or, when the remote assistance request is a video assistance request and the video assistance information is received, sending the video assistance information to the real user for interaction.

In some embodiments of the present disclosure, the first sending unit 12 is further configured to send the first login status of the virtual object terminal to a server; the first login state represents whether the virtual object terminal is in an online state or an offline state.

In some embodiments of the present disclosure, the detecting unit is further configured to detect that the number of times of failure of generating the response information corresponding to the interaction information reaches a first preset number of times; and/or detecting that the frequency of the preset feedback keywords corresponding to the response information in the interactive information reaches a second preset frequency; and the preset feedback keywords represent the real user request to provide correct response information again.

Fig. 23 is another schematic structural diagram of an interaction control apparatus provided in an embodiment of the present disclosure; as shown in fig. 23, the interactive control device 2 includes: a second receiving unit 21, configured to receive a remote assistance request sent by a virtual object terminal; the remote assistance request represents that the virtual object terminal requests the real customer service terminal to provide assistance information; the assistance information is used for the interaction between the virtual object terminal and a real user; a second sending unit 22, configured to send a request acceptance message for the remote assistance request to a server, so that the server sends a feedback message to the virtual object terminal based on the request acceptance message; the second receiving unit 21 is further configured to receive first communication data of the real user, which is sent by the virtual object terminal based on the feedback message; an information generating unit 23, further configured to generate the assistance information corresponding to the first communication data; the second sending unit 22 is further configured to send the assistance information corresponding to the first communication data to the virtual object terminal.

In some embodiments of the present disclosure, the interactive control device 2 further comprises an acquisition unit and a display unit; the acquiring unit is used for acquiring a first login state of the virtual object terminal before the remote assistance request sent by the virtual object terminal is received; the first login state represents whether the virtual object terminal is in an online state or an offline state; the display unit is used for displaying a login state display page according to the first login state and the second login state of the real customer service terminal; the second login state represents whether the real customer service terminal is in an online state or an offline state; wherein, the landing status display page comprises: the first login state and the identification information of the virtual object terminal, and the second login state and the identification information of the real customer service terminal.

In some embodiments of the present disclosure, the display unit is further configured to display a request presentation page based on the remote assistance request before the sending of the request acceptance message for the remote assistance request to the virtual object terminal; the request display page comprises a request accepting inlet; the request acceptance entry is used for receiving an acceptance operation for the remote assistance request.

In some embodiments of the present disclosure, the second sending unit 22 is further configured to send a request acceptance message for the remote assistance request to a server if the acceptance operation for the acceptance entry is received; or, the real customer service terminal is in an idle state, and sends a request acceptance message for the remote assistance request to a server; the idle state represents that the real customer service terminal does not provide assistance information.

In some embodiments of the present disclosure, the information generating unit 23 is further configured to generate an interaction page for displaying interaction data, and generate the assistance information corresponding to the first communication data based on the interaction page.

In some embodiments of the present disclosure, the interaction page comprises: closing the inlet; the interaction control apparatus 2 further includes an assistance ending unit configured to end the remote assistance to the virtual object terminal in a case where an operation for closing the entry is received.

In some embodiments of the present disclosure, the interaction page comprises: a first assistance page and a second assistance page; the remote assistance request comprises: take over requests and video assistance requests; the information generating unit 23 is further configured to generate the first assistance page when the remote assistance request is the takeover request; or, in a case that the remote assistance request is the video assistance request, generating the second assistance page.

In some embodiments of the present disclosure, the interaction page comprises: a first assistance page and a second assistance page; the assistance information includes any one of: text assistance information, voice assistance information, and video assistance information; the information generating unit 23 is further configured to generate the text assistance information or the voice assistance information corresponding to the first communication data when a first input operation for the first assistance page is received; or, in a case where a second input operation for the second assistance page is received, the video assistance information corresponding to the first communication data is generated.

In some embodiments of the present disclosure, in a case where the interaction page is a first assistance page, the interaction data includes at least one of: the video data of the real user, the video data of the virtual object, the text assistance information and the text information sent by the virtual object terminal; the text information is obtained by performing voice recognition on voice data in the first communication data; in a case that the interaction page is a second assistance page, the interaction data includes at least one of: the video data of the real user and the video data of the real customer service.

Fig. 24 is a schematic structural diagram of an interaction control apparatus provided in an embodiment of the present disclosure; as shown in fig. 24, the interactive control device 3 includes: the information forwarding unit 31 is configured to receive a remote assistance request sent by a virtual object terminal, and forward the remote assistance request to a real customer service terminal; the remote assistance request represents that the virtual object terminal requests the real customer service terminal to provide assistance information; the assistance information is used for interaction between the virtual object and a real user; a third receiving unit 32, configured to receive a request acceptance message sent by the real customer service terminal; a third sending unit 33, configured to send a feedback message to the virtual object terminal based on the request acceptance message; the information forwarding unit 31 is further configured to receive first communication data of a real user sent by the virtual object terminal based on the feedback message, and forward the first communication data to the real customer service terminal; and receiving the assistance information which is sent by the real customer service terminal and corresponds to the first communication data, and forwarding the assistance information to the virtual object terminal.

In some embodiments of the present disclosure, the information forwarding unit 31 is further configured to send the received first login status sent by the virtual object terminal to the real customer service terminal; the first login state represents whether the virtual object terminal is in an online state or an offline state.

An embodiment of the present disclosure further provides an electronic device, fig. 25 is a schematic structural diagram of the electronic device provided in the embodiment of the present disclosure, and as shown in fig. 25, the electronic device 4 includes: a memory 22 and a processor 23, wherein the memory 22 and the processor 23 are connected by a communication bus 21; a memory 22 for storing an executable computer program; the processor 23 is configured to implement the method provided by the embodiment of the present disclosure, for example, the interaction control method provided by the embodiment of the present disclosure, when executing the executable computer program stored in the memory 22.

The present disclosure provides a computer-readable storage medium, which stores a computer program for causing the processor 23 to execute a method provided by the present disclosure, for example, an interaction control method provided by the present disclosure.

In some embodiments of the present disclosure, the storage medium may be memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

The computer-readable storage medium may also be a tangible device that retains and stores instructions for use by an instruction execution device, and may be a volatile storage medium or a non-volatile storage medium. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a U disk, a magnetic disk, an optical disk, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (ROM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a recordable encoding device, a punch card or recessed well structure such as those on which instructions are stored, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating battery waves, battery waves propagating through waveguides or other media (e.g., optical pulses through fiber optic cables), or electrical signals transmitted over electrical wires.

In some embodiments of the disclosure, executable instructions may be written in any form of programming language (including compiled or interpreted languages), in the form of programs, software modules, scripts, or code, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

By way of example, executable instructions may correspond, but do not necessarily have to correspond, to files in a file system, and may be stored in a portion of a file that holds other programs or data, such as in one or more scripts in a hypertext Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code).

By way of example, executable instructions may be deployed to be executed on one computing device or on multiple computing devices at one site or distributed across multiple sites and interconnected by a communication network.

To sum up, with the technical implementation scheme, the virtual customer service terminal receives interaction information provided by a real user and used for indicating the interaction between the real user and a virtual object, sends a remote assistance request representing that the virtual object terminal requests the real customer service terminal to provide assistance information to the real customer service terminal when detecting that the interaction information satisfies assistance conditions, sends interaction request data provided by the real user to the virtual object to the real customer service terminal when receiving a feedback message for the remote assistance request, and interacts with the real user by using the assistance information when receiving assistance information corresponding to the interaction request data sent by the real customer service terminal, so as to improve the response coverage of the virtual object terminal on the interaction request data of the real user and the correctness of response information provided to the real user, therefore, the function diversity of the virtual object terminal is increased, and the intelligence of the virtual object terminal is improved.

The above description is only an example of the present disclosure, and is not intended to limit the scope of the present disclosure. Any modification, equivalent replacement, and improvement made within the spirit and scope of the present disclosure are included in the protection scope of the present disclosure.

Claims

1. An interaction control method, comprising:

receiving interactive information provided by a real user; the interaction information is used for indicating the interaction between the real user and the virtual object;

detecting that the interaction information meets assistance conditions, and sending a remote assistance request to a real customer service terminal; the remote assistance request representation virtual object terminal requests the real customer service terminal to provide assistance information;

receiving a feedback message aiming at the remote assistance request, and sending first communication data of the real user to the real customer service terminal; the first communication data is interaction request data which is made by the real user to the virtual object;

and receiving the assistance information which is sent by the real customer service terminal and corresponds to the first communication data, and adopting the assistance information to interact with the real user.

2. The method of claim 1, wherein the detecting that the interaction information satisfies an assistance condition comprises:

determining that the interaction state between the real user and the user meets a preset condition based on the interaction information; alternatively, the first and second electrodes may be,

detecting that a preset keyword appears in second communication data contained in the interactive information; alternatively, the first and second electrodes may be,

and detecting that the interaction information is assistance operation.

3. The method of claim 1 or 2, wherein the first communication data comprises: voice data or video data; the remote assistance request comprises: taking over the request;

the sending the first communication data of the real user to the real customer service terminal includes:

and sending the collected voice data or the collected video data of the real user to the real customer service terminal under the condition that the remote assistance request is the takeover request.

4. The method of claim 1 or 2, wherein the first communication data comprises: video data; the remote assistance request comprises: a video assistance request;

and sending the collected video data of the real user to the real customer service terminal under the condition that the remote assistance request is the video assistance request.

5. The method according to any one of claims 1-4, further comprising:

and under the condition that the first communication data comprises voice data, performing voice recognition on the voice data to obtain text information, and sending the text information to the real customer service terminal.

6. The method of claim 2, wherein the preset keyword comprises: a first type of preset keyword; the remote assistance request comprises: taking over the request;

the step of sending a remote assistance request to a real customer service terminal when the interactive information is detected to meet the assistance condition comprises the following steps:

and sending the takeover request to the real customer service terminal when the first type of keywords appear in the second communication data in the interaction information.

7. The method of claim 2, wherein the preset keyword comprises: presetting keywords in a second type; the remote assistance request comprises: a video assistance request;

and sending the video assistance request to the real customer service terminal when the second type of keywords appear in the second communication data in the interactive information.

8. The method of claim 1, wherein the assistance information comprises: any one of text assistance information, voice assistance information, and video assistance information;

the receiving the assistance information which is sent by the real customer service terminal and corresponds to the first communication data, and adopting the assistance information to interact with the real user includes:

when the remote assistance request is a takeover request and the text assistance information is received, adopting the text assistance information to interact with the real user through the virtual object; alternatively, the first and second electrodes may be,

when the remote assistance request is the takeover request and the voice assistance information is received, recognizing the voice assistance information to obtain text assistance information, and interacting with the real user by adopting the obtained text assistance information through the virtual object; alternatively, the first and second electrodes may be,

and sending the video assistance information to the real user for interaction under the condition that the remote assistance request is a video assistance request and the video assistance information is received.

9. The method according to any one of claims 1 to 8, further comprising:

sending the first login state of the virtual object terminal to a server; the first login state represents whether the virtual object terminal is in an online state or an offline state.

10. The method according to claim 2, wherein the determining that the interaction state with the real user satisfies a preset condition based on the interaction information includes at least one of:

detecting that the failure times of generating the response information corresponding to the interactive information reach a first preset time;

detecting that the frequency of a preset feedback keyword corresponding to the response information in the interactive information reaches a second preset frequency; and the preset feedback keywords represent the real user request to provide correct response information again.

11. An interaction control method, comprising:

receiving a remote assistance request sent by a virtual object terminal; the remote assistance request represents that the virtual object terminal requests the real customer service terminal to provide assistance information; the assistance information is used for interaction between the virtual object and a real user;

sending a request acceptance message for the remote assistance request to a server, so that the server sends a feedback message to the virtual object terminal based on the request acceptance message;

receiving first communication data of the real user sent by the virtual object terminal based on the feedback message;

and generating the assistance information corresponding to the first communication data and sending the assistance information to the virtual object terminal.

12. The method according to claim 11, wherein before said receiving the request for remote assistance sent by the virtual object terminal, the method further comprises:

acquiring a first login state of the virtual object terminal; the first login state represents whether the virtual object terminal is in an online state or an offline state;

displaying a login state display page according to the first login state and the second login state of the real customer service terminal; the second login state represents whether the real customer service terminal is in an online state or an offline state;

wherein, the landing status display page comprises: the first login state and the identification information of the virtual object terminal, and the second login state and the identification information of the real customer service terminal.

13. The method according to claim 11 or 12, wherein prior to said sending a request acceptance message for the remote assistance request to a server, the method further comprises:

displaying a request presentation page based on the remote assistance request; the request display page comprises a request accepting inlet; the request acceptance entry is used for receiving an acceptance operation for the remote assistance request.

14. The method of claim 13, wherein sending a request acceptance message for the remote assistance request to a server comprises:

receiving the acceptance operation aiming at the acceptance entry, and sending a request acceptance message aiming at the remote assistance request to a server; alternatively, the first and second electrodes may be,

the real customer service terminal is in an idle state and sends a request receiving message aiming at the remote assistance request to a server; the idle state represents that the real customer service terminal does not provide assistance information.

15. The method of claim 12, wherein the generating the assistance information corresponding to the first communication data comprises:

and generating an interaction page for displaying interaction data, and generating the assistance information corresponding to the first communication data based on the interaction page.

16. The method of claim 15, wherein the interaction page comprises: a first assistance page and a second assistance page; the assistance information includes any one of: text assistance information, voice assistance information, and video assistance information;

the generating the assistance information corresponding to the first communication data based on the interaction page includes:

generating the text assistance information or the voice assistance information corresponding to the first communication data in a case where a first input operation for the first assistance page is received; alternatively, the first and second electrodes may be,

generating the video assistance information corresponding to the first communication data if a second input operation for the second assistance page is received.

17. The method of claim 15,

in a case that the interaction page is a first assistance page, the interaction data includes at least one of: the video data of the real user, the video data of the virtual object, the text assistance information and the text information sent by the virtual object terminal; the text information is obtained by performing voice recognition on voice data in the first communication data;

in a case that the interaction page is a second assistance page, the interaction data includes at least one of: the video data of the real user and the video data of the real customer service.

18. An interaction control method, comprising:

receiving a remote assistance request sent by a virtual object terminal, and forwarding the remote assistance request to a real customer service terminal; the remote assistance request represents that the virtual object terminal requests the real customer service terminal to provide assistance information; the assistance information is used for interaction between the virtual object and a real user;

receiving a request acceptance message sent by the real customer service terminal;

sending a feedback message to the virtual object terminal based on the request acceptance message;

receiving first communication data of a real user sent by the virtual object terminal based on the feedback message, and forwarding the first communication data to the real customer service terminal;

and receiving the assistance information which is sent by the real customer service terminal and corresponds to the first communication data, and forwarding the assistance information to the virtual object terminal.

19. The method of claim 18, further comprising:

sending the received first login state of the virtual object terminal sent by the virtual object terminal to the real customer service terminal; the first login state represents whether the virtual object terminal is in an online state or an offline state.

20. An interactive control apparatus, comprising:

the first receiving unit is used for receiving interactive information provided by a real user; the interaction information is used for indicating the interaction between the real user and the virtual object;

the first sending unit is used for sending a remote assistance request to a real customer service terminal when detecting that the interaction information meets the assistance condition; the remote assistance request representation virtual object terminal requests the real customer service terminal to provide assistance information; receiving a feedback message aiming at the remote assistance request, and sending first communication data of the real user to the real customer service terminal; the first communication data is interaction request data which is made by the real user to the virtual object;

and the interaction unit is further used for receiving the assistance information which is sent by the real customer service terminal and corresponds to the first communication data, and adopting the assistance information to interact with the real user.

21. An interactive control apparatus, comprising:

the second receiving unit is used for receiving the remote assistance request sent by the virtual object terminal; the remote assistance request represents that the virtual object terminal requests the real customer service terminal to provide assistance information; the assistance information is used for interaction between the virtual object and a real user;

a second sending unit, configured to send a request acceptance message for the remote assistance request to a server, so that the server sends a feedback message to the virtual object terminal based on the request acceptance message;

the second receiving unit is further configured to receive first communication data of the real user, which is sent by the virtual object terminal based on the feedback message;

an information generating unit further configured to generate the assistance information corresponding to the first communication data;

the second sending unit is further configured to send the assistance information corresponding to the first communication data to the virtual object terminal.

22. An interactive control apparatus, comprising:

the information forwarding unit is used for receiving a remote assistance request sent by the virtual object terminal and forwarding the remote assistance request to the real customer service terminal; the remote assistance request represents that the virtual object terminal requests the real customer service terminal to provide assistance information; the assistance information is used for interaction between the virtual object and a real user;

a third receiving unit, configured to receive a request acceptance message sent by the real customer service terminal;

a third sending unit, configured to send a feedback message to the virtual object terminal based on the request acceptance message;

the information forwarding unit is further configured to receive first communication data of a real user sent by the virtual object terminal based on the feedback message, and forward the first communication data to the real customer service terminal; and receiving the assistance information which is sent by the real customer service terminal and corresponds to the first communication data, and forwarding the assistance information to the virtual object terminal.

23. An electronic device, comprising:

a memory for storing an executable computer program;

a processor for implementing the method of any one of claims 1 to 10, or the method of any one of claims 11 to 17, or the method of any one of claims 18 to 19, when executing an executable computer program stored in the memory.

24. A computer-readable storage medium, characterized in that a computer program is stored for causing a processor, when executed, to carry out the method of any one of claims 1 to 10, or the method of any one of claims 11 to 17, or the method of any one of claims 18 to 19.