JP6795028B2 - Information processing system and information processing method - Google Patents

Description

The present disclosure relates to an information processing system and an information processing method.

In recent years, with the development of communication technology, messages are frequently exchanged via networks. The user can confirm a message sent from another terminal or send a message by using an information processing terminal such as a smartphone, a mobile phone terminal, or a tablet terminal.

Further, in an information processing terminal, an agent system that automatically responds to a user's message has been proposed. Regarding such a system, for example, Patent Document 1 below describes an agent creation device capable of creating an agent by combining agent creation data such as clothes, hairstyle, belongings, and personality as desired by the user.

Further, Patent Document 2 below describes a device that interprets a cultural background including a user's hobbies / preferences from user information and causes an agent to perform an operation corresponding to the cultural background. Further, in Patent Document 3 below, the voice interface agent learns an unpredictable incidental condition that appears peculiar to a situation that generates a predetermined emotion, and the predetermined incidental condition is satisfied under a new situation that satisfies the learned incidental condition. An emotion generating device capable of recalling emotions is described.

Japanese Unexamined Patent Publication No. 2003-186589 Japanese Unexamined Patent Publication No. 2003-106846 Japanese Unexamined Patent Publication No. 11-265239

However, even if the character of the agent can be set in various ways, the agent exists only as a dialogue partner of the user, and the user cannot get the experience of becoming the character of the agent he / she likes.

Therefore, the present disclosure proposes an information processing system and an information processing method capable of further enhancing the entertainment of the agent system by allowing the user to experience the character of the agent through the agent.

According to the present disclosure, an agent storage unit that stores a phonograph database and an utterance phrase database corresponding to a plurality of types of characters, and a selection signal for selecting a specific character are received via a user's client terminal, and the specific character is specified. Based on the communication unit that transmits the utterance phrase corresponding to the utterance phrase database of the character and the user's message received via the communication unit, the specific phonology database corresponding to the specific character is used. A conversion message converted into a character's voice is generated, a utterance phrase of the specific character corresponding to the user's message is generated using the utterance phrase database, and the generated conversion message and utterance phrase are generated by the client. We propose an information processing system equipped with a control unit that controls the return to the terminal.

According to the present disclosure, the processor stores a phonological database and an utterance phrase database corresponding to a plurality of types of characters in an agent storage unit, and receives a selection signal for selecting a specific character via a user's client terminal. At the same time, the communication unit transmits the utterance phrase corresponding to the utterance phrase database of the specific character, and the phonetic element corresponding to the specific character is based on the user's message received via the communication unit. A conversion message converted into the voice of the specific character is generated using the database, and an utterance phrase of the specific character corresponding to the message of the user is generated using the utterance phrase database, and the generated conversion is performed. We propose an information processing method including controlling by a control unit to return a message and an utterance phrase to the client terminal.

As described above, according to the present disclosure, it is possible to further enhance the entertainment of the agent system by allowing the user to experience the character of the agent himself / herself through the agent.

It should be noted that the above effects are not necessarily limited, and either in combination with or in place of the above effects, any of the effects shown herein, or any other effect that can be grasped from this specification. May be played.

It is a figure explaining the outline of the information processing system by one Embodiment of this disclosure. It is a figure which shows the whole structure of the communication control system by this embodiment. It is a block diagram which shows an example of the configuration of the voice agent server by this embodiment. It is a figure which shows the structural example of the dialogue processing part by this embodiment. It is a flowchart which shows the generation process of the conversation DB by this embodiment. It is a flowchart which shows the generation process of the phoneme DB by this embodiment. It is a flowchart which shows the dialogue control processing by this embodiment. It is a figure explaining the data structure example of the conversation DB by this embodiment. It is a flowchart which shows the update process of the conversation DB by this embodiment. It is a flowchart which shows the conversation data migration process from the personalization layer to the common layer by this embodiment. It is a figure explaining the transfer of conversation data to the conversation DB for basic dialogue by this embodiment. It is a flowchart which shows the conversation data transfer process to the basic dialogue DB by this embodiment. It is a figure which shows an example of the advertisement information registered in the advertisement DB by this embodiment. It is a flowchart which shows the insertion process of the advertisement content by this embodiment. It is a figure which shows the structural example of the dialogue processing part by this embodiment. It is a figure which shows the configuration example of the user management part by this embodiment. It is a figure which shows the configuration example of the automatic utterance control unit by this embodiment. It is a figure which shows the configuration example of the scenario management part by this embodiment. It is a sequence diagram which shows the purchase process of the agent application by this Embodiment. It is a figure which shows the display screen example at the time of purchase of the agent application by this embodiment. It is a figure which shows the account registration screen example by this embodiment. It is a figure which shows the example of the main screen by this embodiment. It is a figure explaining the voice conversion process by this Embodiment. It is a sequence diagram which shows the voice conversion processing by this embodiment. It is a figure explaining the formation by AR transformation by this embodiment. It is a sequence diagram which shows the automatic utterance processing according to the place by this embodiment. It is a sequence diagram which shows the automatic utterance processing according to the person attribute and the facial expression by this embodiment. It is a sequence diagram which shows the automatic utterance processing according to the user action by this embodiment. It is a sequence diagram which shows the automatic utterance processing according to the psychological state by this embodiment. It is a flowchart which shows the scenario acquisition process by this Embodiment. It is a figure which shows the screen display example in the client terminal until the scenario purchase by this embodiment. It is a figure which shows the screen display example in the client terminal until the scenario purchase by this embodiment. It is a sequence diagram which shows the scenario participation registration process by this embodiment. It is a figure explaining the scenario composition by this embodiment. It is a figure which shows an example of the scenario participation screen by this embodiment. It is a figure which shows the example of the scenario holding start in the case where the agent App by this embodiment is running in the foreground. It is a figure which shows the example of the scenario holding start usual in the case where the agent App by this embodiment is not started. It is a sequence diagram which shows the scenario execution process by this Embodiment. It is a figure which shows the example of the display screen at the time of the event occurrence by this embodiment. It is a sequence diagram which shows the execution process of the event triggered by the position of the user by this embodiment. It is a sequence diagram which shows the execution process of the event triggered by the position of a plurality of users by this embodiment. It is a figure which shows the example of the display screen which prompts a user to act to hold a camera by this embodiment. It is a figure explaining the overlay display of another character by this embodiment. It is a sequence diagram which shows the execution process of the event triggered by the output result from each sensor by this embodiment. It is a sequence diagram which shows the determination process of the scenario clear by this embodiment. It is a figure which shows the example of the notification screen at the time of clearing a scenario by this embodiment.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

In addition, the explanation shall be given in the following order.
1. 1. Outline of the information processing system according to the embodiment of the present disclosure 2. Configuration 2-1. System configuration 2-2. Server configuration 3. System operation processing 3-1. Conversation data registration process 3-2. Phoneme DB generation process 3-3. Dialogue control processing 3-4. Conversation DB update process 3-5. Advertisement insertion process 4. Audio output control processing 4-1. Configuration 4-2. Operation processing (4-2-1. Agent purchase processing)
(4-2-2. Voice conversion processing)
(4-2-3. Automatic utterance processing)
(4-2-4. Scenario acquisition process)
(4-2-5. Scenario execution processing)
5. Summary

<< 1. Outline of information processing system according to one embodiment of the present disclosure >>
The information processing system according to the embodiment of the present disclosure makes it possible to further enhance the entertainment of the agent system by allowing the user to experience the character of the agent through the agent. Hereinafter, an outline of the information processing system according to the present embodiment will be described with reference to FIG.

FIG. 1 is a diagram illustrating an outline of an information processing system according to an embodiment of the present disclosure. The dialogue with the agent is performed, for example, via a client terminal 1 such as a smartphone owned by the user. The client terminal 1 has a microphone and a speaker, and enables a voice dialogue with the user.

Here, as described above, even if the agent character can be set in various ways, the agent exists only as a dialogue partner of the user, and it is not possible to obtain the experience that the user himself / herself can become the character of the agent he / she likes. I could not do it.

Therefore, in the present embodiment, in addition to the agent having an automatic conversation with the user by voice, it is possible to further enhance the entertainment of the agent system by allowing the user to experience the character of the agent through the agent.

For example, when the agent program is running, the information processing system according to the present embodiment converts the spoken voice W ₁ into the voice W ₂ of the agent character 10 when the user speaks, as shown in FIG. Play from the user's earphones (voice conversion process). In this way, since the words spoken by the user are heard in the voice of the agent character 10 (for example, a hero), the user can obtain the experience of becoming the agent character 10. Further, the information processing system according to the present embodiment may output a predetermined phrase corresponding to the user's utterance voice W ₁ in succession with the voice of the agent character 10 following the converted voice W ₂ (FIG. 1). Speech voice W ₃ ) (automatic speech processing) shown in. The predetermined phrase is, for example, a phrase registered in association with a keyword (or phrase) included in the utterance voice W _{1 of the} user. By automatically uttering such a phrase following the converted voice of the user's voice, it is possible to further enhance the experience of impersonating the character. Further, the information processing system according to the present embodiment may also reproduce a keyword (or phrase) included in the spoken voice W ₁ and a sound effect corresponding to the corresponding predetermined phrase. Further, the automatic utterance processing according to the present embodiment is not limited to the utterance voice of the user, and may reproduce phrases and sound effects corresponding to the user's actions, moving places, facial expressions, date and time, and the like.

Further, the information processing system according to the present embodiment can allow the user himself / herself to participate in the scenario by impersonating the agent character 10. For example, when a user purchases a favorite agent and scenario in advance, the information detected from the positioning unit (GPS, etc.) and various sensors (acceleration sensor, gyro sensor, geomagnetic sensor, microphone, camera, etc.) mounted on the client terminal 1 Scenarios are developed according to the user's real-world situation based on. Specifically, various events are held according to the progress of the scenario, and the user can experience the event by speaking the action according to the event.

Further, in the present embodiment, it is also possible to hold an event to meet a person who has become another agent character who participates in the scenario. For example, when a scenario program that includes a scene where characters meet at a specific time and place is being executed, a special event will occur if the person who has become each character moves to a specific place at a specific time. .. Specifically, for example, in an information processing system, each client terminal 1 converts the voice of the other party into the voice of the other party's agent character and reproduces it from an earphone or the like, so that users can talk with each other as characters appearing in the scenario. You may be able to enjoy it. Further, in the present embodiment, when the user holds the client terminal 1 over the other party and the other party's figure is captured by the camera of the client terminal 1, the image of the other party's agent character is superimposed and displayed on the other party's figure. It is possible to intuitively provide the agent character of the other user not only audibly but also visually.

The information processing system (agent system) according to the present embodiment is not limited to the voice agent that responds by voice, and may be a text-compatible agent that responds on a text basis at the client terminal 1.

Hereinafter, the system configuration of the information processing system according to the present embodiment, the basic configuration of each device, and the operation processing will be specifically described. Hereinafter, the information processing system according to the present embodiment will be referred to as a communication control system.

<< 2. Configuration >>
<2-1. System configuration>
Subsequently, the overall configuration of the communication control system according to the present embodiment described above will be described with reference to FIG. FIG. 2 is a diagram showing an overall configuration of a communication control system according to the present embodiment.

As shown in FIG. 2, the communication control system according to the present embodiment includes a client terminal 1 and an agent server 2.

The agent server 2 connects to the client terminal 1 via the network 3 and transmits / receives data. Specifically, the agent server 2 generates a response voice to the utterance voice that is picked up by the client terminal 1 and transmitted, and transmits the response voice to the client terminal 1. The agent server 2 has a phoneme DB (database) corresponding to one or more agents, and can generate a response voice with the voice of a specific agent. Here, the agent may be a character such as a manga, an animation, a game, a drama, a movie, a celebrity, a celebrity, a historical person, or the like, and is not specified to an individual, for example, by generation. It may be an average person. The agent may also be an animal or anthropomorphic character. Further, the agent may be a person who reflects the personality of the user himself / herself, or a person who reflects the personality of the user's friends, family, acquaintances, and the like.

Further, the agent server 2 can generate a response content that reflects the character of each agent. The agent server 2 can provide various services such as user schedule management, message transmission / reception, and information provision through a dialogue with the user via the agent.

The client terminal 1 is not limited to a smartphone as shown in FIG. 2, for example, a mobile phone terminal, a tablet terminal, a PC (personal computer), a game machine, a wearable terminal (smart eyeglass, smart band, smart watch, smart neck). Etc.) etc. Further, the client terminal 1 may be a robot.

The outline of the communication control system according to the present embodiment has been described above. Subsequently, the configuration of the agent server 2 of the communication control system according to the present embodiment will be specifically described with reference to FIG.

<2-2. Agent server 2>
FIG. 3 is a block diagram showing an example of the configuration of the agent server 2 according to the present embodiment. As shown in FIG. 3, the agent server 2 includes a voice agent I / F (interface) 20, a dialogue processing unit 30, a phoneme storage unit 40, a conversation DB generation unit 50, a phoneme DB generation unit 60, and an advertisement insertion processing unit 70. It has an advertisement DB 72 and a feedback acquisition processing unit 80.

The voice agent I / F 20 functions as a voice data input / output unit, a voice recognition unit, and a voice generation unit. As the input / output unit, a communication unit that transmits / receives to / from the client terminal 1 via the network 3 is assumed. The voice agent I / F20 can receive the user's uttered voice from the client terminal 1 and convert it into text by voice recognition. Further, the voice agent I / F 20 converts the agent's response text data (text) output from the dialogue processing unit 30 into voice using the phoneme data corresponding to the agent, and the generated agent's response voice is generated by the client terminal 1. Send to.

The dialogue processing unit 30 functions as an arithmetic processing unit and a control device, and controls the overall operation in the agent server 2 according to various programs. The dialogue processing unit 30 is realized by an electronic circuit such as a CPU (Central Processing Unit) or a microprocessor. Further, the dialogue processing unit 30 according to the present embodiment functions as a basic dialogue processing unit 31, a character A dialogue processing unit 32, a person B dialogue processing unit 33, and a person C dialogue processing unit 34.

The character A dialogue processing unit 32, the person B dialogue processing unit 33, and the person C dialogue processing unit 34 realize a dialogue specialized for each agent. Here, "character A", "person B", and "person C" are given as examples of agents, but the present embodiment is not limited to this, and each dialogue that realizes a dialogue specialized for a larger number of agents. It may have a processing unit. The basic dialogue processing unit 31 realizes a general-purpose dialogue that is not specialized for each agent.

Here, a basic configuration common to the basic dialogue processing unit 31, the character A dialogue processing unit 32, the person B dialogue processing unit 33, and the person C dialogue processing unit 34 will be described with reference to FIG.

FIG. 4 is a diagram showing a configuration example of the dialogue processing unit 300 according to the present embodiment. As shown in FIG. 4, the dialogue processing unit 300 includes a question sentence search unit 310, an answer sentence generation unit 320, a phoneme data acquisition unit 340, and a conversation DB 330. The conversation DB 330 stores conversation data in which question text data and answer text data are combined. The agent-specific dialogue processing unit stores agent-specific conversation data in the conversation DB 330, and the general-purpose dialogue processing unit stores general-purpose conversation data (that is, basic conversation) that is not agent-specific in the conversation DB 330. Data) is saved.

The question sentence search unit 310 recognizes the user's question voice (an example of the spoken voice) output from the voice agent I / F20 and searches the conversation DB 330 for question text data that matches the textualized question text. The answer sentence generation unit 320 extracts the answer sentence data stored in association with the question sentence data searched by the question sentence search unit 310 from the conversation DB 330, and generates the answer sentence data. The phoneme data acquisition unit 340 acquires phoneme data for converting the answer sentence generated by the answer sentence generation unit 320 into a voice from the phoneme storage unit 40 of the corresponding agent. For example, in the case of the character A dialogue processing unit 32, the phoneme data for reproducing the answer sentence data by the voice of the character A is acquired from the character A phoneme DB 42. Then, the dialogue processing unit 300 outputs the generated answer sentence data and the acquired phoneme data to the voice agent I / F20.

The phoneme storage unit 40 stores a phoneme database for generating voice for each agent. The phoneme storage unit 40 can be realized by a ROM (Read Only Memory) and a RAM (Random Access Memory). In the example shown in FIG. 3, the basic phoneme DB41, the character A phoneme DB42, the person B phoneme DB43, and the person C phoneme DB44 are stored. In each phoneme DB, for example, a phoneme piece and a prosody model which is control information thereof are stored as phoneme data.

The conversation DB generation unit 50 has a function of generating the conversation DB 330 of the conversation processing unit 300. For example, the conversation DB generation unit 50 collects expected question sentence data, collects answer sentence data corresponding to each question, and then saves the question sentence data and the answer sentence data as a set. Then, when a predetermined number of conversation data (a set of question sentence data and answer sentence data, for example, 100 sets) is collected, the conversation DB generation unit 50 registers it in the conversation DB 330 as an agent conversation data set.

The phoneme DB generation unit 60 has a function of generating a phoneme DB stored in the phoneme storage unit 40. For example, the phoneme DB generation unit 60 analyzes the voice information read out from a predetermined text, decomposes it into a phoneme piece and a phoneme model which is control information thereof, and when a predetermined number or more of voice information can be collected, the phoneme DB is used as phoneme data. Perform the process of registering in.

The advertisement insertion processing unit 70 has a function of inserting advertisement information into the dialogue of the agent. The advertisement information to be inserted can be extracted from the advertisement DB 72. In the advertisement DB 72, advertisement information requested from the provider (vendor, supplier) of a company or the like (for example, advertisement content such as text, image, voice, advertiser, advertisement period, advertisement target person, etc.) is registered. There is.

The feedback acquisition processing unit 80 has a function of inserting a question for acquiring feedback into the dialogue of the agent and obtaining feedback from the user.

The configuration of the agent server 2 according to the present embodiment has been specifically described above. The configuration of the agent server 2 according to the present embodiment is not limited to the example shown in FIG. For example, each configuration of the agent server 2 may be configured by another server on the network.

Subsequently, the basic operation processing of the communication control system according to the present embodiment will be described with reference to FIGS. 5 to 14.

<< 3. System operation processing >>
<3-1. Conversation data registration process>
FIG. 5 is a flowchart showing the generation process of the conversation DB 330 according to the present embodiment. As shown in FIG. 5, first, the conversation DB generation unit 50 saves the assumed question sentence (step S103).

Next, the conversation DB generation unit 50 saves (paired) answer sentences corresponding to the question sentences (step S106).

Next, the conversation DB generation unit 50 determines whether or not a predetermined number of pairs of question sentences and answer sentences (also referred to as conversation data) have been collected (step S109).

Then, when a predetermined number of pairs of question sentences and conversation sentences are collected (step S109 / Yes), the conversation DB generation unit 50 registers a data set composed of a large number of pairs of question sentences and answer sentences in the conversation DB 330 (step). S112). As an example of a pair of a question sentence and an answer sentence, for example, the following is assumed.

Question and answer pair example Pair 1
Question: Good morning.
Answer: How are you doing today?
Pair 2
Question: What's the weather today?
Answer: Today's weather is XX.

Such a pair can be registered in the conversation DB 330 as conversation data.

<3-2. Phoneme DB generation process>
FIG. 6 is a flowchart showing a phoneme DB generation process according to the present embodiment. As shown in FIG. 6, first, the phoneme DB generation unit 60 displays an example sentence (step S113). For the display of example sentences, for example, an example sentence necessary for generating phoneme data is displayed on a display of an information processing terminal (not shown).

Next, the phoneme DB generation unit 60 records the voice read out from the example sentence (step S116) and analyzes the recorded voice (step S119). For example, voice information read aloud by a person in charge of voice of an agent is collected by a microphone of an information processing terminal, and the phoneme DB generation unit 60 receives and stores the voice information, and further performs voice analysis.

Next, the phoneme DB generation unit 60 generates a prosody model based on the voice information (step S122). The prosodic model extracts prosodic parameters indicating prosodic characteristics of speech (for example, pitch of sound, strength of sound, speech speed, etc.), and is different for each individual.

Next, the phoneme DB generation unit 60 generates a phoneme piece (phoneme data) based on the voice information (step S125).

Next, the phoneme DB generation unit 60 saves the prosody model and the phoneme piece (step S128).

Subsequently, the phoneme DB generation unit 60 determines whether or not a predetermined number of prosodic models and phoneme pieces have been collected (step S131).

Then, when a predetermined number of prosody models and phoneme pieces are collected (step S131 / Yes), the phoneme DB generation unit 60 registers the prosody model and phoneme pieces in the phoneme storage unit 40 as a phoneme database for a predetermined agent (step S131 / Yes). Step S134).

<3-3. Dialogue control process>
FIG. 7 is a flowchart showing the dialogue control process according to the present embodiment. As shown in FIG. 7, first, the voice agent I / F20 confirms whether or not the user's question voice and the agent ID have been acquired (step S143). The agent ID is identification information indicating a specific agent such as character A, person B, and person C. The user can purchase phoneme data for each agent, for example, the ID of the agent purchased at the time of purchase processing is stored in the client terminal 1.

Next, when the user's question voice and agent ID are acquired (step S146 / Yes), the voice agent I / F20 voice-recognizes the question voice and converts it into text (step S149). The voice agent I / F20 outputs a textualized question sentence to the dialogue processing unit of the specific agent specified by the agent ID. For example, in the case of "agent ID: character A", the voice agent I / F 20 outputs a textualized question sentence to the character A dialogue processing unit 32.

Next, the dialogue processing unit 30 searches the conversation DB of the specific agent designated by the agent ID for a question text that matches the textualized question text (step S152).

Next, when there is a matching question (step S155 / Yes), the character A dialogue processing unit 32 acquires the answer sentence data corresponding to the question (stored in pairs) from the conversation DB of the specific agent. (Step S158).

On the other hand, when there is no matching question (step S155 / No), a question sentence matching the textualized question sentence is searched from the conversation DB of the basic dialogue processing unit 31 (step S161).

When there is a matching question sentence (step S161 / Yes), the basic dialogue processing unit 31 acquires the answer sentence data corresponding to the question (stored in pairs) from the conversation DB of the basic dialogue processing unit 31. (Step S167).

On the other hand, when there is no matching question sentence (step S164 / No), the basic dialogue processing unit 31 inputs the answer sentence data (for example, the answer sentence such as "I do not understand the question") when there is no matching question sentence. Acquire (step S170).

Next, the character A dialogue processing unit 32 refers to the phoneme DB of the specific agent designated by the agent ID (here, the character A phoneme DB 42), and the phoneme data of the character A for generating the voice of the answer sentence data is obtained. Acquired (step S173).

Next, the acquired phoneme data and the answer sentence data are output to the voice agent I / F20 (step S176).

Then, the voice agent I / F20 voices (speech synthesis) the answer sentence data (text) using the phoneme data and transmits it to the client terminal 1 (step S179). On the client terminal 1, the answer sentence is reproduced by the voice of the character A.

<3-4. Conversation DB update process>
Next, the update process of the conversation DB 330 of each dialogue processing unit 300 will be described. In the present embodiment, it is possible to grow the conversation DB 330 by talking with the user.

First, a supplementary explanation will be given with reference to FIG. 8 regarding a data configuration example of the conversation DB 330. FIG. 8 is a diagram illustrating a data configuration example of the conversation DB 330 according to the present embodiment. As shown in FIG. 8, each conversation DB 330 has two layers, a personalized layer 331 and a common layer 332. For example, in the case of the conversation DB 330A for character A, conversation data reflecting the character and characteristics of character A is held in the common layer 332A. On the other hand, the personalization layer 331A holds conversation data customized for the user by talking with the user. That is, when the character A phonetic element DB 42 and the character A dialogue processing unit 32 are provided (sold) to the user as a set, a certain user X and the user Y initially interact with the same character A (held in the common layer 332A). However, as the conversation continues, the conversation data customized for each user is accumulated in the personalized layer 331A for each user. As a result, it becomes possible to provide a dialogue with the character A according to the preference of each of the user X and the user Y.

The conversation data can also be customized for the user when the agent "person B" is an average generational person such as character A who does not have a specific personality. That is, for example, when "person B" is "a person in his twenties", the average conversation data in his twenties is held in the common layer 332B, and customized conversation data is provided for each user by continuing the dialogue with the user. It is retained in the personalization layer 331B of. Further, the user can select and purchase favorite phoneme data such as "male", "female", "high voice", and "low voice" as the voice of the person B from the person B phoneme DB43.

Specific processing when customizing the conversation DB 330 will be described with reference to FIG. FIG. 9 is a flowchart showing an update process of the conversation DB 330 according to the present embodiment.

As shown in FIG. 9, first, the voice agent I / F20 acquires (receives) a user's question voice from the client terminal 1 and converts it into text by voice recognition (step S183). The textualized data (question text data) is output to the dialogue processing unit (here, for example, the character A dialogue processing unit 32) of the specific agent specified by the agent ID.

Next, the character A dialogue processing unit 32 determines whether or not the question text data is a predetermined command (step S186).

Next, in the case of a predetermined command (step S186 / Yes), the character A dialogue processing unit 32 registers the answer sentence data specified by the user in the personalized layer 331A of the conversation DB 330A as a pair with the question sentence data (step S189). ). The predetermined command may be, for example, words such as "NG" and "setting". For example, the conversation DB of character A can be customized by the following conversation flow.

User: "Good morning"
Character A: "Good morning"
User: "NG. Answer that you are fine and do your best"
Character A: "Be fine and do your best"

In the above conversation flow, "NG" is a predetermined command, and the character A dialogue processing unit 32 issues the user-specified answer sentence data "Energetic and do your best" after the user issues "NG". , The question text data "Good morning" is paired and registered in the personalized layer 331A of the conversation DB 330A.

On the other hand, when the command is not a predetermined command (step S186 / No), the character A dialogue processing unit 32 searches the character A conversation DB 330A for the answer sentence data held as a pair with the question sentence data. When the answer sentence data held as a pair with the question sentence data is not held in the conversation DB 330A for character A, that is, when the user's question is a question without an answer sentence (step S192 / Yes). The character A dialogue processing unit 32 registers the answer sentence data specified by the user in the personalization layer 331A as a pair with the question sentence (step S195). For example, the conversation DB of character A can be customized by the following conversation flow.

User: "How are you?"
Character A: "I don't understand the question" (example of answer data when there is no corresponding answer)
User: "When you ask,'How are you?', Answer'I'm fine today.'"
Character A: "I'm fine today"

In the flow of the above conversation, "How are you? Since there is no answer sentence data that is held as a pair with "", "I don't understand the question", which is an example of answer data when there is no corresponding answer, is acquired by the character A dialogue processing unit 32, and the corresponding character A It is output to the voice agent I / F20 together with the phoneme data of, and is reproduced by the client terminal 1. Next, when the user-specified answer sentence "I'm fine today" is input, the character A dialogue processing unit 32 asks the question sentence data "How are you?". ] And register it in the personalized layer 331A.

If the question has an answer sentence (step S192 / No), the character A dialogue processing unit 32 acquires the answer sentence data and outputs it to the voice agent I / F20 together with the corresponding character A phonetic data. , The answer sentence is reproduced by the voice of the character A on the client terminal 1 (step S198).

Next, the transfer of conversation data from the personalized layer to the common layer will be described with reference to FIG. FIG. 10 is a flowchart showing a conversation data transfer process from the personalized layer to the common layer according to the present embodiment. Here, as an example, the conversation data transfer process from the personalized layer 331A of the character A dialogue processing unit 32 to the common layer 332A will be described.

As shown in FIG. 10, first, the character A dialogue processing unit 32 periodically searches for the personalized layer 331A for each user (step S203), and has substantially the same conversation pair (question text data and answer text). A pair of data) is extracted (step S206). Conversation pairs with substantially the same content are, for example, a pair of the question sentence "How are you?" And the answer sentence "I'm fine today!" And the question sentence "How are you?" And the answer sentence "I'm fine today." The pair of "!" Can be judged to be a conversation pair with substantially the same content, only the difference is whether the question sentence is a polite word or not.

Next, when the character A dialogue processing unit 32 extracts a predetermined number or more of conversation pairs from the personalized layer 331A for each user (step S209 / Yes), the character A dialogue processing unit 32 registers the conversation pairs in the common layer 332A (for each user). (Step S212).

In this way, by migrating the conversation pairs having substantially the same contents in the personalization layer 331 for each user to the common layer 332, it is possible to grow the common layer 332 (expand the conversation pairs).

Further, in the present embodiment, it is also possible to grow the conversation DB for basic dialogue by migrating the conversation data from the conversation DB (specifically, the common layer) of the specific agent to the conversation DB for basic dialogue. FIG. 11 is a diagram illustrating the transfer of conversation data to the basic conversation conversation DB 330F according to the present embodiment. For example, when user X and user Y each select (purchase) agent "character A" and user Z select (purchase) agent "person B", as shown in FIG. 11, the character of user X The conversation DB330A-X for A, the conversation DB330A-Y for character A of user Y, and the conversation DB330B-Z for person B of user Z may exist in the dialogue processing unit 30. In this case, each personalized layer 331A-X, 331A-Y, 331B-Z is registered with its own (customized) conversation pair according to the dialogue with each user X, user Y, and user Z. (See FIG. 9). Next, if there are a predetermined number of substantially the same conversation pairs in the personalized layers 331A-X and 331A-Y of the same agent, they are registered in the common layers 332A-X and 332A-Y for each user (see FIG. 10).

Then, the dialogue processing unit 30 is a higher-level basic when a predetermined number or more of substantially the same conversation pairs are extracted from the common layers 332A-X, 332A-Y, and 332B-Z of a plurality of agents (which may include different agents). The conversation pair is transferred to the conversation DB330F for dialogue. The basic dialogue DB 330F is a conversation DB owned by the basic dialogue processing unit 31. This makes it possible to grow the conversation DB330F for basic dialogue (expand conversation pairs). Such data migration processing will be specifically described with reference to FIG. FIG. 12 is a flowchart showing a conversation data transfer process to the basic dialogue DB 330F according to the present embodiment.

As shown in FIG. 12, first, the dialogue processing unit 30 periodically searches a plurality of common layers 332 of the conversation DB 330 (step S223), and extracts substantially the same conversation pair (step S226).

Next, when a predetermined number or more of substantially the same conversation pairs are extracted from the plurality of common layers 332 (step S229 / Yes), the dialogue processing unit 30 registers the conversation pairs in the basic conversation conversation DB 330F (step S232). ..

In this way, the conversation pair for basic dialogue DB330F is grown (expanded conversation pair) by migrating the conversation pair having substantially the same content in the common layer 332 of the conversation DB330 among the plurality of agents to the conversation DB330F for basic dialogue. It becomes possible.

<3-5. Advertisement output processing>
Subsequently, the process of inserting the advertisement information by the advertisement insertion processing unit 70 will be described with reference to FIGS. 13 to 14. In the present embodiment, the advertisement insertion processing unit 70 can insert the advertisement information stored in the advertisement DB 72 into the remarks of the agent. Advertising information can be registered in advance in the advertising DB 72. FIG. 13 is a diagram showing an example of advertising information registered in the advertising DB 72 according to the present embodiment.

As shown in FIG. 13, the advertisement information 621 includes, for example, an agent ID, a question text, an advertisement content, a condition, and a probability. The agent ID specifies an agent who speaks the advertisement content, the question text specifies the question text of the user who triggers to insert the advertisement content, and the advertisement content is the advertisement text to be inserted into the dialogue of the agent. Further, the condition is a condition for inserting the advertisement content, and the probability indicates the probability of inserting the advertisement content. For example, in the example shown in the first row of FIG. 13, in the dialogue with the agent "Character A", when the word "chocolate" is included in the question text from the user under 30 years old, "BB company new The chocolate that was released contains a lot of milk and is delicious. " In addition, since the user may be annoyed if the advertisement content is inserted every time a question sentence that serves as a trigger is issued, the probability of inserting the advertisement may be set in the present embodiment. Such a probability may be determined according to the advertising fee. For example, the higher the advertising fee, the higher the probability is set.

Such an advertisement content insertion process will be specifically described with reference to FIG. FIG. 14 is a flowchart showing an advertisement content insertion process according to the present embodiment.

As shown in FIG. 14, first, the advertisement insertion processing unit 70 monitors the dialogue between the user and the agent (specifically, the dialogue processing by the dialogue processing unit 30) (step S243).

Next, the advertisement insertion processing unit 70 determines whether or not a question sentence having the same content as the question sentence registered in the advertisement DB 72 appears in the dialogue between the user and the agent (step S246).

Next, when a question sentence having the same content appears (step S246 / Yes), the advertisement insertion processing unit 70 confirms the condition and probability of the advertisement insertion associated with the corresponding question sentence (step S249).

Subsequently, the advertisement insertion processing unit 70 determines whether or not the advertisement is currently in a state of being able to be put out based on the condition and the probability (step S252).

Next, when the advertisement can be put out (step S252 / Yes), the advertisement insertion processing unit 70 suspends the dialogue processing by the dialogue processing unit 30 (step S255) and inserts the advertisement content into the dialogue (step S258). ). Specifically, for example, the advertisement content is inserted into the agent's answer to the user's question.

Then, the dialogue (conversation text data) including the advertisement content is output from the dialogue processing unit 30 to the voice agent I / F20, transmitted from the voice agent I / F20 to the client terminal 1, and reproduced by the voice of the agent (step S261). ). Specifically, for example, the advertisement content can be presented to the user as a statement of the character A by the following conversation.

User: "Good morning"
Character A: "Good morning! How are you doing today?"
User: "I'm fine. I want to eat something delicious."
Character A: "It seems that the grilled meat at the CC store is delicious."

In the above conversation, first, in response to the user's question sentence "Good morning", the corresponding answer sentence "Good morning! How are you doing today?" Searched from the conversation DB of the character A is output by voice. Next, because the user's question "I'm fine. I want to eat something delicious" includes the question "I want to eat something delicious" that triggers the insertion of advertisements (2nd row in Fig. 13). (See eyes), the advertisement insertion processing unit 70 performs the advertisement insertion processing, and outputs an answer sentence such as "It seems that the grilled meat of the CC store is delicious" by the voice of the character A.

As described above, conversation data registration processing, phoneme DB generation processing, dialogue control processing, conversation DB update processing, and advertisement insertion processing have been described as basic operation processing of the communication control system according to the present embodiment.

Further, the dialogue processing unit 30 of the communication control system according to the present embodiment can provide the agent character with an experience of being the user himself / herself by using the voice utterance function of the agent, and can enhance the enjoyment of the agent system. To do. The voice output control process of the dialogue processing unit 30 according to the present embodiment will be specifically described with reference to FIGS. 15 to 42.

<< 4. Audio output control processing >>
<4-1. Configuration>
First, the configuration of the dialogue processing unit 30a that performs the voice output control processing according to the present embodiment will be described with reference to FIG.

FIG. 15 is a diagram showing a configuration example of the dialogue processing unit 30a according to the present embodiment. As shown in FIG. 15, the dialogue processing unit 30a includes a basic dialogue processing unit 31, a character A dialogue processing unit 32, a person B dialogue processing unit 33, a person C dialogue processing unit 34, a user management unit 35, and an automatic utterance control unit 36. , And a scenario management unit 37.

The basic dialogue processing unit 31, the character A dialogue processing unit 32, the person B dialogue processing unit 33, and the person C dialogue processing unit 34 are agents corresponding to the user's utterances, as described with reference to FIGS. 3 and 4. It has a function to generate a response of. The basic dialogue processing unit 31 generates a general-purpose response that is not specific to the agent, and the character A dialogue processing unit 32, the person B dialogue processing unit 33, and the person C dialogue processing unit 34 are each agent character (character A, person). Generate a response specialized for each of B and C).

(User Management Department 35)
The user management unit 35 manages (registers, changes, updates, deletes) user information. FIG. 16 shows a configuration example of the user management unit 35 according to the present embodiment. As shown in FIG. 16, the user management unit 35 includes a login management unit 351, a user information DB 352, a face information registration unit 353, and a user location information registration unit 354.

The login management unit 351 performs login authentication of the user in response to a request from the client terminal 1. Specifically, for example, the login management unit 351 refers to the account information (account name, password) input by the user on the client terminal 1 with the user information DB 352, and performs login authentication. The user information DB 352 stores user information including user ID, account information, user attribute information (birthday, gender, zip code, etc.), face information, purchased (user-owned) agent ID, scenario ID, and the like. These user information are transmitted from the client terminal 1 and registered via, for example, the voice agent I / F20 (see FIG. 3) of the agent server 2.

The face information registration unit 353 registers the user's face information in the user information DB 52. The user's face information is, for example, the result of analyzing the user's face image captured by a camera provided in the client terminal 1, and can be transmitted from the client terminal 1 to the agent server 2. A face image may be transmitted from the client terminal 1 and analyzed by the face information registration unit 353.

The user position information registration unit 354 registers the user's current position information in the user information DB 52. The user's current position information is positioned by a positioning unit such as GPS (Global Positioning System) provided in the client terminal 1, and is periodically transmitted to the agent server 2.

(Automatic utterance control unit 36)
The automatic utterance control unit 36 has a function of controlling automatic utterance by an agent. FIG. 17 shows a configuration example of the automatic utterance control unit 36 according to the present embodiment. As shown in FIG. 17, the automatic utterance control unit 36 includes a user voice extraction unit 361, a phoneme data acquisition unit 362, a position information acquisition unit 363, a phrase search unit 364, a phrase DB 365, and an information analysis unit 366.

The user voice extraction unit 361 analyzes the input voice information and extracts the user voice. Such voice information is picked up by, for example, the microphone of the client terminal 1 and transmitted from the client terminal 1 to the agent server 2 via the network. The agent server 2 outputs the voice information received by the voice agent I / F 20 to the dialogue processing unit 30a. The client terminal 1 collects peripheral voice information continuously, periodically, or at a predetermined timing and transmits it to the agent server 2. The user voice extraction unit 361 converts the extracted user voice into text (generation of utterance text) and outputs it to the phoneme data acquisition unit 362.

The position information acquisition unit 363 acquires the user's current position information and outputs it to the phrase search unit 364. The current position information of the user can be transmitted from the client terminal 1.

The information analysis unit 366 analyzes various information indicating the user status transmitted from the client terminal 1 and outputs the analysis result to the phrase search unit 364. Specifically, for example, the information analysis unit 366 uses facial expressions from the face information transmitted from the client terminal 1 (the face information of the current user or the face information of a person in the vicinity, which is analyzed based on the captured image). To analyze. In addition, the information analysis unit 366 analyzes the user behavior (running, jumping, sleeping, etc.) from the acceleration information (information detected by the acceleration sensor) transmitted from the client terminal 1. In addition, the information analysis unit 366 analyzes environmental sounds (noise around the user, etc.) from the voice information (voice information picked up by the microphone) transmitted from the client terminal 1. In addition, the information analysis unit 366 is nervous from the biometric information (information detected by the pulse sensor, heart rate sensor, sweating sensor, body temperature sensor, blood pressure sensor, brain wave sensor, etc.) transmitted from the client terminal 1. , Angry, sad, happy, etc.). Then, the information analysis unit 366 outputs the analysis result (the situation of the user or a peripheral person) to the phrase search unit 364.

The phrase search unit 364 uses a phrase (also referred to as an utterance phrase) according to the user position acquired by the position information acquisition unit 363, the facial expression of the user or the other party analyzed by the information analysis unit 366, the user behavior, or the user situation. It searches from the phrase DB 365 and outputs the search result to the utterance data acquisition unit 362. Narration and sound effects may be associated with the phrase. Further, the phrase DB 365 stores phrase data for each agent character. Here, Table 1 below shows an example of phrase data of the agent character "hero" stored in the phrase DB 365. As shown in Table 1 below, the phrase DB 365 stores the situation and the phrase or sound effect in association with each other. In the example shown in Table 1 below, the "situation" is associated with one sensor type, but the present embodiment is not limited to this, and the "situation" is comprehensively based on the analysis results of a plurality of sensors. It may be judged. Further, in the present embodiment, "phrases, sound effects" corresponding to cases where a plurality of situations (place, facial expression, time, state, etc.) satisfy the conditions may be associated with each other.

The phoneme data acquisition unit 362 acquires the phoneme data for converting the utterance text output from the voice extraction unit 361, the phrase output from the phrase search unit 364, and the narration into voice from the phoneme storage unit 40 of the corresponding agent. To do. For example, the phoneme data acquisition unit 362 acquires the phoneme data of the agent according to the utterance text in order to convert the user voice into the voice of the agent specified by the user. A selection signal for selecting a specific agent character may be transmitted from the client terminal 1.

(Scenario Management Department 37)
The scenario management unit 37 manages each scenario associated with the agent character. FIG. 18 shows a configuration example of the scenario management unit 37 according to the present embodiment. As shown in FIG. 18, the scenario management unit 37 has a data management unit 371, a scenario execution unit 372, an information analysis unit 373, and a scenario DB 374.

The data management unit 371 manages the scenario stored in the scenario DB 374, such as registration, change, update, and deletion. One or more scenario data corresponding to each agent character is stored in the scenario DB 374. The scenario data is given a title, a synopsis, a purchase price, and the like as incidental information, and further includes data related to an event (also referred to as a scenario event). The data related to the event includes a situation (place, user behavior, facial expression, user utterance, etc.) that triggers the occurrence of the event, an event holding time (holding period), and the like.

The scenario execution unit 372 controls to present the voice or image of the agent character to the user according to the scenario in which the user is participating. Specifically, the scenario execution unit 372 controls the voice agent I / F20 to transmit the presentation information such as voice and images based on the scenario to the client terminal 1 via the network. Further, the scenario execution unit 372 makes a trigger determination of the event included in the scenario based on the analysis result by the information analysis unit 373, and when the event occurs, the presentation information such as the sound or image of the event is transmitted to the voice agent I. Controls to transmit from / F20 to the client terminal 1 via the network.

The information analysis unit 373 analyzes various information indicating the user status transmitted from the client terminal 1 and outputs the analysis result to the phrase search unit 364. Various information indicating the user situation includes, for example, position information, face information (face information of the current user or face information of a person in the vicinity analyzed based on the captured image), acceleration information, voice information, and biological information. And so on.

Here, Table 2 below shows an example of event data included in the scenario data stored in the scenario DB 374. As shown in Table 2 below, the event data is associated with trigger generation conditions, event contents, and actions.

The configuration of the dialogue processing unit 30a according to the present embodiment has been specifically described above. Subsequently, the operation processing according to the present embodiment will be specifically described with reference to FIGS. 19 to 42.

<4-2. Operation processing>
(4-2-1. Agent purchase process)
FIG. 19 is a sequence diagram showing a purchase process of the agent application according to the present embodiment. Here, the agent application is software used to enjoy automatic dialogue by a specific agent character on the client terminal 1, and the purchase of the agent application can be said to be "purchase of the agent". Hereinafter, the application is also referred to as "App".

As shown in FIG. 19, first, the client terminal 1 downloads and installs an arbitrary (that is, user-selected) Agent App from the website of the application shop provided by the agent server 2 (step S270). .. The client terminal 1 and the agent server 2 are connected via a network. Data transmission / reception of the agent server 2 may be performed by the voice agent I / F20.

Next, the client terminal 1 starts (first starts) the agent application (step S273). If the user account has not been registered (step S276 / No), the agent server 2 is requested to register the account (step S279).

Next, the user management unit 35 (see FIG. 16) of the agent server 2 registers new account information in the user information DB 352 in response to a request from the client terminal 1 (step S282). The new account information is account name, password, user attribute information (gender, date of birth, nickname), etc., which is input by the user on the client terminal 1 and transmitted together with a request for account registration processing.

On the other hand, when the user account is already registered (step S276 / Yes), the client terminal 1 requests the agent server 2 to perform the login process (step S285).

Next, when a new account is registered or a login processing request is received, the login management unit 351 of the agent server 2 refers to the user information DB 352 and performs the login processing of the account (step S288). In the login processing request, the account name and password entered by the user on the client terminal 1 are transmitted, so the login management unit 351 refers to the user information DB 352 and collates them.

Next, when the login process is normally completed, the agent server 2 transmits a login completion notification from the voice agent I / F 20 to the client terminal 1 via the network (step S291).

Next, the client terminal 1 activates the camera of the client terminal 1 (or a camera provided in a communicable external terminal existing in the vicinity), images the user's face, and performs face information from the captured image (face image). (Step S294). The face information may be the analysis result of the captured image (face image) or the face image itself.

Next, the client terminal 1 transmits the face information to the agent server 2 (step S297), and the agent server 2 registers the face information in the user information DB 352 (step S300).

Subsequently, the client terminal 1 accepts the user's selection as to whether or not to execute the agent application in the background (step S303), and transmits the selected content as setting information to the agent server 2 (step S306).

Next, the agent server 2 saves the setting information in the user information DB 352 (step S309). The setting information may be stored in the storage unit of the client terminal 1.

Then, the client terminal 1 displays the main screen according to the activated agent App (step S312).

The processing for purchasing the Agent App and starting the Agent App for the first time has been explained above. Here, an example of a display screen on the client terminal 1 at the time of purchasing the agent application and starting the agent application for the first time will be described with reference to FIGS. 20 to 22.

FIG. 20 is a diagram showing an example of a display screen at the time of purchasing the agent application according to the present embodiment. On the screen 100 shown on the left side of FIG. 20, the titles of a plurality of agent apps that are candidates for purchase are displayed. The title of each agent App shown on the screen 100 is, for example, the name of the agent character. For example, when the user wants to purchase the agent character "Powerful Man", the user selects "Agent App" Powerful Man "" on the screen 100. In this case, the screen 100 transitions to the screen 101 shown in the center of FIG.

On the screen 101, an account information input field, an account creation button, and a "Click here if you have an account" button are displayed. If you have already registered an account to use the application shop, the user selects the "If you have an account, click here" button. In this case, the screen 101 transitions to the screen 102 shown on the right side of FIG.

On the screen 102, an account name input field, a password input field, and a login button are displayed. The user enters the registered account name (also referred to as user name / ID and login name / ID) and password, and selects the login button. When the login button is selected, the client terminal 1 makes a login processing request to the agent server 2 together with the entered account name and password.

On the other hand, if the account is not registered, the account name and the like are input on the screen 101, and the agent server 2 is requested to create the account. FIG. 21 is a diagram showing an example of an account registration screen according to the present embodiment. As shown in the screen 103 on the left side of FIG. 21, when the account name and the like are input and the "Create account" button is selected, the client terminal 1 sends a request for account registration processing to the agent server 2 together with the input information. To do.

When the account registration process is normally completed on the agent server 2, a screen 104 notifying that the account creation is completed is displayed as shown on the right side of FIG. 21. On the screen 104, a text such as "Would you like to register your face information?" Is displayed. When the "Yes" button is selected, the camera of the client terminal 1 is activated and the user's face is imaged. , And face information extraction (analysis) is performed. The extracted face information is transmitted to the agent server 2 and registered as user information.

FIG. 22 is a diagram showing an example of a main screen according to the present embodiment. When the login process and the account registration process are completed in the initial startup of the agent application, a screen 105 for confirming whether or not to start the agent application is displayed as shown on the left side of FIG. 22. To start, the user selects the "Yes" button displayed on the screen 105. A check box for setting whether or not to execute in the background is also displayed on the screen 105. If the user wants to run the agent app in the background, check it. The client terminal 1 transmits to the agent server 2 whether or not the check box is checked as background setting information for whether or not to execute the check box.

When the "Yes" button on the screen 105 is selected, the main screen 106 of the agent app is displayed. Here, for example, the image of the hero character "Powerful Man" is displayed on the display of the client terminal 1, and the voice and theme song of "Powerful Man" such as "I will protect this city!" Are displayed on the speaker of the client terminal 1. Played from.

(4-2-2. Voice conversion processing)
Subsequently, the voice conversion process according to the present embodiment will be described with reference to FIGS. 23 to 24. FIG. 23 is a diagram illustrating the voice conversion process according to the present embodiment. In the present embodiment, as shown in FIG. 23, when the user's utterance voice W ₄ is picked up by the microphone of the client terminal 1 (or a wearable device existing in the vicinity of communicating with the client terminal 1), this is picked up by the dialogue processing unit. The automatic utterance control unit 36 of 30a converts the voice W ₅ of a specific agent character and reproduces it from an earphone or the like worn by the user. The specific agent character is, for example, a character corresponding to the agent app running on the client terminal 1, and the operation of starting the agent app is recognized as a substantial user selection of the agent character, and a selection signal is output. It is sent to the agent server 2. In this way, the user can impersonate the agent character by hearing his / her utterance voice as the voice of the agent character.

FIG. 24 is a sequence diagram showing the voice conversion process according to the present embodiment. As shown in FIG. 24, first, when the client terminal 1 picks up the sound data by the microphone (step S320), the client terminal 1 transmits the picked up sound data to the agent server 2 (step S323). At this time, the client terminal 1 may also transmit a selection signal indicating the agent character selected by the user. These data can be transmitted from the client terminal 1 to the agent server 2 via the network 3 and received by the voice agent I / F20 (functioning as a communication unit) of the agent server 2.

Next, the agent server 2 analyzes the sound data by the user voice extraction unit 361 (see FIG. 17) of the automatic utterance control unit 36, and extracts the user voice. When the user voice can be extracted, the agent server 2 converts it into the voice of a specific agent character (here, for example, "hero character") selected by the user (step S326). More specifically, the agent server 2 extracts phoneme data for converting the user voice sentence extracted by the user voice extraction unit 361 of the dialogue processing unit 30a into a text by the voice of the agent character, and the phoneme data of the dialogue processing unit 30a. It is acquired by the data acquisition unit 362. Then, based on the user voice sentence output from the dialogue processing unit 30a and the phonetic data of the corresponding specific agent character, the voice agent I / F20 converts the user voice sentence into a voice of the specific agent character (voice conversion). ), The voiced data (voice data) is transmitted from the voice agent I / F 20 to the client terminal 1 via the network (step S329).

Next, the client terminal 1 reproduces the voice data of the voice color of the hero character converted by the agent server 2 from an earphone or the like (an example of a speaker) (step S332).

Although the voice conversion process has been described above, the present embodiment is not limited to the above-mentioned auditory pretext, and it is possible to provide a visual prescription experience. Hereinafter, description will be made with reference to FIG.

FIG. 25 is a diagram for explaining the visual premise of AR (Augmented Reality) transformation according to the present embodiment. In the present embodiment, as shown in FIG. 25, for example, a camera provided in the client terminal 1 captures the user's own face, and a screen 107 in which the face image of the agent character is superimposed and displayed on the captured user's face image is displayed. By generating and displaying it, it is possible to provide a visual experience. At this time, the agent server 2 performs face recognition based on the captured face image, that is, confirms whether or not the face information matches the face information registered in the user information DB 3532, and if they match, the face image of the agent character is displayed. The images may be superimposed and displayed. Further, a visual pretending experience may be provided in addition to the above-mentioned auditory pretending experience of voice conversion.

(4-2-3. Automatic utterance processing)
Subsequently, the automatic utterance processing of the specific agent character according to the user situation will be described with reference to FIGS. 26A to 26D. The user situation is assumed to be, for example, the user's location, personal attributes and facial expressions, behavioral state (behavior recognition), psychological state, and the like.

FIG. 26A is a sequence diagram showing automatic utterance processing according to a location according to the present embodiment. As shown in FIG. 26A, first, the client terminal 1 acquires the current position information by GPS or the like (step S340), and transmits the current position information to the agent server 2 (step S342). Such acquisition and transmission of the current position information can be performed periodically, for example, when the agent program is running on the client terminal 1.

Next, the automatic speech control unit 36 of the agent server 2 uses the phrase search unit 364 to input the phrase corresponding to the position information (location) of the client terminal 1 transmitted from the client terminal 1 and acquired by the position information acquisition unit 363 to the phrase DB365. (Step S344). For example, as shown in Table 1 above, phrases and sound effects associated with specific locations (XX cities, Y parks, Z stations, etc.) and general locations (stations, post offices, parks, seas, etc.) are searched. Will be done.

Next, when a phrase corresponding to the location is found (that is, the search is hit) (step S346 / Yes), the automatic utterance control unit 36 converts the searched phrase into the voice of a specific agent character specified by the user, for example, a hero character. (Step S348). Specifically, the phoneme data acquisition unit 362 acquires the phoneme data of the hero character for converting the phrase into voice from the phoneme storage unit 40, and the acquired phoneme data and the phrase are output to the voice agent I / F20 to make a voice. The agent I / F20 performs a phrase voice processing (for example, voice synthesis).

Subsequently, the agent server 2 transmits the voice data generated by the voice agent I / F 20 to the client terminal 1 (step S350).

Then, the client terminal 1 reproduces the voice data received from the agent server 2, that is, the voice data in which a predetermined phrase is voiced with the voice of the hero character (step S352). As a result, when the user moves to a predetermined place, the speaker of the client terminal 1 reproduces the phrase corresponding to the place with the voice of a specific agent character.

FIG. 26B is a sequence diagram showing each automatic utterance process according to a person's attributes and facial expressions according to the present embodiment. As shown in FIG. 26B, first, the client terminal 1 activates the camera, acquires the captured image (step S354), and transmits the captured image to the agent server 2 (step S356). Such camera activation, imaging, and transmission may be performed periodically, for example, when the agent program is activated on the client terminal 1. The cameras are an outward camera (also called an out-camera) in which the client terminal 1 captures the user's surroundings such as the line-of-sight direction of the user, and an inward-facing camera (also referred to as an in-camera) in which the user who operates the client terminal 1 is imaged. It is assumed that it will be called).

Next, the automatic utterance control unit 36 of the agent server 2 determines whether or not the captured image transmitted from the client terminal 1 is captured by the in-camera (step S358). Whether or not the image is captured by the in-camera can be determined from, for example, the metadata attached to the captured image.

Next, in the case of the in-camera (step S358 / Yes), it is determined that the user is shown in the captured image, and the phrase search unit 364 changes the facial expression based on the face image analysis result by the information analysis unit 366. The corresponding phrase is searched from the phrase DB 365 (step S360). For example, as shown in Table 1 above, a phrase or sound effect corresponding to the case where the user smiles, or a phrase or sound effect corresponding to the case where the user has an angry face is searched.

On the other hand, when it is not an in-camera (step S358 / No), that is, when it is captured by an out-camera, it is determined that a person around the user (for example, a person facing the user) is captured in the captured image, and the phrase The search unit 364 searches the phrase DB 365 for phrases corresponding to the attributes (age, gender, atmosphere, etc.) and facial expressions of the person facing each other based on the face image analysis result by the information analysis unit 366 (step S362). For example, as shown in Table 1 above, phrases, sound effects, etc. corresponding to the case where the person to be faced is a woman are searched.

Next, when a phrase corresponding to the person attribute or facial expression is found (that is, the search is hit) (step S364 / Yes), the automatic utterance control unit 36 uses the searched phrase as a user-specified specific agent character, for example, a hero character. It is converted into a voice color (step S366). The conversion process to the specific agent character is the same as the process described in step S348 above.

Subsequently, the agent server 2 transmits the voice data generated by the voice agent I / F 20 to the client terminal 1 (step S368).

Then, the client terminal 1 reproduces the voice data received from the agent server 2, that is, the voice data in which a predetermined phrase is voiced with the voice of the hero character (step S370). As a result, a phrase corresponding to the facial expression or attribute at that time is reproduced from the speaker of the client terminal 1 with the voice of a specific agent character, triggered by the facial expression of the user or the attribute or facial expression of the other party facing the user.

FIG. 26C is a sequence diagram showing each automatic utterance process according to the user action according to the present embodiment. As shown in FIG. 26C, first, the client terminal 1 acquires the acceleration sensor information by the acceleration sensor (step S372) and transmits the acceleration sensor information to the agent server 2 (step S374). The transmission of the acceleration sensor information may be performed periodically, for example, when the agent program is running on the client terminal 1.

Next, the automatic speech control unit 36 of the agent server 2 performs an action recognition process by the information analysis unit 366 based on the acceleration sensor information transmitted from the client terminal 1, and the phrase corresponding to the action state indicated by the action recognition result. Is searched by the phrase search unit 364 with reference to the phrase DB 365 (step S376). For example, as shown in Table 1 above, phrases and sound effects associated with a running state or a sleeping state are searched. Although acceleration sensor information is used here as sensor data used for behavior recognition processing, the present embodiment is not limited to this, and is detected by various sensors such as a gyro sensor and a geomagnetic sensor in addition to the acceleration sensor. Data may be used.

Next, when a phrase corresponding to the action state is found (that is, the search is hit) (step S378 / Yes), the automatic utterance control unit 36 converts the searched phrase into the voice of a specific agent character specified by the user, for example, a hero character. Convert (step S380).

Subsequently, the agent server 2 transmits the voice data generated by the voice agent I / F 20 to the client terminal 1 (step S382).

Then, the client terminal 1 reproduces the voice data received from the agent server 2, that is, the voice data in which a predetermined phrase is voiced with the voice of the hero character (step S384). As a result, when the user is in a predetermined action state, the speaker of the client terminal 1 reproduces a phrase corresponding to the action state at that time with the voice of a specific agent character.

FIG. 26D is a sequence diagram showing each automatic utterance process according to the psychological state according to the present embodiment. As shown in FIG. 26D, first, the client terminal 1 detects the biometric information of the user by the biometric sensor (step S386) and transmits the biometric information to the agent server 2 (step S388). The transmission of biometric information can be performed periodically, for example, when the agent program is running on the client terminal 1.

Next, the automatic utterance control unit 36 of the agent server 2 analyzes the biometric information transmitted from the client terminal 1 by the information analysis unit 366, and outputs a phrase corresponding to the user's psychological state (that is, emotion) obtained by the analysis. , The phrase search unit 364 refers to the phrase DB 365 and searches (step S390). For example, as shown in Table 1 above, phrases and sound effects associated with a state in which the pulse is fast and tense are searched. Biosensors are various sensors that detect, for example, pulse, heartbeat, blood pressure, sweating amount, respiration, brain wave, myoelectricity, and the like. The information analysis unit 366 analyzes the psychological state of the user, that is, emotions such as joy, anger, sadness, tension, and excitement, based on such biometric information.

Next, when a phrase corresponding to the psychological state is found (that is, the search is hit) (step S392 / Yes), the automatic utterance control unit 36 uses the searched phrase as the voice of a specific agent character specified by the user, for example, a hero character. Convert (step S394).

Subsequently, the agent server 2 transmits the voice data generated by the voice agent I / F 20 to the client terminal 1 (step S396).

Then, the client terminal 1 reproduces the voice data received from the agent server 2, that is, the voice data in which a predetermined phrase is voiced with the voice of the hero character (step S398). As a result, a phrase corresponding to the psychological state at that time is reproduced from the speaker of the client terminal 1 with the voice of a specific agent character triggered by the psychological state of the user.

The automatic utterance control process of the agent according to the user situation has been described above. The automatic utterance control process according to the present embodiment is not limited to the above-mentioned example, and even if a phrase corresponding to at least one or more user situations among places, facial expressions, actions, emotions, user utterances, date and time, etc. is searched Alternatively, the phrase search may be performed in a predetermined order (higher priority order, etc.) among a plurality of user situations.

(4-2-4. Scenario acquisition process)
Subsequently, the scenario mode according to the present embodiment will be described. In the dialogue processing unit 30a according to the present embodiment, in addition to the above-mentioned conversion of the user voice into the agent character voice and the automatic utterance of the agent character according to the user situation, the user becomes an agent character and a scenario (story) ) Can provide an experience of participating. The acquisition of the scenario program used to provide such an experience will be described below with reference to FIGS. 27-29.

FIG. 27 is a sequence diagram showing a scenario acquisition process according to the present embodiment. As shown in FIG. 27, when "scenario list" is first selected from the menu screen displayed on the display unit of the client terminal 1 (step S410), the client terminal 1 sets the scenario list for the agent server 2. An acquisition request is made (step S413). Here, for example, a request is made to acquire a scenario list of the agent character "hero character" Powerful Man "" purchased by the user.

Next, the scenario management unit 37 of the agent server 2 acquires a list of scenarios associated with the hero character from the scenario DB 374 (step S416), and transmits the list to the client terminal 1 from the voice agent I / F 20 via the network (step S419). ).

Next, the client terminal 1 displays the scenario list received from the agent server 2 on the display unit (step S422), and accepts the user to select the scenario (step S425).

Next, the client terminal 1 transmits selection information indicating the scenario selected by the user to the agent server 2 (step S428).

Next, the scenario management unit 37 of the agent server 2 determines whether or not the scenario selected by the user has been purchased (step S431), and if not purchased (step S431 / Yes), the client terminal 1 is notified. On the other hand, the display instruction of the purchase screen of the scenario is given (step S434).

Next, the client terminal 1 displays the scenario purchase screen (step S437). For example, the user taps the purchase button displayed on the scenario purchase screen to decide to purchase the scenario.

Subsequently, when the purchase of the scenario is decided (step S440 / Yes), the client terminal 1 requests the agent server 2 to purchase the scenario (step S443).

Then, the agent server 2 performs the scenario purchase process (step S446). The scenario purchase process can be performed by, for example, a payment process (billing process in the application) using a credit card or electronic money. Further, the purchase of the scenario by the user is registered as user information in the user information DB 352 by the user management unit 35.

When the purchase is postponed after the scenario purchase screen is displayed (step S440 / No), for example, the user may tap the back button or the like to return to the menu list screen and consider acquiring another scenario.

Here, FIGS. 28 and 29 show an example of screen display on the client terminal 1 until the scenario purchase. For example, FIG. 28 left shows the main screen 110 when a user purchases a hero character. When the user selects the menu button 111 displayed on the main screen 110, the menu screen 112a is displayed as shown in the center of FIG. 28. Then, when the item of the "scenario list" included in the menu displayed on the menu screen 112a is selected, the scenario list screen 113 is displayed as shown on the right side of FIG. 28. The scenarios that can be participated in are arranged on the scenario list screen 113, and the user selects a scenario that he / she wants to purchase. Scenarios for which the participation period has already expired or sold out are grayed out and cannot be selected. For example, on the screen 113, scenarios 113a and 113b are displayed as selectable scenarios, and scenarios 113c are grayed out as non-selectable scenarios.

Next, when the user selects, for example, scenario 113a “Battle in XX City”, the purchase screen 114 as shown on the left of FIG. 29 is displayed. On the purchase screen 114, the title, synopsis, purchase amount, purchase button 114a, participation place and period of the scenario are displayed, and the user confirms the content and amount of the scenario, participation place, period and the like. For example, multiple events are prepared for the scenario, and the goal is to clear the scenario by clearing all of them. The "participation place" is a place that triggers an event, and it is possible to participate in the scenario even if you are not at that place, but there are few events that will be triggered and it is difficult to clear the scenario. It becomes.

Then, when the user selects the purchase button 114a, the purchase decision screen 115 shown on the right side of FIG. 29 is displayed, and when the "Yes" button is selected, the purchase process (settlement process) is performed by the agent server 2, and the scenario purchase is completed. ..

Subsequently, the participation registration process for the scenario will be described with reference to FIG. FIG. 30 is a sequence diagram showing a scenario participation registration process according to the present embodiment. The process shown in FIG. 30 is a process that is continuously performed when the scenario selected by the user has already been purchased in step S431.

The scenario management unit 37 of the agent server 2 searches for a scenario in which a specific agent character purchased by the user, for example, a hero character, is not registered in the scenario selected by the user (step S450). Here, the scenario configuration according to the present embodiment will be described with reference to FIG. 31.

FIG. 31 is a diagram illustrating a scenario configuration according to the present embodiment. One or more characters appear in the scenario according to this embodiment, and one user is registered in one character. However, since it is assumed that a plurality of users have purchased the agent program of the same character, as shown in FIG. 31, a plurality of participating groups are set for each scenario, and user registration is performed for each participating group. For example, in scenario # 1, the participating groups # 1-1, # 1-2, # 1-3, etc. are linked to a plurality of participating groups, and the users are sorted so that the characters do not overlap in the participating groups. .. Specifically, at present, users who can be "Character A" and "Character B" are registered in the participating group # 1-1 of scenario # 1, but "Character C" is vacant. Is. Also, in the participating group # 1-2 of the same scenario, "Character A" is vacant. In addition, in the participating groups # 1-3 of the same scenario, "Character A", "Character B", and "Character C" are vacant. When the user selects scenario # 1, the scenario management unit 37 grasps the user registration status in the participating group of the scenario, and searches for the participating group in which the character purchased by the user is not registered (vacant). For example, if the user's character is "character A", the participating group # 1-2 is searched, if it is "character B", the participating group # 1-3 is searched, and if it is "character C", the participating group # 1 is searched. Search for -1. It is possible for the user to simultaneously register for participation in different scenarios (for example, scenario # 1 and scenario # 2) with the same character.

Next, the scenario management unit 37 transmits the searched scenario information of the participating group to the client terminal 1 (step S453).

Next, the client terminal 1 displays the received scenario information on the display unit (step S456). On the scenario information display screen, a participation button is displayed along with a synopsis of the scenario, a place of participation, and details of the date and time. Here, FIG. 32 shows an example of the scenario participation screen according to the present embodiment. The screen 116 shown on the left side of FIG. 32 is registered for participation in the scenario selected on the scenario list screen 113 (see FIG. 28) or the scenario in which the purchase is decided on the scenario purchase decision screen 115 (see FIG. 29) and the purchase process is completed. It is a screen that accepts the operation for. On the screen 116, for example, the title and synopsis of the selected scenario, the display of the purchased scenario, the participation place, the participation period information, and the participation button 116a are displayed. The user can input the intention to participate in this scenario by selecting the participation button 116a.

Next, when the participation button is selected and the user's intention to participate is input (step S459 / Yes), the client terminal 1 requests the agent server 2 to participate in this scenario (participation group) (step S462). ..

Next, the scenario management unit 37 of the agent server 2 performs the participation registration process for the user's scenario in response to the request from the client terminal 1 (step S465). The information of user registration to the character cou appearing in the participating group of each scenario may be registered in the scenario DB 374 (see FIG. 18) or in the user information DB 352 (see FIG. 16).

Next, when the scenario in which the user has registered to participate is before the start time (step S468 / Yes), the scenario management unit 37 notifies the user of the start time (step S471), and the client terminal 1 starts the scenario. Notify the user of the time via a display screen or the like (step S474). As an example of the display screen, for example, the screen 117 shown in the upper right of FIG. 32 can be mentioned. On the screen 17, the scenario title, the fact that the participation has been reserved, and the countdown to the scenario start time are displayed.

Subsequently, when the start time of the scenario is reached (step S477 / Yes), or when the scenario registered for participation is already after the start time of the scenario (step S468 / No) and the scenario is being held (step S469 / Yes). ), The scenario management unit 37 notifies the user of the start notification of the scenario (step S480). For example, when the scenario registered for participation is already after the start time (that is, when the scenario is being held), as shown in FIG. 32, when the participation button 116a is tapped, as shown in the screen 118 shown in the lower right of FIG. 32. , "Participating!" Is displayed along with the scenario title, and the scenario starts immediately. Further, when the holding start time of the scenario already registered for participation is reached, the holding start notification as shown in FIG. 33 or FIG. 34 is given, for example. If the holding period of the scenario has ended (step S469 / No), the participation registration process ends because participation in the scenario is not possible.

FIG. 33 is a diagram showing a customary example of starting scenario holding when the agent App according to the present embodiment is running in the foreground. As shown on the left side of FIG. 33, when the screen 120 of the agent app (for example, the main screen) is displayed and the start time of the scenario is reached, the screen 120 of the agent app is displayed as shown on the right side of FIG. 33. , A pop-up display 120a indicating that the scenario is started is displayed. When the user confirms the content of the notification and presses the "OK" button, the pop-up display 120a is closed.

FIG. 34 is a diagram showing a customary example of starting scenario holding when the agent App according to the present embodiment is not started. As shown on the left side of FIG. 34, when the agent app is not started (for example, when the home screen 122 is displayed), when the start time of the scenario is reached, the scenario is started on the home screen 122. The pop-up display 122a (Push notification) is displayed. When the user confirms the notification content and presses the "Open" button, the agent app is started and the main screen 123 of the agent app is displayed as shown on the right side of FIG. 34.

Next, the client terminal 1 notifies the user that the scenario is started via a display screen or the like (step S483).

Then, the scenario management unit 37 starts the scenario execution process (step S486).

For example, when a scenario such as "Battle in XX City" is started, for example, an image of an agent character (for example, a hero character) is displayed on the display unit of the client terminal 1, and further, the narration and the voice of the hero character are heard from the earphones and the like. Is output as.
・ Scenario voice narration "On October 12, 2015, the stage is XX city. The warriors were exhausted by the fierce battle that took place ..."
Hero character "XX city is waiting for me!"
Narration "It is our mission to defeat eight enemies in that XX city."

Next, when the scenario is in progress, the hero character is displayed on the client terminal 1, and the hero character automatically speaks or plays a corresponding sound effect according to the user's situation. The automatic utterance according to the situation is the same as the process described with reference to FIGS. 26A to 26D.
-Automatic voice user status: By analyzing location information, it is recognized that the user has moved to XX city.
Hero character "Is this the XX city? Where are the enemies !?"

Then, under certain conditions, an event prepared in advance for the scenario occurs. When an event occurs, the hero character or narration will notify you that the event has occurred and the actions required to clear the event. The event is cleared when the user performs the action correctly. Scenario execution processing such as event occurrence according to such conditions will be described in detail below.

(4-2-5. Scenario execution processing)
The scenario management unit 37 according to the present embodiment uses the scenario execution unit 372 to generate a scenario event (referred to as an “event” in the present embodiment) triggered by a user's utterance, movement location, action (behavior), or the like. Performs processing such as instructing the user of a predetermined action for clearing the event. Hereinafter, a specific description will be given with reference to FIGS. 35 to 42.

-Generation of Event Triggered by User Voice FIG. 35 is a sequence diagram showing execution processing of an event triggered by user voice according to the present embodiment. As shown in FIG. 35, first, the client terminal 1 collects ambient sound data by a microphone (step S490), and transmits the collected sound data to the agent server 2 (step S493).

Next, the scenario management unit 37 of the agent server 2 analyzes the sound data by the information analysis unit 373 and extracts the user voice. When the user voice is generated, the scenario execution unit 372 searches for an event corresponding to the user's utterance from the scenario in which the user is participating (step S496). The event search is performed by referring to the scenario data of the scenario in which the user is participating, which is stored in the scenario DB 374. As described above, each scenario includes one or more events, and data in which an event occurrence trigger (condition), an event content, and an action for clearing the event are associated with each other is stored in the scenario DB 374 as scenario data. Has been done. Specific examples of scenario data are as shown in Table 2 above. The scenario management unit 37 refers to the event data as shown in Table 2 and searches for an event triggered by the user voice (that is, the utterance content) (occurrence condition).

Next, the scenario execution unit 372 extracts the information of the searched event from the scenario data (step S499), and transmits the information regarding the corresponding designated action (designated action for clearing the event) to the client terminal 1 (step). S502). If the corresponding event cannot be searched, the scenario execution unit 372 may not particularly transmit information to the client terminal 1, or may give a hint (trigger suggestion) for event occurrence. You may. Further, the sound data collected by the client terminal 1 and transmitted to the agent server 2 may be periodically performed during the scenario holding period.

Next, the client terminal 1 instructs the user to perform a designated action for clearing the event by displaying output, voice output, or the like (step S505). Here, a specific example of the display screen when an event occurs will be described with reference to FIG.

FIG. 36 is a diagram showing an example of a display screen when an event occurs according to the present embodiment. As shown on the left side of FIG. 36, for example, a certain utterance voice W ₆ of a user is converted into the voice W ₇ of an agent character by the above-mentioned voice conversion function, and the utterance voice W ₆ is a specific utterance and corresponds to an event. If is searched, the action to clear the event is instructed. For example, as shown on the right side of Fig. 36, the agent's utterance voice W ₈ such as "Jump up! Now!" Is output from earphones, etc., or a screen containing text such as "Jump!" And an image of the agent. 124 may be displayed on the display unit of the client terminal 1. This allows the user to perform a specified action corresponding to the event.

Subsequently, the client terminal 1 acquires the output result from each sensor that detects the user's behavior or the like (step S508), and transmits the output result of each sensor to the agent server 2 (step S511). The sensors that detect the user's behavior are, for example, an acceleration sensor, a gyro sensor, a geomagnetic sensor, a camera, and the like.

Next, the agent server 2 analyzes the output result from each sensor by the information analysis unit 366 (for example, analysis of behavior recognition), and determines whether or not the specified action is performed by the scenario execution unit 372 based on the analysis result. Determine (step S514).

Next, when it is determined that the specified action has been performed (step S514 / Yes), the scenario execution unit 372 determines that the corresponding event has been cleared (step S517), and issues an event to the client terminal 1. The fact that it has been cleared is transmitted (step S520). Further, the scenario execution unit 372 registers (updates) the event clear information in the scenario DB 374.

Then, the client terminal 1 notifies the user that the event has been cleared by displaying output, voice output, or the like (step S523).

As described above, in the present embodiment, a predetermined event is generated by triggering a specific utterance of the user, the user is urged to perform a predetermined action, and when the action is detected, the scenario is advanced assuming that the event is cleared. Can be made to.

-Generation of an event triggered by a movement location (user's position) FIG. 37 is a sequence diagram showing an event execution process triggered by a user's position according to the present embodiment. As shown in FIG. 37, first, the client terminal 1 acquires the current position information by GPS or the like (step S530), and transmits the acquired current position information to the agent server 2 (step S533).

Next, the scenario management unit 37 of the agent server 2 analyzes the location information by the information analysis unit 373, and identifies the location indicated by the location information. For example, the information analysis unit 373 refers to the map data associated with the landmark information, and refers to the name (place name, city name, building name, park name, etc.) and type (station, park, seaside, etc.) of the place where the user is currently located. , Post office, etc.). When the location can be specified, the scenario execution unit 372 searches for an event corresponding to the specified location from the scenarios in which the user is participating (step S536). The event search is performed by referring to the scenario data of the scenario in which the user is participating, which is stored in the scenario DB 374. The scenario management unit 37 refers to the event data as shown in Table 2 above, and searches for an event triggered by the user's current location (that is, the movement location).

Next, the scenario execution unit 372 extracts the information of the searched event from the scenario data (step S539), and transmits the information regarding the corresponding designated action (designated action for clearing the event) to the client terminal 1 (step). S542). The acquisition of the current position information by the client terminal 1 and the transmission to the agent server 2 may be performed periodically during the scenario holding period.

Next, the client terminal 1 instructs the user to perform a designated action for clearing the event by displaying output, voice output, or the like (step S545).

Subsequently, the client terminal 1 acquires the output result from each sensor that detects the user's behavior or the like (step S548), and transmits the output result of each sensor to the agent server 2 (step S551).

Next, the agent server 2 analyzes the output result from each sensor by the information analysis unit 366 (for example, analysis of behavior recognition), and determines whether or not the specified action is performed by the scenario execution unit 372 based on the analysis result. Determine (step S554).

Next, when it is determined that the specified action has been performed (step S554 / Yes), the scenario execution unit 372 determines that the corresponding event has been cleared (step S557), and issues an event to the client terminal 1. The fact that it has been cleared is transmitted (step S560). Further, the scenario execution unit 372 registers (updates) the event clear information in the scenario DB 374.

Then, the client terminal 1 notifies the user that the event has been cleared by display output, voice output, or the like (step S563).

As described above, in the present embodiment, a predetermined event is generated by using the position of the user as a trigger, the user is urged to perform a predetermined action, and when the action is detected, the scenario is advanced assuming that the event is cleared. Can be done.

-Generation of an event triggered by the encounter of a plurality of users (positions of a plurality of users) FIG. 38 is a sequence diagram showing an event execution process triggered by the positions of a plurality of users according to the present embodiment. As shown in FIG. 38, first, the client terminal 1 acquires the current position information by GPS or the like (step S570), and transmits the acquired current position information to the agent server 2 (step S572).

Next, the scenario management unit 37 of the agent server 2 determines whether or not another user who uses another character participating in the same scenario as an agent is near the user (step S573). The location information of each user participating in the scenario is periodically transmitted from the client terminal 1 and managed on the agent server 2 side. Further, the scenario management unit 37 may search for a user of another character who is in the vicinity when the user moves to a specific place. Further, the scenario management unit 37 may search for users of other unspecified characters participating in the same scenario.

Next, when it is determined that there is another user who uses another character as an agent nearby (step S573 / Yes), the scenario management unit 37 searches for the corresponding event (step S576). The scenario management unit 37 refers to the event data as shown in Table 2 above, and triggers an event (for example, "overlay display") triggered by "there are other characters participating in the same scenario nearby". To search for.

Next, the scenario execution unit 372 extracts the information of the searched event from the scenario data (step S579), and executes the event execution process. Here, for example, in the case of an event called "overlay display", the scenario execution unit 372 requests the client terminal 1 to acquire a face image of a nearby person (step S582).

Next, the client terminal 1 activates the camera in response to the request from the agent server 2 and instructs the user to hold the camera over a nearby person (step S585). Here, the action of activating the camera and holding it over a nearby person is the designated action for clearing the event.

Subsequently, the client terminal 1 captures the face of a nearby person with a camera to acquire a captured image (step S588), and transmits the captured image to the agent server 2 (step S591).

Next, the scenario management unit 37 of the agent server 2 analyzes the captured image by the information analysis unit 366 and recognizes the face of a person in the vicinity of the user (step S594). Further, the scenario execution unit 372 refers to the face information of another user determined to be in the vicinity of the user based on the position information in step S573 and the face recognition result based on the captured image, and refers to a person in the vicinity. Face recognition may be performed.

Next, when the face of a person in the vicinity can be recognized (step S594 / Yes), the scenario execution unit 372 acquires information on other characters in the vicinity determined in step S573 from the scenario DB 374 (step S597). ), Sent to the client terminal 1 (step S600). The character information includes an image of the character.

Subsequently, the client terminal 1 is based on the character information transmitted from the agent server 2 when the user holds the camera of the client terminal 1 over a nearby person (other user) and the through image is displayed on the display unit. Then, the image of the agent character that the other party is made up of is overlaid on the other party on the through image (step S603). This allows the user to meet other characters appearing in the same scenario in real space. The agent server 2 not only superimposes the character image of the other party on the through image of the other party, but also converts the voice of the other party into the voice of the other character and reproduces it from the user's earphone or the like. Good. Further, also on the client terminal 1 of the other user, the user's character may be superimposed and displayed on the user's through image, or the user's voice may be converted into the user's character's voice and reproduced. As a result, both users can meet and talk with each other as characters appearing in the same scenario.

Here, a specific example of overlay display of another character according to the present embodiment will be described with reference to FIGS. 39A and 39B. FIG. 39A is a diagram showing an example of a display screen prompting the user to hold the camera according to the present embodiment. The illustrated screen 125 is a guidance screen displayed on the display unit of the client terminal 1 in step S585, and includes an image of the agent character and a camera activation button 125a. In addition, in the tone of voice of the agent character, speech W _9, such as "I try held up close to the people to start the camera!" It may be played. As a result, the user can take a designated action for clearing the event, such as tapping the camera activation button 125a according to the guidance of the agent character to activate the camera and holding it over a nearby person.

FIG. 39B is a diagram illustrating overlay display of other characters according to the present embodiment. As shown in FIG. 39B, when the user holds the client terminal 1 over a nearby person, the through image captured by the camera of the client terminal 1 is displayed on the display unit of the client terminal 1, and the other user reflected in the through image. The image 126 on which the character image of the other party is superimposed in real time is displayed on the screen. At this time, the client terminal 1 earphones the voice obtained by converting the voice of the other user into the character voice of the other user by the agent server 2 and the predetermined phrase voice W ₁₀ of the character of the other user automatically spoken according to the situation. It may be reproduced from the above.

Next, the scenario execution unit 372 determines that the corresponding event has been cleared (step S606), and transmits to the client terminal 1 that the event has been cleared (step S609). Further, the scenario execution unit 372 registers (updates) the event clear information in the scenario DB 374.

Then, the client terminal 1 notifies the user that the event has been cleared by display output, voice output, or the like (step S612).

As described above, in the present embodiment, a predetermined event can be generated based on the positions of a plurality of users, triggered by the encounter of characters participating in the same scenario in the real space.

-Generation of an event triggered by an output result from each sensor FIG. 40 is a sequence diagram showing an event execution process triggered by an output result from each sensor according to the present embodiment. As shown in FIG. 40, first, the client terminal 1 acquires the output result from each sensor (step S620) and transmits it to the agent server 2 (step S623). Each sensor is, for example, an acceleration sensor, a gyro sensor, a geomagnetic sensor, a camera, etc., and is provided on a client terminal 1 or a wearable terminal (for example, a smart band, a smart watch, a smart eyeglass) that communicates with the client terminal 1. , Recognize user behavior.

Next, the scenario management unit 37 of the agent server 2 analyzes the output results of each sensor by the information analysis unit 373, and identifies the user's behavior. When the user's behavior (sleeping, waking up, running, walking, riding a train / bicycle / car, etc.) can be identified, the scenario execution unit 372 changes the user's behavior from the scenario in which the user is participating. Search for the corresponding event (step S626). The event search is performed by referring to the scenario data of the scenario in which the user is participating, which is stored in the scenario DB 374. The scenario management unit 37 searches for an event triggered by the user's action by referring to the event data as shown in Table 2 above.

Next, the scenario execution unit 372 extracts the information of the searched event from the scenario data (step S629), and transmits the information regarding the corresponding designated action (designated action for clearing the event) to the client terminal 1 (step). S632). The acquisition of the output result from each sensor by the client terminal 1 and the transmission to the agent server 2 may be performed periodically during the scenario holding period.

Next, the client terminal 1 instructs the user to perform a designated action for clearing the event by displaying output, voice output, or the like (step S635).

Subsequently, the client terminal 1 acquires the output result from each sensor that detects the user's behavior or the like (step S638), and transmits the output result of each sensor to the agent server 2 (step S641).

Next, the agent server 2 analyzes the output result from each sensor by the information analysis unit 366 (for example, analysis of behavior recognition), and determines whether or not the specified action is performed by the scenario execution unit 372 based on the analysis result. Determine (step S644).

Next, when it is determined that the specified action has been performed (step S644 / Yes), the scenario execution unit 372 determines that the corresponding event has been cleared (step S647), and issues an event to the client terminal 1. The fact that it has been cleared is transmitted (step S650). Further, the scenario execution unit 372 registers (updates) the event clear information in the scenario DB 374.

Then, the client terminal 1 notifies the user that the event has been cleared by displaying output, voice output, or the like (step S653).

As described above, in the present embodiment, a predetermined event is generated by using the user's action as a trigger, the user is urged to perform the predetermined action, and when the action is detected, the scenario is advanced assuming that the event is cleared. Can be done.

The execution process of the scenario event according to the present embodiment has been specifically described above. The triggers for generating a scenario event according to the present embodiment are the above-mentioned utterance (user voice), moving location (position information), encounter of multiple users (position information of multiple users), and output result of each sensor (user behavior). Alternatively, the condition may include at least one or more of the user's facial expression (captured image), date and time, and the like. For example, it may be a condition that a certain utterance is made at a specific place, or a condition that the person moves to a certain place at a specific time. Further, among the above-mentioned triggers, it may be determined whether or not an event has occurred in a predetermined order (preset order, high priority order, etc.).

The above-mentioned event can also occur when the agent application is not started (running in the background). When an event occurs, for example, a push notification will notify you (such as "Agent App notification event has occurred!"), And you can check the contents by starting the Agent App.

-Scenario clear One scenario includes, for example, multiple events, and it is required to clear all the events during the scenario holding period. Hereinafter, an example of scenario clearing according to the present embodiment will be described with reference to FIGS. 41 to 42.

FIG. 41 is a sequence diagram showing a scenario clear determination process according to the present embodiment. As shown in FIG. 41, first, the scenario management unit 37 of the agent server 2 determines whether or not all the events in the scenario in which the user is participating have been cleared (step S660).

Next, when it is determined that all the events have been cleared (step S660 / Yes), the scenario execution unit 372 determines that the scenario has been cleared (step S663), and indicates that the scenario has been cleared for the client terminal 1. Is transmitted (step S666). Further, the scenario execution unit 372 registers (updates) the scenario clear information in the scenario DB 374.

Then, the client terminal 1 notifies the user that the scenario has been cleared by display output, voice output, or the like (step S669). Here, FIG. 42 shows an example of a notification screen when the scenario is cleared according to the present embodiment.

In the illustrated example, a notification such as "Scenario # 1" Battle in XX City "has been cleared !!" and an OK button are displayed on the screen 128. As a result, the user knows that he / she has cleared all the events of the participating scenario # 1. Also, when the OK button is tapped, the display screen of the notification is closed, and the screen returns to, for example, the main screen of the agent app.

<< 5. Summary >>
As described above, in the communication control system according to the embodiment of the present disclosure, it is possible to further enhance the entertainment of the agent system by allowing the user to experience the character of the agent through the agent.

Although the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the present technology is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can come up with various modifications or modifications within the scope of the technical ideas described in the claims. Of course, it is understood that the above also belongs to the technical scope of the present disclosure.

For example, it is possible to create a computer program for exerting the functions of the client terminal 1 or the agent server 2 on the hardware such as the CPU, ROM, and RAM built in the client terminal 1 or the agent server 2 described above. .. Also provided is a computer-readable storage medium that stores the computer program.

Further, in the above-described embodiment, various functions are realized by the agent server 2 connected to the client terminal 1 via the Internet, but the present embodiment is not limited to this. For example, at least a part of each configuration of the agent server 2 shown in FIGS. 3 and 15 to 18 may be in the client terminal 1 (smartphone, wearable terminal, etc.). Further, all the configurations of the agent server 2 shown in FIGS. 3 and 15 to 18 may be provided in the client terminal 1 so that the client terminal 1 can perform all the processing.

In addition, the effects described herein are merely explanatory or exemplary and are not limited. That is, the techniques according to the present disclosure may exhibit other effects apparent to those skilled in the art from the description herein, in addition to or in place of the above effects.

The present technology can also have the following configurations.
(1)
An agent storage unit that stores phoneme databases and utterance phrase databases that support multiple types of characters,
A communication unit that receives a selection signal for selecting a specific character via the user's client terminal and transmits an utterance phrase corresponding to the utterance phrase database of the specific character.
Based on the user's message received via the communication unit, a converted message converted into the voice of the specific character is generated using the phoneme database corresponding to the specific character;
Further, the utterance phrase of the specific character corresponding to the message of the user is generated by using the utterance phrase database;
A control unit that controls the generated conversion message and utterance phrase to be returned to the client terminal.
Information processing system equipped with.
(2)
The information processing system according to (1) above, wherein the control unit generates an utterance phrase corresponding to the user's message based on the context of the user received via the communication unit and the utterance phrase database. ..
(3)
The information processing system according to (2) above, wherein the user's context is at least one of the user's position, face recognition, acceleration information, or biosensor information.
(4)
The information processing system according to (2) or (3) above, wherein the user's message is a user's uttered voice or a text of the uttered voice.
(5)
The information processing system further includes a scenario storage unit that stores a plurality of scenarios in which the user can participate as a character.
The control unit;
A scenario stored in the scenario storage unit is selected according to a scenario selection signal from the user received via the communication unit;
When the context of the user received via the communication unit matches the event occurrence condition described in the selected scenario, the user is controlled to notify the occurrence of a predetermined event. The information processing system according to any one of (4).
(6)
The content notified to the user when the predetermined event occurs includes information indicating a specific action.
The control unit determines whether or not the specific action has been executed based on the context of the user newly received via the communication unit, and if it determines that the specific action has been executed, the control unit issues a notification indicating event clear. The information processing system according to (5) above, which controls transmission to a user.
(7)
The information processing system according to (5) or (6) above, wherein the user's context used for determining the event occurrence condition is at least one of the user's position, utterance, acceleration information, or biosensor information. ..
(8)
The control unit receives each position information indicating the position of the client terminal of each user participating in the selected scenario by the communication unit, and another user who participates in the same scenario around the user. The information processing system according to any one of (5) to (7) above, which controls to notify the user of the occurrence of an event via the communication unit when it is determined that the information system exists.
(9)
When the control unit recognizes a person's face in the image captured by the image pickup unit of the client terminal received via the communication unit, the control unit displays an image of the character of the other user to be superimposed and displayed on the face. The information processing system according to (8) above, which controls transmission to the client terminal.
(10)
When the control unit recognizes the uttered voice of another user from the voice picked up by the sound collecting unit of the client terminal received via the communication unit, the uttered voice is converted into the voice of the character of the other user. The information processing system according to (8) or (9) above, wherein the converted conversion message and the corresponding phrase based on the utterance phrase database are generated and controlled to be transmitted to the client terminal.
(11)
The processor
Storing phoneme databases and utterance phrase databases corresponding to multiple types of characters in the agent storage section,
In addition to receiving a selection signal for selecting a specific character via the user's client terminal, the communication unit transmits a utterance phrase corresponding to the utterance phrase database of the specific character.
Based on the user's message received via the communication unit, a converted message converted into the voice of the specific character is generated using the phoneme database corresponding to the specific character;
Further, the utterance phrase of the specific character corresponding to the message of the user is generated by using the utterance phrase database;
Control by the control unit to return the generated conversion message and utterance phrase to the client terminal, and
Information processing methods, including.

1 Client terminal 2 Agent server 30 Dialogue processing unit 300 Dialogue processing unit 310 Question text search unit 320 Answer text generation unit 330 Conversation DB
340 Phoneme data acquisition unit 30a Dialogue processing unit 31 Basic dialogue processing unit 32 Character A Dialogue processing unit 33 Person B Dialogue processing unit 34 Person C Dialogue processing unit 35 User management unit 351 Login management unit 352 User information DB
353 Face information registration unit 354 User location information registration unit 36 Automatic utterance control unit 361 User voice extraction unit 362 Phoneme data acquisition unit 363 Location information acquisition unit 364 Phrase search unit 365 Phrase DB
366 Information Analysis Department 37 Scenario Management Department 371 Data Management Department 372 Scenario Execution Department 373 Information Analysis Department 374 Scenario DB
40 Phoneme storage 41 Basic phoneme DB
42 Character A Phoneme DB
43 Person B Phoneme DB
44 Person C Phoneme DB
50 Conversation DB generation unit 60 Phoneme DB generation unit 70 Advertisement insertion processing unit 72 Advertisement DB
80 Feedback acquisition processing unit 3 Network 10 Agent

Claims (11)

An agent storage unit that stores phoneme databases and utterance phrase databases that support multiple types of characters,
A communication unit that receives a selection signal for selecting a specific character via the user's client terminal and transmits an utterance phrase corresponding to the utterance phrase database of the specific character.
Based on the user's message received via the communication unit, a converted message converted into the voice of the specific character is generated using the phoneme database corresponding to the specific character;
Further, the utterance phrase of the specific character corresponding to the message of the user is generated by using the utterance phrase database;
A control unit that controls the generated conversion message and utterance phrase to be returned to the client terminal.
Information processing system equipped with. The information processing system according to claim 1, wherein the control unit generates an utterance phrase corresponding to the user's message based on the context of the user received via the communication unit and the utterance phrase database. The information processing system according to claim 2, wherein the user's context is at least one of the user's position, face recognition, acceleration information, or biosensor information. The information processing system according to claim 2 or 3 , wherein the user's message is a user's uttered voice or a text of the uttered voice. The information processing system further includes a scenario storage unit that stores a plurality of scenarios in which the user can participate as a character.
The control unit;
A scenario stored in the scenario storage unit is selected according to a scenario selection signal from the user received via the communication unit;
Claims 1 to 4 control to notify the user of the occurrence of a predetermined event when the context of the user received via the communication unit matches the event occurrence condition described in the selected scenario. The information processing system according to any one of the above. The content notified to the user when the predetermined event occurs includes information indicating a specific action.
The control unit determines whether or not the specific action has been executed based on the context of the user newly received via the communication unit, and if it determines that the specific action has been executed, the control unit issues a notification indicating event clear. The information processing system according to claim 5, which controls the transmission to the user. The information processing system according to claim 5 or 6 , wherein the user's context used for determining the event occurrence condition is at least one of the user's position, utterance, acceleration information, or biosensor information. The control unit receives each position information indicating the position of the client terminal of each user participating in the selected scenario by the communication unit, and another user who participates in the same scenario around the user. The information processing system according to any one of claims 5 to 7, which controls to notify the user of the occurrence of an event via the communication unit when it is determined that the information system exists. When the control unit recognizes a person's face in the image captured by the image pickup unit of the client terminal received via the communication unit, the control unit displays an image of the character of the other user to be superimposed and displayed on the face. The information processing system according to claim 8, which controls transmission to the client terminal. When the control unit recognizes the uttered voice of another user from the voice picked up by the sound picking unit of the client terminal received via the communication unit, the uttered voice is converted into the voice of the character of the other user. The information processing system according to claim 8 or 9 , wherein the converted conversion message and the corresponding phrase based on the utterance phrase database are generated and controlled to be transmitted to the client terminal. The processor
Storing phoneme databases and utterance phrase databases corresponding to multiple types of characters in the agent storage section,
In addition to receiving a selection signal for selecting a specific character via the user's client terminal, the communication unit transmits a utterance phrase corresponding to the utterance phrase database of the specific character.
Based on the user's message received via the communication unit, a converted message converted into the voice of the specific character is generated using the phoneme database corresponding to the specific character;
Further, the utterance phrase of the specific character corresponding to the message of the user is generated by using the utterance phrase database;
Control by the control unit to return the generated conversion message and utterance phrase to the client terminal, and
Information processing methods, including.

Images