JP2009151766A - Life adviser support system, adviser side terminal system, authentication server, server, support method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a life adviser support system, an adviser side terminal system, a support method and a program, for enabling an adviser to efficiently give advice to a user and to smoothly develop an interaction. <P>SOLUTION: This life adviser support system has a user side terminal unit having a photographing means for photographing the user, an interaction means for performing the interaction and a display means for displaying a companion and various information, and an adviser side terminal system having a photographing means for photographing the adviser, an interaction means and a display means. The adviser side terminal system has a determining means for determining present emotion from an image and voice of the user, a selecting means for selecting a mark of expressing the present emotion of the user for displaying, a profile storage means, a display means and a retrieval means of the user side terminal unit for displaying and retrieving information required at present by the user. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a life advisor support system, an advisor side terminal system, an authentication server, a server, a support method, and a program.

In recent years, various services have been proposed with the development of Internet technology. For example, advice work on the Internet can be mentioned. In this advice work, a user consults with a life advisor (hereinafter referred to as an advisor) using a microphone and a speaker connected to a computer.
A technique for communicating with a bank consultant while a user is at home is described in Patent Document 1, and as a system for monitoring an operator's work in a call center or the like, an operator's voice conversation is monitored, and a voice recognition result is used in advance. Patent Document 2 describes a technique for estimating a user's psychological state from the frequency of occurrence of a specific word, a silent interval, or the like.

  The call center system and the consultation system of Patent Document 1 includes one or more consulting TV (TeleVision) conference means, and the means is a call center system for a call center that can be connected to a customer TV conference means via a communication line. A data server that holds customer information, a telemarketing server that obtains customer information from the data server, and creates a list of customer TV conference means to be connected via a communication line from the consultant TV conference means, and the connection relationship A video conference relay server that mutually distributes video signals and customer information between the customer TV conference means and the consultant TV conference means, and the customer TV conference means and the consultant TV conference means that are connected to each other. A CTI (Computer Telephony Integration) server that delivers audio signals between each other, One or more customer video conference means and one or more consultant video conference means are made to correspond one-to-one to realize one or more video conferences simultaneously, and operate as follows.

  The consultant TV conference apparatus installed in the call center connects to the customer TV conference apparatus installed in the customer's home based on the customer information acquired from the data server by the telemarketing server. The video conference relay server and the CTI server mutually distribute video signals and audio signals between the two video conference apparatuses that are connected. Customer information is also output to the displays of both TV conference devices by the TV conference relay server.

  Further, the voice monitoring system, method, and program disclosed in Patent Document 2 include voice recognition processing, power extraction processing, pitch extraction processing, and voice section extraction processing for an input voice signal input from at least one voice input means. A situation in which at least one of the voice processing is performed, a voice processing result corresponding to at least one voice input means is output, a parameter corresponding to the voice processing result is derived, and a warning should be issued based on the parameter value Means for determining whether or not, and operates as follows.

According to this voice monitoring system, by estimating the psychological state of the user and the operator using at least one of a voice recognition result, pitch, power, speech speed, speech time, or a combination thereof as a parameter, When applied to a call center, the operator's condition can be accurately monitored and evaluated.
JP 2003-264636 A JP 2007-33754 A

  However, the inventions described in Patent Documents 1 and 2 described above do not take into account the user's reaction with respect to the profile related to the user and the contents presented in the past. For this reason, the advisor has to rely on the memory of the user's profile, so there are cases in which the situation at the time of the previous response cannot be accurately remembered or the dialogue does not proceed smoothly.

  Therefore, an object of the present invention is to provide a life advisor support system, an authentication server, a server, an advisor side terminal system, a support method, and an advisor that can efficiently advise a user or develop a conversation smoothly. To provide a program.

  The first life advisor support system of the present invention is a user having a photographing means for photographing a user who seeks advice from the advisor, a dialogue means for carrying out a dialogue, and a display means for displaying a dialogue partner and various information. An advisor that is connected to the user terminal device via a network and has a photographing means for photographing the advisor, a conversation means for performing a conversation, and a display means for displaying the conversation partner and various information. A life advisor support system comprising a side terminal system, wherein the advisor side terminal system stores a profile relating to a judgment means for judging a current emotion from the image and sound of the user, and a profile relating to the user Means for displaying information currently required by the user on the user side terminal device and the advisor Characterized by comprising a search means for displaying on the display means of the terminal device, the

  The second life advisor support system of the present invention is a user having a photographing means for photographing a user who seeks advice from the advisor, a dialogue means for carrying out a dialogue, and a display means for displaying a dialogue partner and various information. An advisor that is connected to the user terminal device via a network and has a photographing means for photographing the advisor, a conversation means for performing a conversation, and a display means for displaying the conversation partner and various information. A life advisor support system comprising: an authentication server for identifying a user of the user side terminal device; and the user side terminal device and a service providing side. A server for connecting to a terminal device, the advisor-side terminal system, the authentication server, or the server Includes a determination unit that determines a current emotion from the user's image and sound, a profile storage unit that stores a profile relating to the user, and displays information currently required by the user on the user side terminal device. And a search means for displaying on the display means of the advisor-side terminal device.

  The authentication server of the present invention is an authentication server used in a life advisor support system connected to a user side terminal device operated by a user who seeks advice from an advisor via a network, and the authentication server includes the user's terminal device. Judgment means for judging current emotion from image and sound, profile storage means for storing a profile relating to the user, display means for the user side terminal device and information on the advisor side for information currently required by the user Search means for displaying on the display means of the apparatus.

  The server of the present invention is a server used in a life advisor support system connected via a network to a user terminal device operated by a user who seeks advice from an advisor, and the server includes images and sounds of the user. Determination means for determining the current emotion from, profile storage means for storing a profile relating to the user, display means for the user-side terminal device and display of the advisor-side terminal device for information currently required by the user Search means for displaying on the means.

  The support method of the present invention is a life advisor support method in which a user who seeks advice from an advisor and the advisor interact with each other while watching a captured image on the display means via the net, and display various information on the display means. Determining a current emotion from the image and sound of the user, storing a profile relating to the user, searching for information currently required by the user, and the user side terminal device And a step of displaying on the display means of the advisor-side terminal device.

  The program of the present invention is a computer-readable program for an advisor terminal system used in a life advisor support system that is connected to a user terminal device operated by a user who seeks advice via a network. And an advisor side terminal system, an authentication server for specifying a user of the user side terminal device, and a server for connecting the user side terminal device and the service providing side terminal device. A process for judging a current emotion from the image and sound of the user, a profile storage process for storing the profile of the user, a search means for displaying information currently required by the user on a display means of the user side terminal device, and Executing search processing for display on the display means of the terminal device on the advisor side And features.

  According to the present invention, the content of dialogue between the user and the advisor, the access history to the Web (network), the dialogue history, the profile related to the user, the emotion, and the emotion are analyzed, summarized, and accumulated in real time. Introducing service providers and related information that meet specific requirements, we can deal with customer responses and requests more appropriately in terms of timing and content, so that customer satisfaction can be improved. it can.

Embodiments of the present invention will be described.
FIG. 1 is a conceptual diagram showing an embodiment of a life advisor support system according to the present invention. FIG. 2 is a block diagram showing an embodiment of an advisor-side terminal device used in the life advisor support system shown in FIG.
A life advisor support system 10 shown in FIG. 1 includes a user terminal device (for example, a personal computer) 11 operated by a user (not shown) seeking advice, and a member of an organization that provides a service (not shown). ) Operated by a service providing side terminal device (for example, a personal computer) and 13a,..., 13n, a user side terminal device 11 and a service providing side terminal device 13a,. Is provided with an advisor-side terminal system 12 operated by (1).

The advisor side terminal system 12 has a function of accumulating and analyzing emotion information representing the relationship between the user's dialogue contents and emotions and displaying the obtained information on the advisor side terminal device 12a of the advisor side terminal system 12 in real time. Have.
The advisor-side terminal system 12 includes an advisor-side terminal device 12a, a web consultation server 12b, an authentication server 12c, a profile database (hard disk drive) 12d as a first recording device, and experience information / promotion information / It comprises a reputation information database (hard disk drive) 12e and a search terminal device (personal computer) 12f.

Further, as shown in FIG. 2, the advisor-side terminal device 12a includes a determination unit 12-1, a selection unit 12-5, and a search unit 12-6. The determination unit 12-1 includes a facial expression extraction unit 12-3, an emotion extraction unit 12-2, and a voice recognition unit 12-4.
The advisor-side terminal device 12a includes a data storage unit 12-8, a meta information addition unit 12-9, and a unit 12-7 including a determination unit 12-1, a selection unit 12-5, and a search unit 12-6. You may provide the keyword extraction means 12-10.
The means 12-7, the selection means 12-5, the meta information addition means 12-9, and the keyword extraction means 12-10 are configured by computer software of the advisor side terminal device 12a. The data storage unit 12-8 corresponds to the profile database 12d and the experience information / promotion information / reputation information database 12e. The search means corresponds to the personal computer 12f. Examples of the dialogue means include a microphone and a built-in speaker (not shown).

In FIG. 1, the advisor side terminal system 12 guesses a potential dream, hope, desire, longing based on keywords, emotion information, and attributes during the user's dialogue, and provides related services to the advice side terminal device and use. It has a function of simultaneously displaying on the display device of the person-side terminal device 11 in real time.
The advisor-side terminal system 12 has a function of displaying emotion information in real time on the advisor-side terminal device 12a of the advisor-side terminal system 12 during consultation with the user.
The advisor-side terminal system 12 records the voice / image at the time of dialogue as stream data in the profile database 12d, and records the emotion information obtained from the dialogue contents as metadata in the experience information / promotion information / reputation information database 12e. As a result, the optimality of the proposal content obtained from the user's voice / image for each scene and customer satisfaction can be transferred to other advisors.
The advisor-side terminal system 12 enables each terminal to perform a two-party call with the service provider or a three-party call including the user when it is necessary to interact with the service provider during the dialogue with the user. The devices 11, 12, 13 a,..., 13 n have a function of transmitting and receiving image and audio data of the dialog participants.

  Attributes include user age, gender, appearance, birthplace, educational background, work history, field of expertise, exercise history, driving history, vehicle inspection history, travel history, transfer history, residence, housing, commuting time, commuting time, brother , Parents, relatives, qualifications, hobbies, feats, preferences, subscription newspapers, subscription magazines, annual income, assets, and access history. This attribute is information obtained directly from the user by the advisor and is used only for advice. In the present embodiment, this attribute is referred to as a profile (asset).

  Here, for example, the age information can be used by the advisor to search for service provider information used by the same generation of users.

The sex information can be used by an advisor to search for preference information of the same-sex people.
Appearance information can be used by the advisor to search for information on beauty-related service providers suitable for the user when the user desires.

  Hometown information can be used by advisors to develop dialogues with users, and can be used to search for and provide information on prefectural associations and related information that users are considered to belong to.

  Educational information can be used to introduce the best traders to the alumni meeting venue as the advisor develops dialogue with users.

  Employment history information allows the advisor to obtain information on the occupation to which the user belongs in order to proceed with the dialogue with the user, grasps whether the user is independent or not, and if the user wishes to start a business It can be used to introduce entrepreneur support service providers and to provide information on entrepreneurs.

  The information on the athletic history can be obtained by knowing the types of sports that the advisor was good at developing dialogues with users, the types of sports you want to try in the future, etc. Can be used to introduce sports equipment and sports equipment stores.

  The information of driving history can be compared and analyzed with the same type of cars and motorcycles that the admirers have long been unable to buy in the dialogue with the advisor, and the dealer Can be used to introduce maintenance contractors.

  The travel history information allows the advisor to obtain information on the country or region that the user has visited in the dialog with the user, and can also know the country or region that the user wanted to go to. Based on this information, the advisor can introduce the travel agency, airline company, hotel, inn, hot spring, theme park, etc. to the user. In particular, when the user is a so-called baby boomer generation, the user is very interested in travel and is likely to obtain useful information.

  For information on the transfer history, the advisor can obtain information on the country / region where he / she once lived and information on the moving company (reputation) in the dialogue with the user, and know the country / region where he / she wanted to live. Can do. From this information, the advisor can provide moving support by introducing the moving company to the user.

The information on the place of residence can be used by the advisor to obtain regional information from the user, information about the area where the user wants to live in the future, and information on the desired housing.

  Housing information can be used to introduce real estate companies and construction companies, as advisors can obtain information on the layout and structure of houses they want to live in the future in dialogue with users. Can do. Moreover, the information (reputation) regarding the seller (house sales company) of the house where the user once lived can be obtained from the user, and the information can be provided to other users.

  The information on the commuting time can be obtained or provided by the advisor during the dialogue with the user. That is, it can be used to introduce restaurants, boutiques, shopping centers and the like along the way.

  The information on the qualification can be obtained by the advisor in a dialogue with the user, and can be used to introduce a business that requires the qualification if the user wants to use it.

  The hobby information is information that the advisor can obtain in the dialog with the user, and is the information that the user is most interested in and it is easy to extract the potential hope of the user. Hobbies include travel, sports, shopping, gourmet music, reading, etc. in addition to the travel and sports described above.

  If the user's hobbies are shopping, the advisor will provide information to the user about what is prevalent, where the product he wants to buy can be purchased, what is going to be sold at the midsummer, year-end gifts and presents, etc. can do.

  When the user's hobby is gourmet, the advisor can provide information (reputation) about popular restaurants and restaurants to the user.

  When the user's hobby is music, the advisor can provide the user with the location of the musical instrument and the handling musical instrument store, business hours, regular holidays, special sales dates, reputation, and the like.

  When the user's hobbies are music appreciation, information on performance contents and performance date, venue, entrance fee, and reputation can be provided to the user.

  If the user's hobby is reading, the advisor can provide the user with information about books, magazines, bookstores, used bookstores, libraries, and used book markets.

  If the user's hobby is horticulture, the advisor can be used to introduce the user to garden supplies and horticultural stores and to provide information on the timing of sowing and replanting.

  When the user's hobby is a game, it can be used to introduce game centers, game software, and game software stores.

  If the user's hobbies are movies and theatre, the advisor can provide the user with information on movies and cinemas, theater and theater, rakugo, and attendance.

  Information on feats can be obtained by the advisor in dialogue with the user, and can be used to introduce circles and clubs.

  Preference information can be obtained by the advisor in a dialogue with the user, and the advisor can provide the user with information about food, drink, tea, coffee, tobacco, liquor and store, location.

  The access history information is a clue to grasp the image of the user such as the appearance, personality, and personality of the user, and can be smoothly transferred from the advisor to another advisor.

The user-side terminal device may be a personal computer that can be connected to a network, but it needs to have a microphone and a speaker as interactive means.
Examples of terminal devices on the service providing side include terminal devices on the municipality side that provide administrative services, terminal devices on the travel agency side, real estate companies, moving companies, financial institutions, large-scale department stores, automobile sales companies, sports clubs, circles However, it is necessary to have a microphone and a speaker as interactive means.

[Operation 1]
Next, the operation of the life advisor terminal system shown in FIG. 1 will be described with reference to FIG.
FIG. 3 is an example of a sequence diagram in the case of a two-party call of the life advisor support system to which the life advisor support method according to the present invention is applied.
The user operates the user side terminal device 11 to connect to the network (Web) 14 and accesses the WEB consultation server 12b to make a consultation request.
The WEB consultation server 12b transmits an advisor connection request signal to the advisor-side terminal device 12a.

  When the advisor side terminal device 12a performs an operation for approval by the advisor, the advisor side terminal device 12a transmits a consent signal to the WEB consultation server 12b. When receiving a consent signal from the advisor side terminal device 12a, the WEB consultation server 12b transmits a sharing start signal to the user side terminal device 11 and the advisor side terminal device 12a in order to share the network.

  With these operations, a two-party call is started. If the number of consultations of the user is the second or later, click on the image of an advisor familiar with the monitor screen of the user terminal device 11 (for example, a person who has a common feeling with close feelings such as hometown and hobbies) After that, communication with a sense of security is started with face-to-face (face-to-face) without being aware of operating the user terminal device 11.

When network sharing is started, the advisor-side terminal device 12a transmits a service user authentication signal to the authentication server 12c. This is for security management.
The authentication server 12c transmits a service user specifying signal to the advisor side terminal device 12a in order to specify the user (service user) of the user side terminal device 11. In this case, the user shown in the video camera equipped in the user terminal device 11 can be specified by face authentication.

Upon receiving the service user identification signal, the advisor-side terminal device 12a transmits a user profile information request signal to the profile database 12d.
The profile database 12d transmits the profile information to the advisor terminal device 12a.

  Here, the profile information is obtained by acquiring personal information related to the user from the database and displayed on the monitor of the advisor terminal device 12a. The profile information such as the name, age, address, hometown, etc. General personal information. Further, the profile information includes things that the user was engrossed, longed for, what he wanted to do in the future, memories from the past to the present, hopes from the present to the future, and the like.

  The advisor terminal device 12a transmits experience information / promotion information / reputation information request signal to the experience information / promotion information / reputation information database 12e.

Here, CGM (Consumer Generation Media: Utilizing the Internet) describes the experience of people with similar preferences to service users, the reputation of services (services) and products (products) recommended for users, and promotions by businesses. The terminal device 12a on the advisor side obtains the results of analysis from various information such as the user's clothes, access time, family environment, etc. The advisor proposes the best service and product to the user, and is intended to improve CS (Customer Satisfaction) (experience information / reputation information / promotion information).
With response management technology, a template is created according to the content of the consultation from various consultations that had been in the past for multiple advisors, and it is possible to provide a high level of service regardless of the skill of the advisor (service provision template ).

The experience information / promotion information / reputation information database 12e transmits an analysis result signal of experience information / promotion information / reputation information to the advisor-side terminal device 12a.
Upon receiving the experience information / promotion information / reputation information analysis result signal, the advisor terminal device 12a transmits a related information collection request signal to the Web 14.
Receiving the related information collection request signal, the Web 14 transmits the related information to the user side terminal device 11 and the advisor side terminal device 12a.

  Here, when a service user accepts the information proposed by the advisor, the advisor performs end-user computing in real time and introduces it to the user, such as multiple companies related to the proposed service or product It is.

For example, it collects and introduces to a user a plurality of travel companies and local governments that provide country life experience → country life experience services, travel methods, travel plans, souvenir information, nearby tourist attractions, and the like.
In addition, there is a contradiction to the provision of a uniform service, but it is possible to provide a service that takes advantage of the personality of the advisor by utilizing various information.

[Operation 2]
FIG. 4 is a diagram illustrating an example of a flowchart of the advisor-side terminal device 12a in the sequence diagram illustrated in FIG. FIG. 5 is an example of a flowchart showing the state of facial expression and voice analysis in the flowchart shown in FIG.
First, the advisor side terminal device 12a determines whether or not there is an access from the user side terminal device 11 (step S100), and waits until there is an access (step S100 / NO). Proceed to
In step S101, it is determined whether or not there is an access history. If it is determined that there is an access history (step S / YES), it is displayed on a monitor which is a display means of the advisor side terminal device 12a (step S102).
If it is determined that there is no access history (NO in step S101), the user's facial expression and voice are analyzed to determine the current emotion.
An emotion mark representing the user's emotion is displayed on the monitor of the advisor terminal device 12a (step S104), the keyword extracting means 12-10 extracts the keyword (step S105), and the keyword is displayed on the monitor of the advisor terminal device 12a. It is displayed (step S106).

  Here, the term “analysis” means that the facial expression is extracted from at least one feature of the user's eyebrows, eyes, and mouth by the facial expression extraction means, the conversation content is converted into text data by the voice recognition means, and the conversation content and facial expression is detected by the emotion extraction means. This means extracting the user's current feelings.

As a result of the analysis, only the monitor of the advisor side terminal device 12a displays a mark (see FIGS. 6A to 6F) representing the user's current emotion according to the flowchart shown in FIG.
Here, the face mark is selected by the selection means 12-5 according to the analysis result.
In FIG. 5, it is first determined whether or not you are angry. This is to determine whether or not the customer is complaining, and if the response is delayed, it may develop into a credit problem on the advisor side. ). Judgment criteria are the case where the eyebrows and eyes are raised and the mouth is shaped like a lip, the voice is strong and there are many bass parts, `` what '', `` responsible person '', `` ~ thing '', `` explanation '', It is determined that the user is angry when a keyword indicating anger, such as “apology”, “~ shiro”, “~ teyo”, “~ da”, is included (step S301 / YES), and an anger mark (FIG. 6 (a)) is displayed on the monitor of the advisor terminal device 12a (step S302).

When it is determined that the determination means is not angry (step S301 / NO), it is determined whether or not the user has doubts. Judgment criteria are used when the eyebrows are praised, the eyes are enlarged, the mouth is widened, and the keywords include misrepresentation such as “Eh (Rise Up)”, “Real”, “No way”, “Lies”, etc. It is determined that the person is looking at the advisor with suspicion (step S303 / YES), and the suspicion mark is displayed on the monitor of the advisor side terminal device 12a (step S304).
If the determination means determines that there is no suspicion (step S303 / NO), it is determined whether the user is thinking. Judgment criteria are that the user thinks when the eyebrows are slightly lifted, the position of the eye is either left or right, the mouth is a single character, and includes keywords such as "Noo", "Sodana", "Sone" It is determined that it is in the middle (step S305 / YES), and the thinking mark (FIG. 6C) is displayed on the monitor of the advisor side terminal device 12a (step S306).

When it is determined that the determination means is not thinking (step S305 / NO), it is determined whether or not the user is calm. Judgment criteria are: the eyebrows are almost horizontal, the eyes are normal, the mouth spreads sideways, the tone is gentle or high, and "~", "~", "like", " The user is determined to be calm when a keyword such as “I will do” or “I want to” is included, and the peace mark is displayed on the monitor of the advisor side terminal device 12a (step S308).
If it is determined that the determination means is not calm (step S307 / NO), it is determined whether the user is sad. Judgment criteria are eyebrows and eyes falling, the mouth becomes rounded, the tone is heavy, and keywords such as “painful” and “don't listen”, phlegm, spear, and mute, and sorrow marks (Fig. 6 ( e)) is displayed on the monitor of the advisor terminal device 12a (step S310).
When it is determined that the determination means is not sad (step S309 / NO), it is determined whether or not the user is pleased. Judgment criteria include eyebrows arcing up, eyes large (open pupil), mouth arcing down, and so on. When it is determined that the determination means is joy (step S311 / YES), a joy mark (FIG. 6 (f)) is displayed on the monitor of the advisor terminal device 12a (step S312).
If it is determined that the determination means is not joy (step S311 / NO), it is determined that the determination means is impossible, and the face? The mark (FIG. 6G) is displayed on the monitor of the advisor side terminal device 12a (step S313).

Here, FIGS. 6A to 6G are examples of marks representing the emotions of the user, which are displayed only on the monitor of the advisor terminal device 12a shown in FIG.
6 (a) shows anger, FIG. 6 (b) shows suspicion and distrust, FIG. 6 (c) shows thinking, FIG. 6 (d) shows peace and security, FIG. e) is sad, FIG. 6 (f) is joy and fun, and FIG. 6 (g) is indistinguishable.
By displaying these marks, the advisor can grasp the current emotion of the user in real time, and can reflect it on the topic or grasp the introduction timing of the service provider (step S103).

Here, the emotion extraction means will be described in detail with reference to the drawings.
FIG. 7 is an example of a flowchart of emotion extraction in step S103 in the flowchart shown in FIG.
First, the user terminal device 11 captures a user's face with a built-in (or external) camera, and inputs (images input) the obtained image (step S401).
The user side terminal device 11 transmits the image data to the WEB consultation server 12b (step S402).
The WEB consultation server 12b analyzes the user's facial expression from the received image data and detects features (step S403).

On the other hand, the user terminal device 11 converts the user's voice into an electric signal and inputs (voice input) with a built-in (or external) microphone (step S404).
The user terminal device 11 transmits the audio data to the WEB consultation server 12b (step S405).
The WEB consultation server 12b analyzes the user's voice from the received voice data and detects a keyword (step S406).

User emotions, which are user emotions, are extracted by the processing in steps S301, S303, S305, S307, S309, and S311 described above.
The WEB consultation server 12b transmits emotion extraction result data to the advisor terminal device 12a (step S407), and reflects the emotion extraction result in the profile data DB (database) 12b together with the date and time and the event that triggered it ( Step S408).
Based on the interpretation result, the WEB consultation server 12b transmits an emotion mark to the advisor terminal device 12a and returns to step S103 (steps S302, S304, S306, S308, S310, S312, and S313).

Returning to FIG. 4, emotion marks, keywords, and the like are recorded together with data of dialogue with the user (step S107).
Here, the advisor asks the user whether or not to continue the dialogue, for example, “Is there any other request?” (Step S108), and if the user wishes to end (step S108 / NO) ends. When the user wishes to continue the dialogue (step S108 / YES), the advisor can determine whether or not to search using the keyword displayed on the monitor during the dialogue with the user. A keyword can be extracted from the dialog, displayed in the advisor, and automatically searched without issuing a search instruction. (Step S109). If it is determined not to search (step S109 / NO), the process returns to step S101.
When an advisor needs to search using a keyword in a dialog with a user, the keyword search is performed. The keyword search is performed using search means (personal computer 12f) of the advisor-side terminal device 12a. (Step S109 / YES).

The search result is simultaneously displayed in real time from the search means on the monitor of the advisor terminal device 12a and the monitor of the user terminal device 11. At this time, if the search result is a product or service provider, the reputation of the product or service provider is also displayed (step S110).
The user's facial expression and voice while looking at the monitor are analyzed (step S111), and an emotion mark representing the current emotion for the user's product or service provider is displayed on the monitor of the advisor side terminal device 12a as described above. To do. At this time, even if the public has a good reputation, the user may not like it and vice versa, so if the user does not like it, do not recommend the product or service provider, This is because CS can be improved by recommending a product or service provider (step S112).
At the same time, keywords are extracted from the dialogue with the user (step S113), the keywords are displayed on the monitor of the advisor side terminal device 12a (step S114), and data of the reaction of the user to the product or service provider is recorded (step S114). Step S115).
It is determined whether or not the dialogue is continued (step S116). If the dialogue is continued, the process returns to step S103, and if not continued, the process is terminated (step S116 / NO).
The above is the description of the flowchart in the case of a two-party call.

[Operation 3]
Next, a case of a three-way call between a user, an advisor, and a service provider (business operator) will be described with reference to FIG.
FIG. 8 is an example of a sequence diagram in the case of a three-party call of the life advisor support system to which the life advisor support method according to the present invention is applied. It is assumed that the service providing side terminal device is 13.
After the user operates the user terminal device (PC) 11 to connect to the network (Web) 14 and accesses the WEB consultation server 12b to make a consultation request, the advisor further operates the advisor terminal device 12a. Then, the operator consultation request signal is transmitted to the service providing side terminal device 13 to the WEB consultation server 12b.
When the WEB consultation server 12b receives the provider consultation request signal, the WEB consultation server 12b transmits the provider connection request signal to the service providing side terminal device 13.
The service providing terminal device 13 that has received the provider connection request signal transmits a consent signal to the WEB consultation server 12b.
Two-party calls are ongoing, new operators are added and three-party calls are started.

  Here, in the three-party call, the advisor connects the advisor-side terminal device 12a with the service-providing terminal device 13 that meets the needs of the user, and explains detailed services and products. Basically, an advisor who is a third party who fully understands the user's needs can negotiate with the operator and make a contract smoothly.

In the three-party call, the advisor operates the advisor-side terminal device 12a and transmits a partial proxy input signal to the WEB consultation server 12b instead of the user.
Upon receiving the partial proxy input signal from the advisor side terminal device 12a, the WEB consultation server 12b transmits a signal reflecting the partial proxy input to the service providing side terminal device 13 and the user side terminal device 11.
The service providing side terminal device 13 transmits image information to the WEB consultation server 12b using a draw (drawing) function.
The WEB consultation server 12b transmits information reflecting the draw function to the advisor terminal device 12a and the user terminal device 11.
When the advisor needs to print the necessary information, the advisor operates the terminal unit 12a on the advisor side and transmits a remote print request signal to the WEB consultation server 12b.
When receiving the remote print request signal, the WEB consultation server 12 b transmits a remote print execution signal to the user terminal device 11. When receiving the remote printing execution signal, the user terminal device 11 performs printing with a printer (not shown) of the user terminal device 11.

Here, “partial proxy input” means a part of input work required for settlement (for example, user address, name, name, etc.) in order to reduce the burden of key input when the user operates the terminal device. Bibliographic items such as age, phone number, order item, order number, unit price, quantity, total price, desired delivery date, payment method (credit card, cash on delivery, transfer, registered mail, etc.) are used instead, and final approval (For example, “button to order” or “send” ON) means that the user performs.
The partial proxy input and the final approval status are recorded by a progress recording means (a log recording means described later) to prevent trouble.
In addition, the “draw function” is a function that is used to convey in an easy-to-understand manner the parts that the operator wants to emphasize.

  Here, the “draw function” is a function scattered even when the screen is shared between the advisor and the user. If only the advisor uses the “draw function” to write, it is considered that there is no problem, but when writing from both sides is permitted, access control such as “handling of operation right” is performed so that only one person can operate at the same time. There is a need to do it.

  For example, if you want to acquire the operation right, click the task bar in the terminal device display means screen to start the "Application" to be shared, select "Application sharing selection", and the file you want to display in the "Application selection" window Is selected, the “OK” button is clicked, and “OPERATION RIGHT” is clicked to obtain “OPERATION RIGHT”. After completion of the operation of the draw function, it may be controlled to release the “operation right” by selecting and releasing “release of the operation right”.

  “Remote printing” is for outputting necessary materials to a user's printer. These operations are for improving CS. Note that both the advisor and the operator can operate the proxy input, the draw function, and the remote printing.

The advisor-side terminal device 12a records profile information in the profile database 12d in order to perform profile registration / my portal registration.
Here, the “my portal” refers to a place (Web) that is always used as an entrance or base when the user uses the Internet. In general, portal sites use personal Internet mailboxes, search functions, communication spaces such as chat, daily information such as news, stock prices, weather forecasts, etc. High-level utility functions and contents are gathered in one place. The user confirms whether or not an e-mail addressed to the user has arrived there, and checks news of a field in which the user is interested, a match result of a favorite team, today's fortune, and the like.
The profile database 12d transmits information for referring to the My Portal to the user terminal device 11, so that the user can act with reference to the My Portal on the day. My portal information is information that the advisor provides to the user and is registered in the profile database 12e.

  Here, the profile information is voice recognition / mining (dividing a collection of non-standardized sentences into words and phrases using a natural language analysis method without both the user and the advisor being aware of them, and their appearance Dialogue content is automatically analyzed by techniques and systems) that extract frequency and correlation and extract useful information. That is, the user's personality, family structure, housing conditions, work experience, hobbies, special skills, products purchased by the user, products planned for purchase, products desired to be purchased (expensive products), traveled regions, planned travel regions, travel desired regions (Including overseas), experienced services (esthetics, massage), desired services, service providers that users like, etc. are analyzed. The analyzed information is registered as profile information so that it can be displayed on the monitor of the advisor side terminal device 12a or the monitor of the user side terminal device 11 at the next dialogue. In addition, information such as the user's impression felt by the advisor and emotions that react to the nodes of the conversation by emotion recognition technology are accumulated as profile information, and displayed on the monitor of the advisor side terminal device 12a at the time of dialogue with the user. provide feedback.

[Operation 4]
FIG. 9 is a diagram illustrating an example of a flowchart of the advisor-side terminal device 12a in the sequence diagram illustrated in FIG.
In FIG. 9, steps S200 to S210 are the same as steps S100 to S110 of the flowchart shown in FIG.
The search result and reputation are displayed on the monitor of the user terminal device 11 and the monitor of the advisor terminal device 12a, and the user accesses when he / she wants to consult with the service provider (step S211).
The user's facial expression and voice are analyzed even when accessing the supplier, and for example, it is determined whether or not the user likes the supplier (step S212), and an emotion mark is displayed on the monitor of the advisor terminal device 12a. (Step S213).
The keyword extraction means 12-10 is a keyword (a frequent phrase in a user's dialogue with a supplier, for example, “Can you do?”, “Do you have?”, “How much is ~?”, “ How long does it take? ") Is extracted, the degree of interest in the supplier, the degree of trust, etc. are analyzed (step S214), and the keyword is displayed on the monitor of the advisor terminal device 12a (step S215) for use. Data such as whether the person likes the trader is recorded (step S216).
If the user continues the dialogue with the service provider (step S217 / YES), the process returns to step S212, and if not (step S217 / NO), the process proceeds to step S218.
In step S218, it is determined whether or not the user continues the dialogue with the advisor. If it continues (step S218 / YES), the process returns to step S203, and if not (step S218 / NO), the process ends.

<Program and storage medium>
The life advisor support system of the present invention described above is realized by a program that causes a computer to execute processing. Examples of the computer include general-purpose computers such as personal computers and workstations, but the present invention is not limited to this.

Here, the program according to the present invention is a computer-readable program for an advisor terminal system used in a life advisor support system connected to a user terminal device operated by a user who seeks advice via a network. ,
Computer
(A) a process in which the determination means determines the current emotion from the user's image and sound;
(B) Profile storage processing for storing user profiles;
(C) The search means executes a search process for simultaneously displaying information currently required by the user on the display means of the user side terminal device and the display means of the advisor side terminal device.

In addition to the above processing, the computer
(D) a process in which the facial expression extraction means extracts facial expressions from at least one feature of the user's eyebrows, eyes, and mouth;
(E) a process in which the voice recognition means converts the conversation contents into text data;
(F) The emotion extraction unit may execute a determination process including a process of extracting the user's current emotion from the conversation contents and facial expressions.

Furthermore, in addition to the above processing, the computer
(G) Keyword extraction means extracts keywords (for example, “dream”, “want”, “want to see”, “want to eat”, “want to eat”, “want to ride”, “want to do”, etc.) from text data Process (this process can identify the user's actual and potential desires),
(H) a process in which the meta information adding means adds information such as user emotion information (anger, joy, etc.), keywords, dialogue time, etc. to the text data;
(I) The data storage means is data for each user including (1) user attribute information, (2) dialog history, (3) voice data of the dialog contents, text data, meta information and advisor answer, and reaction May be executed.

Thus, the life advisor support system of the present invention can be realized anywhere as long as there is a computer environment capable of executing the program.
Such a program may be stored in a computer-readable recording medium.

  Here, examples of the recording medium include a computer-readable storage medium such as a CD-ROM (Compact Disc Read Only Memory), a flexible disk (FD), a CD-R (CD Recordable), and a DVD (Digital Versatile Disk). , Flash memory, RAM (Random Access Memory), ROM (Read Only Memory), semiconductor memory such as FeRAM (ferroelectric memory), and HDD (Hard Disc Drive).

  The above-described embodiment shows an example of a preferred embodiment of the present invention, and the present invention is not limited thereto, and various modifications can be made without departing from the scope of the invention. is there.

  For example, in the embodiment described above, the case where the emotion extraction means is provided in the advisor side terminal device 12a has been described. However, the present invention is not limited to this, and the advisor side terminal device 12a includes an advisor, an agent, and the like. Comparing means for comparing these may be provided.

FIG. 10 is a block diagram showing another embodiment of the advisor terminal system according to the present invention.
The emotion extraction means 12-2 ′ used in the advisor-side terminal device 12a ′ includes a user-side emotion extraction means (existing 12-2) and a comparison means 100. The comparison unit 100 includes an advisor side emotion extraction unit 101 and an emotion generation unit 102.
The emotion generation means 102 is a means for giving an emotion to the presentation item presented by the computer, and generates an agent's emotion for the presentation item using the emotion of the advisor. Examples of the agent emotion generation include a known technique, for example, a method disclosed in Japanese Patent Application Laid-Open No. 2001-249949.

  Here, “agent” is literally software that performs human work, one of which is an interface agent. The interface agent is an interface that actively works from the device to the advisor. It includes an easy interaction between the device and the advisor and an anthropomorphic interface technology that presents necessary information in an easy-to-understand manner at an exquisite timing. Yes. An anthropomorphic agent that falls under the category of interface agent adds anthropomorphic behavior such as facial expressions and actions of animated characters to the system, and presents the state of the device to the advisor (for example, the degree of understanding of the advisor's question). . In other words, it can be said that the “personification agent” has a face on the interface agent.

  There are various types of users, young and old, and there are people who are short-tempered, stubborn, indecisive, determined, those who are swallowed slowly and those who are swallowed. The advisors corresponding to these users are also human and can be emotional in some cases. For this reason, it is necessary for the advisor to be able to respond calmly by objectively observing himself at the time of customer service.

FIG. 11 is an explanatory diagram showing an example of the operation of the advisor-side terminal system in the life advisor support system according to the present invention.
As shown in FIG. 11, in the advisor terminal system 12a, the user's emotion is extracted (for example, anger) 180, the advisor's (human) emotion is extracted to the user (in this case, anger) 181, and the user's emotion is An agent (computer) emotion is generated (sorrowed in this case) and is considered to be a desirable emotion that the advisor has 182.
If there is a discrepancy between the advisor's emotion and the agent's emotion for a certain period of time, the advisor-side terminal device 12a alerts the advisor by displaying a warning message on the screen or generating a warning sound.
With this attention, the advisor can regain calmness and give appropriate advice to the user.

  Here, the server (WEB consultation server) 12b or the authentication server 12c used in the life advisor support system connected to the user terminal device operated by the user who requests advice from the advisor via the network is provided with the emotion extraction means 152. Also good.

  12A is a block diagram showing an embodiment of a WEB consultation server used in the life advisor support system according to the present invention, and FIG. 12B is a WEB consultation shown in FIG. 12A. FIG. 12C is a conceptual diagram of an authentication server when a server is used. FIG. 12C is a conceptual diagram of a profile database when the WEB consultation server shown in FIG. Is an experience information / promotion information / reputation information database when the WEB consultation server shown in FIG.

The WEB consultation server 12b illustrated in FIG. 12A includes a three-way communication unit 146, a proxy unit 147, and an emotion extraction unit 152.
The three-way communication unit 146 includes a connection unit 140 and a two-party sharing unit 142.
The connection unit 140 includes an advisor connection unit 141 and an operator connection unit 144.
The two-party sharing unit 142 includes a three-party sharing unit 145. The advisor connection unit 141 and the two-party sharing unit 142 constitute a two-party call unit 143.

The connection unit 140 includes an advisor connection unit 141 and an operator connection unit 144.
The advisor connection means 141 connects the user side terminal device 11 and the advisor side terminal device 12a, or connects the user side terminal device 11 and the service providing side terminal device 13a.
The provider connection unit 144 connects the provider side service providing side terminal device 13 a and the user side terminal device 11.
The two-party sharing unit 142 shares necessary data between the user-side terminal device 11 and the advisor-side terminal device 12a.
The tripartite sharing means 145 shares necessary data among the user side terminal device 11, the advisor side terminal device 12a, and the service providing side terminal device 13a.

The two-party call means 143 performs a call between the user side terminal device 11 and the advisor side terminal device 12a.
The three-way call means 145 performs a call with the user side terminal device 11, the advisor side terminal device 12a, and the service providing side terminal device 13a.

The proxy unit 147 includes a partial proxy input reflection unit 148, a draw function reflection unit 149, a log recording unit 150, and a remote print execution unit 151.
The partial proxy input reflecting means 148 is used for the advisor to input data of bibliographic items other than final approval (for example, pressing the “order” button) when, for example, a financial liability is generated. It is an input means.

  The log recording means 150 as the progress recording means is means for recording voice, image, and character input in order to grasp the situation at the time of input, and an HDD is used, for example.

  The emotion extraction unit 152 includes an emotion extraction unit 12-2 that is a user side emotion extraction unit and a comparison unit 160. The comparison unit 160 includes an advisor side emotion extraction unit 161 and an emotion generation unit 162.

  The authentication server 12c shown in FIG. 12B includes a service user authentication unit 153 that authenticates a service user.

  The profile DB 12d shown in FIG. 12C stores user information. For example, the profile DB 12d stores data relating the user's age, gender, date and time, the event that triggered the user, and the user's emotion.

  The database shown in FIG. 12D stores experience information / promotion information / reputation information.

  In this embodiment, the WEB consultation server has been described with the emotion extraction means. However, the present invention is not limited to this, and the authentication server may have an emotion extraction means. .

Next, an embodiment of a life advisor support system according to the present invention will be described.
FIG. 13 is a diagram showing a monitor screen of the user side terminal device 11, a monitor screen of the advisor side terminal device 12a, and a monitor screen of the search means in the life advisor support system according to the present invention.
In FIG. 13, the left side shows the monitor screen of the user side terminal device 11, the upper right side shows the monitor screen of the advisor side terminal device, and the lower right side shows the monitor screen of the search terminal device 12 f of the advisor side terminal device 12.
FIG. 14 is a conceptual diagram of data recorded in the profile database 12e of the advisor terminal device used in the life advisor support system according to the present invention.

In this embodiment, a case will be described in which a user consults an advisor about a present for a grandchild.
First, a user A who is seeking advice operates the user terminal device 11 and accesses the advisor terminal device 12 through the network. When the user terminal device and the advisor side terminal device are connected, the face image of the advisor is displayed on the monitor of the user side terminal device 11.
When connected to the service-side terminal device, the face image of the service provider is displayed. That is, a two-party dialogue is started.
On the monitor of the advisor terminal device 12a, for example, the user's current face image is displayed in the upper left, and the past dialogue history (the number of consultations, age, personality, family composition, hobbies,...) Is displayed in the lower left column. It flows in telop format. In this way, the advisor can instantly determine how many times the user has accessed for the first time or how many times the user has accessed, and in the second and subsequent times, the past dialog history can be displayed on the monitor screen. Can be instantly grasped and misunderstood by other users, and it is easy for a proxy advisor to handle even when the designated advisor is absent, even if the user nominates the advisor It becomes. The monitor screen displays a face image of a service provider when connected to a service-side terminal device, which will be described later, and also displays voice and data. On the monitor screen of the user terminal device 11, the consultation date time is automatically displayed at the corner (for example, the upper right) of the screen.

After the advisor greets the user via a microphone (not shown) on the advisor side terminal device and asks about the requirements, the user sends a message to the grandchild via the microphone (not shown) on the user side terminal device. Suppose that a reply about the present is made, “Gag of a potato ...”. At this time, a joy mark is displayed at the corner (for example, lower right) of the monitor of the advisor side terminal device, and text-format interaction data and a face mark are recorded (FIG. 9).
The advisor asks the user via the microphone, "It's a gift for the grandchild. How many grandchildren are there? How much is the budget?" At this time, “Memo 1 present for wife”, “Memo 2 present for yourself”, etc. are automatically displayed on the monitor of the advisor terminal device 12a in relation to the keyword “present”.

Suppose that the user responds via a microphone, saying, “It's a 5-year-old boy. The budget is… well, 10,000, no 30,000 yen.” At this time, a pleasure mark is displayed in the corner (for example, lower right) of the monitor of the advisor side terminal device, and text-format interaction data and a face mark are recorded (FIG. 10).
The advisor inputs data to the advisor terminal device 12a by key operations such as “about present to grandchild”, “boy”, “5 years old”, and “30,000 yen”. When the data is input, the monitor of the user terminal device 11 and the monitor of the advisor terminal device 12a will simultaneously display “What is this consultation for Mr. △△ Present to grandchild Grandchild: boy 5 years old Budget: “30,000 yen”. At this time, a personal computer monitor (or a monitor of the advisor terminal device 12a) as a search means includes "keyword: present for grandchild", "boy", "5 years old", "budget: 30,000 yen Is displayed.

For example, when the advisor searches for a toy store at a department store using the search terminal device 12f, the advisor is connected to the terminal of the person in charge, the screen is shared among the three parties (user, advisor, and trader), and a three-party dialogue is started. The
When the advisor informs the person in charge, the search result is simultaneously displayed on the monitor of the user terminal device 11 and the monitor of the advisor terminal device 12a. For both monitors, “The same generation of the same case presents such a thing. First place ..., (... yen, ...), second place ..., (... yen) ..., ..." Voice and character data are displayed at the same time. The advisor can introduce the source of the products and the dealers in real time. In addition, the trader can perform product information on the user in real time.

When the advisor is explaining the product etc., the user begins to get angry, saying, "I wonder if you can speak a little louder. I can't hear you ...", an anger mark is displayed in the corner of the monitor of the advisor terminal. Is displayed (flashing), and the dialog data and the face mark in the text format are recorded. Advisors can immediately apologize to users and take action. For example, it can be suggested to the user such as the condition of the terminal device on the advisor side or "I'm sorry, but how is the volume of your personal computer?" (Figure 10).
Suppose the user notices that the volume has been reduced and apologizes, “Summan's trap. Reduced volume.” At this time, a joy mark (actually shy but joy is displayed as a joy mark) is displayed in the corner of the monitor of the advisor terminal device, and text-format interaction data and a face mark are recorded. (FIG. 14).

Suppose a user raises the volume and asks for an explanation of the advisor, and then says, “No, I ’d like to consult with you for a moment.” In this case, the first “n-” indicates thinking, and the next “try to consult with a little” smiles. A thinking mark is displayed at the lower right of the monitor, and a smiley mark is displayed immediately after. Text-format dialogue data and two types of face marks are recorded (FIG. 14).
After the user returns and places an order with the merchant that the present for the grandchild has been decided, the above-mentioned “Memo 1 present for the wife” alerts the advisor and asks the user, “How about the present for the wife? Now, we are offering a bargain sale at Dai ○ Department Store, and a charity sale in Yokohama ○ Town… ”, etc. In addition, after providing information to the wife of the user, “By the way, how about a gift for the customer himself? it can.
These data can be reproduced on the advisor side terminal device when the user consults at a later date, and can be used as a reference for advice.

Next, an embodiment of the life advisor support method according to the present invention will be described.
FIG. 15 is a diagram showing a transition of the shared screen for explaining the life advisor support method according to the present invention, and FIG. 16 is a part of the explanatory diagram of the screen image of the shared screen shown in FIG. FIG. 17 is the remaining part of the explanatory diagram of the screen image of the shared screen shown in FIG.
In the present embodiment, a case will be described in which a user (an end user in the figure) seeks advice from an advisor about country life as a measure for his old age.
In FIG. 15, an elderly couple who thinks that “the work has reached retirement and the child is standing alone. I want to spend more time living in the country and hobbies, but it would be safe if there was a counselor…” Questions arise. Considering that "What should I do first ... Let's consult with that advisor again", the user-side terminal device 11 is operated to access the advisor-side terminal device 12a and start consultation with the life advisor.
After waiting for access, the user-side terminal device 11 and the advisor-side terminal device 12a are connected, and the screen is shared (two people). At this time, “face authentication” is performed without the user's input work.
When the user talks to the advisor through the microphone, “I want to live in the countryside, but what should I do… I ’m worried,” the advisor said, “How about experiencing rural life once? Advise? At this time, the best service can be picked up and provided to the user based on various information such as profile / promotion / reputation / experience.

When the user says, “I want to participate in this experience,” the advisor says, “I understand. I will apply.” And the terminal device as a member of the organization that provides the service. Connect to and share the screen with three people. The advisor begins to inquire and negotiate with the contractor, “Please explain the country experience”.
The user's and the advisor's face and voice of the dealer flow and can interact while looking at the face. The traders are provided with information such as “You can have a“ consciousness to enjoy life ”” for the advisors and users, and the users can get a sense of trust through cooperation with experts.

In addition, the user can receive necessary service information by e-mail, that is, remote printing, URL (Uniform Resource Locator: address on the Internet) and e-mail delivery to a personal computer and a portable terminal. This will increase the enjoyment of the user “I'm looking forward to the day!”, Leading to improved CS.
For example, the advisor will send a substitute entry / guidance / materials to the user, for example, “I will book a flight ticket. I will print out the schedule for the day. It can be performed.
The contractor who has been contacted by the advisor can make preparations for the contract with a greeting such as "We are waiting".

The case where the user seeks advice from the advisor about country life as a countermeasure for the elderly will be described in more detail. FIG. 16 and FIG. 17 are explanatory diagrams for explaining the life advisor support method according to the present invention.
16 and 17 show screen images / outline, scenario outline, and utilization technology.

(Service introduction on the top page)
The introduction of the service is described as the screen image, and when the user clicks the face photo of the advisor, the screen sharing with the advisor is started (the faces of each other are displayed by the Web camera). This embodiment is a service for the active senior generation who wants to learn / teach / play with the concept of “getting consciousness to enjoy life” as the second school year / second youth.
As an outline of the scenario, the user is considering living in the countryside after retirement, but since he does not know what to start with, he decided to consult with an advisor who had previously dealt with another matter ( Click the face on the screen). Here, remote consultation is an example of the technology used.

(Display category)
As a screen image, the user and the advisor share the screen with each other. The category (house / sport / travel / food / cultural / hobby / thing / health / work / etc) is displayed. The user's profile information is displayed by face authentication only on the advisor screen (the face authentication image is displayed on the PDP).
As an overview of the scenario, the user is considering living in the country but does not know what to do, so consult with the advisor, "I want to live in the country but don't know what to start with." Although the plan is important, the advisor recommends that you try to live in the countryside first (name, age,…, memories, family structure, etc, more than sales lady of insurance). Advise, “How about experiencing rural life once?”
The technology used at this time is <face authentication>. Since the user has been consulted before, input operation such as ID / password is unnecessary, and the identity can be confirmed by face authentication. <Profile information / experience information> technology is also used. Various options can be proposed based on the user's preferences and experience information such as the same type of consultation.

(Travel display)
The screen image introduces multiple local governments and travel agencies related to travel. Regarding “house” and “food”, it is an image in which appropriate information is extracted from the user from profile information, experience information, promotion information, and reputation information.
As an outline of the scenario, the advisor asks the user's desired area and selects the country experience that the village office A is recruiting. When the advisor asks the user, "Where do you want to experience it? ... Then how about this?"
The technology used here is <promotion information / reputation information analysis>. It is possible to provide excellent information that meets the user's request.

(Display of village office A top page)
As the screen image, village hall A, village hall B, village hall C,... Are candidates, and the image is shared by the user, the advisor, and the village hall experience person in charge of rural living.
As an outline of the scenario, the advisor explains the situation to the person in charge of living in the countryside of the village office, such as “I want you to explain because XX wants a country experience?”
The technology used here is <Remote Consultation (Transfer)>. You can switch to a three-way call and consult with a specialist.

(Country experience (1) Introduction display)
As the screen image, the person in charge explains the experience contents, the schedule, etc. in order of country experience (1), country experience (2), country experience (3), and so on.
As an overview of the scenario, the person in charge explains about the countryside experience: “Experience content ..., schedule ...”. The user hears the explanation and accepts it.

(Application display)
As a screen image, the advisor inputs the application contents (name, address, etc) on behalf of the application. The three-way call ends after the input.
As an overview of the scenario, the person in charge asks you to fill in the application details, "Please enter if you like?" The advisor inputs the proxy, and draws the completion button on the monitor of the user terminal device 11 with the draw function, and indicates to the user “Is the input content correct? Click here while explaining "Please click".
The technology used here is <remote consultation (substitute input)>, and an advisor inputs necessary items in place of the user. <Remote Consultation (Draw Function)> is also a utilization technique, and the place to leave the judgment to the user is indicated by the draw function.

(Procedure / Indication of document provision)
The screen image returns to a two-party call. Enter the departure point, destination, schedule, name, etc., determined in the previous stage, and display the travel procedure, introduction of nearby tourist attractions, introduction of shopping places, and travel plans. It is the image which displays the process-result of displaying said information with the data mashup technique.
As an overview of the scenario, the advisor enters his / her place of birth, destination and schedule, and clicks on the search button, “Now I will travel.”
The technology used here is <mashup>, and it is possible to provide value-added information to the user by inputting information once.

(Print to completion display)
This is an image of printing completion → top screen → screen sharing end (remote printing, sending data to a mobile phone or PC and displaying it on the mobile phone).
As an overview of the scenario, the advisor will print the travel plan that is found and displayed on the printer of the terminal device 11 on the user's side, so that the travel schedule for the day will be printed on XX's printer. We will send you a URL so that you can see the information on your mobile phone. Thank you for your use. "
The technology used here is <remote consultation (remote printing)>. Necessary items are output remotely to a user's printer.

It is a conceptual diagram which shows one Embodiment of the life advisor assistance system which concerns on this invention. It is a block diagram which shows one Embodiment of the advisor side terminal device used for the life advisor support system shown in FIG. It is an example of the sequence diagram in the case of the two-party call of the life advisor support system to which the life advisor support method concerning this invention is applied. It is a figure which shows an example of the flowchart of the advisor side terminal device 12a among the sequence diagrams shown in FIG. It is an example of the flowchart showing the mode of a facial expression and voice analysis among the flowcharts shown in FIG. (A)-(g) is an example of the mark showing only a user's emotion displayed only on the monitor of the advisor side terminal device 12a shown in FIG. It is an example of the flowchart of the emotion extraction of step S103 among the flowcharts shown in FIG. It is an example of the sequence diagram in the case of the three-party call of the life advisor support system to which the life advisor support method according to the present invention is applied. It is a figure which shows an example of the flowchart of the advisor side terminal device 12a of the sequence diagram shown in FIG. It is a block diagram which shows other embodiment of the advisor side terminal system which concerns on this invention. It is explanatory drawing which shows an example of operation | movement of the advisor side terminal system in the life advisor assistance system which concerns on this invention. (A) is a block diagram which shows one Embodiment of the WEB consultation server used for the life advisor support system which concerns on this invention, (b) is the case at the time of using the WEB consultation server shown to (a). It is a conceptual diagram of an authentication server, (c) is a conceptual diagram of a profile database when using the WEB consultation server shown in (a), (d) is a WEB consultation server shown in (a). This is an experience information / promotion information / reputation information database when used. It is a figure which shows the monitor screen of the user side terminal device 11, the monitor screen of the advisor side terminal device 12a, and the monitor screen of a search means in the life advisor support system concerning this invention. It is a conceptual diagram of the data recorded on the profile database 12e of the advisor side terminal device used for the life advisor support system concerning this invention. It is a figure which shows the transition of the shared screen for demonstrating the life advisor assistance method which concerns on this invention. It is a part of explanatory drawing of the screen image of the shared screen shown in FIG. 16 is the remaining part of the explanatory diagram of the screen image of the shared screen shown in FIG. 15.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Life advisor support system 11 User side terminal device 11p User side portable terminal 12 Advisor side terminal system 12-1 Judgment means 12-2 Emotion extraction means 12-3 Facial expression extraction means 12-4 Voice recognition means 12-5 Selection means 12-6 Retrieval means 12-7 Means comprising judgment means 12-1, selection means 12-5, and retrieval means 12-6 12-8 Data storage means 12-9 Meta information addition means 12-10 Keyword extraction means 12a Advisor Side terminal device 12b Web consultation server 12c Authentication server 12d Profile database 12e Experience information / promotion information / reputation information database 12f Search terminal device 13a to 13n Service providing side device 14 Network 100 Comparison means 101 Advisor side emotion extraction means 102 Emotion generation means 140 Connection means 141 Advisor connection means 142 Two-party sharing means 143 Two-party call means 146 Three-party call means 147 Proxy means 148 Partial proxy input reflection means 150 Log recording means 151 Remote printing execution means

Claims (24)

A user-side terminal device having a photographing means for photographing a user who seeks advice from an advisor, a dialogue means for performing a dialogue, and a display means for displaying a dialogue partner and various types of information;
An advisor-side terminal system connected to the user-side terminal device via a network and having a photographing means for photographing the advisor, a dialogue means for carrying out a dialogue, and a display means for displaying the dialogue partner and various information. A life advisor support system provided,
The advisor terminal system is:
Determination means for determining current emotion from the user's image and sound;
Profile storage means for storing a profile relating to the user;
A life advisor support system comprising: search means for displaying information currently required by the user on display means of the user side terminal device and display means of the advisor side terminal device. A user-side terminal device having a photographing means for photographing a user who seeks advice from an advisor, a dialogue means for performing a dialogue, and a display means for displaying a dialogue partner and various types of information;
An advisor-side terminal system connected to the user-side terminal device via a network and having a photographing means for photographing the advisor, a dialogue means for carrying out a dialogue, and a display means for displaying the dialogue partner and various types of information; A life advisor support system with
The advisor-side terminal system includes an authentication server for specifying a user of the user-side terminal device and a server for connecting the user-side terminal device and the service providing-side terminal device,
The advisor terminal system, the authentication server, or the server is:
Determination means for determining current emotion from the user's image and sound;
Profile storage means for storing a profile relating to the user;
A life advisor support system comprising: search means for displaying information currently required by the user on display means of the user side terminal device and display means of the advisor side terminal device. The determination means includes
Facial expression extraction means for extracting facial expressions from at least one characteristic of the user's eyebrows, eyes, and mouth;
A voice recognition means for converting dialogue contents into text data;
The life advisor support system according to claim 1, further comprising emotion extraction means for extracting the current emotion of the user from the conversation content and the facial expression. The determination means includes
Advisor emotion extraction means for extracting the emotion of the advisor;
Emotion generating means for generating an agent's emotion with respect to the user's emotion;
A comparison means for comparing the emotion of the advisor with the emotion of the agent;
The life advisor support system according to claim 3, further comprising: an alerting means for alerting when the difference in emotion obtained by the comparing means is equal to or longer than a predetermined time. Keyword extracting means for extracting a keyword from the text data;
Meta information adding means for adding information such as the user's emotion information, the keyword, and dialogue time to the text data;
Data storage means for storing data including (1) attribute information of the user, (2) dialog history, (3) voice data of the dialog contents, text data, meta information, advisor answer, and reaction for each user. And a life advisor support system according to claim 4.   A service providing side terminal device comprising: a photographing unit that is connected to a network and photographs a member of an organization that provides a service; and a dialogue unit that interacts with an advisor or a user. The life advisor support system according to any one of 1 to 5. The user terminal system has a final instruction input means for inputting the user's approval,
The advisor-side terminal system comprises partial proxy input means for inputting data other than the final instruction input in place of the user-side terminal system, and progress recording means for recording the data input progress. The life advisor support system according to any one of claims 1 to 6, characterized in that: An advisor side terminal system used in a life advisor support system connected via a network to a user side terminal device operated by a user who seeks advice from an advisor,
The advisor terminal system is:
Determination means for determining current emotion from the user's image and sound;
Profile storage means for storing a profile relating to the user;
An advisor-side terminal system comprising: search means for displaying information currently required by the user on display means of the user-side terminal apparatus and display means of the advisor-side terminal apparatus. The determination means includes
Facial expression extraction means for extracting facial expressions from at least one characteristic of the user's eyebrows, eyes, and mouth;
A voice recognition means for converting dialogue contents into text data;
The advisor-side terminal system according to claim 8, comprising an emotion extraction unit that extracts the user's current emotion from the conversation content and the facial expression. The determination means includes
Advisor emotion extraction means for extracting the emotion of the advisor;
Emotion generating means for generating an agent's emotion with respect to the user's emotion;
A comparison means for comparing the emotion of the advisor with the emotion of the agent;
The advisor-side terminal system according to claim 9, further comprising: an alerting means for alerting when the emotional deviation obtained by the comparing means is longer than a predetermined time. Keyword extracting means for extracting a keyword from the text data;
Meta information adding means for adding information such as the user's emotion information, the keyword, and dialogue time to the text data;
Data storage means for storing data including (1) attribute information of the user, (2) dialog history, (3) voice data of the dialog contents, text data, meta information, advisor answer, and reaction for each user. The advisor-side terminal system according to claim 10, further comprising: An authentication server used in a life advisor support system connected via a network to a user terminal device operated by a user who seeks advice from an advisor,
The authentication server is
Determination means for determining current emotion from the user's image and sound;
Profile storage means for storing a profile relating to the user;
An authentication server comprising: search means for displaying information currently required by the user on display means of the user terminal device and display means of the advisor terminal device. A server used in a life advisor support system connected via a network to a user terminal device operated by a user who seeks advice from an advisor,
The server
Determination means for determining current emotion from the user's image and sound;
Profile storage means for storing a profile relating to the user;
A server comprising: search means for displaying information currently required by the user on display means of the user side terminal device and display means of the advisor side terminal device. A life advisor support method in which a user who seeks advice from an advisor and the advisor interact with each other while viewing a captured image on a display means via the net, and display various information on the display means,
Determining a current emotion from the image and sound of the user;
Storing a profile for the user;
Searching for information currently required by the user;
A life advisor support method, comprising: displaying on the display unit of the user side terminal device and the display unit of the advisor side terminal device. The step of determining includes
Extracting facial expressions from at least one characteristic of the user's eyebrows, eyes, and mouth;
A step of converting the conversation contents into text data;
The life advisor support method according to claim 14, comprising: extracting a current emotion of the user from the conversation content and the facial expression. The step of determining includes
Extracting the advisor's emotions;
Generating an agent's emotion for the user's emotion;
Comparing the emotion of the advisor with the emotion of the agent;
The life advisor support method according to claim 15, further comprising a step of calling attention when the difference in emotion obtained in the comparing step is longer than a predetermined time. A final instruction input step for inputting a final instruction;
A partial proxy input step for performing data input other than the final instruction input on behalf of the user side terminal system;
The life advisor support method according to claim 16, further comprising a progress recording step of recording the data input progress. Extracting keywords from the text data;
Providing the text data with information such as the user's emotion information, the keyword, and dialogue time;
Storing for each user (1) attribute information of the user, (2) dialogue history, (3) voice data of the dialogue content, text data, meta information, advisor answer, and data including reactions; 18. The life advisor support method according to claim 17, further comprising:   19. The life advisor support method according to claim 16, wherein an image of a member of an organization that provides a service is displayed and a dialogue is performed. A computer-readable program for an advisor terminal system used in a life advisor support system connected to a user terminal device operated by a user seeking advice via a network,
The computer
The determination means of any one of the advisor side terminal system, the authentication server for specifying the user of the user side terminal device, and the server for connecting the user side terminal device and the service providing side terminal device, A process to determine the current emotion from the user's image and sound,
A profile storing process for storing the profile of the user;
A program for causing a search means to execute a search process for displaying information currently required by the user on the display means of the user side terminal device and the display means of the advisor side terminal device. The computer
Processing in which facial expression extraction means extracts facial expressions from at least one feature of the user's eyebrows, eyes, and mouth;
A process in which the speech recognition means converts the conversation content into text data
21. The program according to claim 20, wherein an emotion extraction unit executes a determination process including a process of extracting the current emotion of the user from the conversation content and the facial expression. The computer
A process in which the advisor emotion extraction means extracts the emotion of the advisor,
A process in which an emotion generation means generates an agent's emotion with respect to the user's emotion;
A process in which the comparing means compares the emotion of the advisor with the emotion of the agent;
22. The program according to claim 21, wherein the alerting means executes a process of alerting when the emotional deviation obtained by the comparing means is longer than a certain time. The computer
A process in which the keyword extraction means extracts keywords from text data;
A process in which meta information adding means adds information such as the user's emotion information, the keyword, and dialogue time to text data;
For each user, data storage means includes (1) attribute information of the user, (2) dialogue history, (3) voice data of the dialogue content, text data, meta information, advisor answer, and data including responses. 23. The program according to claim 22, wherein the storage process is executed. The computer
A process in which the final instruction input means inputs the final instruction;
A process in which a partial proxy input means performs data input other than the final instruction input on behalf of the user terminal system;
The program according to any one of claims 20 to 23, wherein a progress recording unit executes a process of recording the data input progress.

Images