JP2019191795A - Customer service support system - Google Patents

Abstract

To quickly grasp customer requests.SOLUTION: An information processing apparatus (terminal 100) according to the present invention includes imaging means (imaging unit 180) for capturing an image of a scene around a user, audio acquisition means (audio input unit 190) for acquiring a content spoken by the user, analysis means (analysis unit 130, information generation unit 110) for analyzing the acquired audio on the basis of the captured image, and output means (output unit 120) for outputting a result of the analysis.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a technology for supporting customer service.

  There is a system to support the receipt of goods orders from customers and accounting. By using such a system, it is possible to provide an order to a customer without verbally communicating the details of the order to the customer service.

JP 2014-63516 A

However, if there is a customer unfamiliar with the operation of the device, or if it is difficult to respond to the request depending on the operation of the terminal device due to the nature of the request, the customer calls the customer service. However, the fact is that it is very difficult to arrange a customer service due to factors such as budget and lack of human resources. As a result, there is a concern that customer satisfaction is lowered. Or, in the first place, even if you are dissatisfied with customer service, there are many customers who hesitate to call the customer service. In this case, the customer's dissatisfaction with the customer service cannot be resolved.
Furthermore, even if the customer's request is satisfied as a result of the customer service being taken appropriately, the customer may already have a certain amount of dissatisfaction or stress at the time of calling the customer service. The bad impression is not completely wiped out.

  Thus, the conventional customer service support system has room for improvement. An object of this invention is to grasp | ascertain a customer's request rapidly.

  In one aspect, the present invention provides an image acquisition unit that acquires an image showing a scene around the user, a voice acquisition unit that acquires a voice related to the user's utterance, and the acquired image based on the acquired image. Provided is an information processing apparatus having analysis means for analyzing an utterance and output means for outputting the result of the analysis.

  According to the present invention, it is possible to quickly grasp a customer's request.

1 is a schematic diagram of a customer service support system 10. The figure which shows the installation state of the terminal 100. FIG. The functional block diagram of the terminal 100. FIG. The functional block diagram of the server 200. FIG. The figure which shows the operation example of the terminal 100. FIG. The example of the content displayed on the screen of the terminal 100 (the 1). The example of the content displayed on the screen of the terminal 100 (the 2). The example of the content displayed on the screen of the terminal 100 (the 3). The example of the content displayed on the screen of the terminal 100 (the 4). The example of the content displayed on the screen of the terminal 100 (the 5). An example (the 6) of the content displayed on the screen of the terminal 100. FIG. An example of the content displayed on the screen of the terminal 100 (part 7). An example of the content displayed on the screen of the terminal 100 (part 8).

  FIG. 1 shows an outline of the customer service support system 10. The customer service support system 10 includes a terminal 100, a server 200, and an employee terminal 300. Staff STF1 and STF2 are employees who are on the floor and serve customers. Employees include a waitress who carries food to the table, a boy, or a floor manager who supervises. Staff STF3 is a cook who is in the kitchen and makes food. The employee's role need not be clear. In the following, it is simply called an employee.

Terminals 100 (terminals 100-1, 100-2, 100-3) are provided corresponding to one or more customers who receive service provision. Further, the number of terminals 100 is an example.
Server 200 communicates with terminal 100 and employee terminal 300 (employee terminals 300-1, 300-2, 300-3).
Each employee terminal 300 (employee terminal 300-1, 300-2, 300-3) is carried by an employee or installed in a store, and notifies by a function of performing wireless communication with the server 200 and an information image or voice. It is an information processing terminal equipped with functions, and notifies employees such as customer service and cooks. Each employee terminal 300 performs wireless communication with the server 200 and, when receiving a customer service or other instruction from the server 200, notifies the employee of information. The number of employee terminals 300 shown in the figure is an example.

  FIG. 2 shows the installation state of the terminal 100. At least one terminal 100 is provided for a table, a counter, one user who receives service provision, or one user group. In this example, it is a tablet type information terminal placed on one table TBL1 on which four customers CST1 to CST4 (hereinafter referred to as users) receiving services are seated. The terminal 100 includes a microphone camera 101, a speaker 102, and a touch panel 103. In addition, the form and installation position of the terminal 100 are examples, and may be a wall hanging, for example.

The microphone camera 101 captures a scene including the table TBL1 and the customer CST and collects environmental sounds including the utterance of the customer CST.
The speaker 102 is a speaker for notifying information to the user by voice.
The touch panel 103 is a display device with a touch panel, which is operated by the customer CST to receive services such as ordering and calling, and information and instructions are input to the terminal 100, and the customer CST is serviced by characters and images. A device that displays content and other information.

In addition, the table, the content of food and drink, the form of provision of food and drink, the seating, the seat position, and the number of people are examples.
Microphone camera 101 and / or speaker 102 may be provided separately from touch panel 103. The speaker 102 may be omitted. In this case, notification of information to the user is performed only with characters and images. In short, while using the terminal 100, the user is allowed to perform an input operation related to an order acceptance or other service request, or browse information related to service contents such as a menu, while at least one user's voice and the surroundings of the user It is only necessary that the terminal 100 can acquire an image showing a scene. The surrounding scene of the user is, for example, the contents of the provided dishes, the number of tables, the table image reflecting the situation where the dishes are consumed, etc. and the user situation (contents of actions and actions such as eating and drinking, It includes at least one of images indicating a situation in which a service is provided (eating and drinking) such as an aspect of an action such as speed and a facial expression. Note that the acquired image may be a still image or a moving image.

  FIG. 3 is a functional block diagram of the terminal 100. The terminal 100 includes an information generation unit 110, an output unit 120, an analysis unit 130, an operation unit 140, a timing unit 150, a storage unit 160, a photographing unit 180, an image processing unit 181, an audio input unit 190, and an audio signal processing unit 191. Including.

  The operation unit 140 is realized as the touch panel 103, for example, and is operated by the user.

  The time measuring unit 150 is realized by a clock circuit or the like, and is appropriately referred to by the information generating unit 110 to provide current time information.

  The photographing unit 180 and the image processing unit 181 are realized as the microphone camera 101. The photographing unit 180 includes an optical system, a lens, a photographing mechanism, a light receiving element, and a control mechanism, and photographs a scene around the terminal 100 (in other words, around the user). The image processing unit 181 is realized by an image processing processor, and extracts the feature amount of the shot scene from the image data shot by the shooting unit 180 using a known algorithm. The feature amount is, for example, the content and timing of various operations including the presence / absence of a plate on the table, the degree to which the dish is consumed, the facial expression of the user, the movement of the user's mouth and the movement of chopsticks. The feature amount extraction timing is executed, for example, every predetermined number of frames. The imaging unit 180 and the image processing unit 181 function as an image acquisition unit that acquires an image showing a scene around the user. The generated feature amount is supplied to the analysis unit 130 in real time or at predetermined time intervals.

  The audio input unit 190 and the audio signal processing unit 191 are realized as the microphone camera 101. The voice input unit 190 is a condenser microphone. The voice signal processing unit 191 includes a voice signal processor that performs text removal processing and voice recognition processing from waveform data acquired by the voice input unit 190 to generate text data. The voice input unit 190 and the voice signal processing unit 191 function as a voice acquisition unit that acquires voice related to the user's speech. That is, the user's utterances are collected and information (words, phrases, etc.) for adding the contents of the utterances is generated. Information characterizing the content of the generated utterance is supplied to the analysis unit 130.

  The analysis unit 130 is realized by one or more general-purpose or dedicated processors, and serves as an analysis unit that analyzes a user's utterance using an image acquired by the image acquisition unit and a voice acquired by the audio acquisition unit. Function. Here, the analysis of the utterance is not only specifying the dictionary meaning of the word extracted by the voice analysis but also determining the background and situation of the utterance, the intention of the utterance, and the like. For example, in general, even if it is detected that a certain word is uttered, the meaning such as a demonstrative pronoun cannot be determined unambiguously. There are cases in which it is not possible to determine the situation or background, such as a statement made as an instruction to another person.

In view of this point, the analysis unit 130 attempts to extract deeper level information such as intention and background, in addition to specifying the content of the utterance using the image showing the scene.
For example, the analysis unit 130 analyzes the acquired audio content based on the captured image, and determines whether the acquired utterance means the content of the order based on the captured user action. Determine. The analysis result is supplied to the information generation unit 110. Or the analysis part 130 determines whether an utterance means the content of an order according to the progress degree of the eating and drinking determined based on the image | photographed image. At this time, when the word extracted by the voice analysis is a significant word other than the pronoun or exclamation, the utterance may be analyzed in consideration of the meaning of the word in addition to the image.

When it is determined that it means the content of the order, flag information indicating that it is an order item is added to the text information indicating the item of the order and supplied to the information generation unit 110. The information generation unit 110 generates order information based on the supplied information.
If it is determined that the content of the order is not meant, it is determined whether or not it means the user's impression, evaluation, dissatisfaction, demand, etc. (hereinafter referred to as an impression) for the product and / or service. Specifically, the voice data of the utterance part is cut out or converted into text information indicating the utterance content, flag information indicating the impression information is added, and the information is supplied to the information generation unit 110.
If it is determined that the order is neither content nor impression, no information is supplied to the information generation unit 110.

The information generation unit 110 is realized by one or more general-purpose or dedicated processors, and controls each unit of the terminal 100 by reading a control program from the storage unit 160 and executing it. Specifically, based on information (analysis result) supplied from the analysis unit 130 and signals supplied from the operation unit 140 and the time measuring unit 150, and using information read from the storage unit 160 as necessary, Information including a notification execution instruction and notification content is generated and output to the output unit 120.
Specifically, one or more analysis results generated by the analysis unit 130, operation contents acquired from the operation unit 140, analysis results generated by the analysis unit 130, and operation signals input from the operation unit 140, The information is generated based on at least one of the timing information acquired from the time measuring unit 150 and the user attribute read from the storage unit 160.

  The information generated by the information generation unit 110 includes the following four types. The first is information related to the order received by the operation unit 140 or the voice input unit 190 (information about the order item and provision timing), which is mainly provided to the employee terminal 300-3 (order). Information). The second is information mainly transmitted to the employee terminals 300-1 and 300-2, and the signal input from the operation unit 140 or the information generation unit 110 is generated based on the information from the analysis unit 130. , Information (referred to as “necessity information”) regarding the necessity of reception of employees and specific contents of customer service (for example, asking user requests, providing water, lowering dishes, etc.). The third is information (referred to as user support information) that is supplied to the display unit 121 or the audio output unit 122 to make various notifications, suggestions, requests, and the like to the user. The fourth is information (hereinafter referred to as feedback information) of user impressions, evaluations, dissatisfactions, requests, etc. (hereinafter referred to as impressions, etc.) regarding the product and / or service.

The method for generating the necessity information or the user support information is executed as follows, for example.
If it is presumed that the ordering operation is not successful based on the user's action taken, such as when the operation unit 140 detects that the user repeats the same operation, ? ","Don't find a tuna ... ", or when the analysis unit 130 detects an utterance, or a user's facial expression that an empty plate has accumulated on the table from the captured image If the analysis unit 130 analyzes that the scene is stiff or the user is jerky (the face direction changes drastically), the information generation unit 110 is not operating well and wants to lower the dish. , Guess that the user has some dissatisfaction or request, such as not knowing the accounting method, and based on the content of the guess, generate a screen that supports the operation and Determine materiality, based on the result and the determination result of the determination, generates the answering information and user assistance information.

The user support information includes information for identifying products or services (such as recommended products) corresponding to the photographed scene, information for supporting operations related to orders, and messages for prompting the user to provide feedback and evaluation. May be included. Further, the user support information may include an image corresponding to the information. The image corresponding to the scene may be, for example, information related to a product or service determined in accordance with the content (amount or type) of the user's order, the speed or timing of the order, Information that is not directly related to a product or service such as an avatar image may be used. By displaying such an image, the entertainment property at the time of ordering operation is improved.

Regarding the proposal of product information among user support information, when the analysis result that the user is laughing, is happy, satisfied, the plate is empty (the pace of eating and drinking) is faster than the standard, The information generation unit 110 may determine that it should be recommended to place an order having the same content as the immediately preceding order content. Regarding the pace of eating and drinking, for example, by storing the standard time required for one item of eating and drinking in the storage unit 160, and comparing this standard time and the time calculated from the analysis result of the analysis unit 130, Determine the pace of eating and drinking.
In addition, when the user attribute of the subject user is included in the storage unit 160, the information generation unit 110 always displays the order contents of this time. If it is different from the order item, it may be determined that the product should be proposed based on the order history of other users having similar user attributes or the usual product.
In addition, the information generation unit 110 gives impressions and evaluations regarding products and services such as “This is really delicious!”, “Let's go to this store again”, “How about the attitude of the employee just before ...” When the analysis unit 130 confirms an utterance that is estimated to mean, it may be determined that an opportunity to request provision of the evaluation has arrived, and it may be determined that the request should be executed to the user. .

  Regarding the order information, the information generation unit 110 may adjust the timing of providing food and drink based on the order timing received from the operation unit 140 and the analysis result of the scene generated by the analysis unit 130. For example, when the analysis unit 130 obtains an analysis result that the user's utterance amount is large, the speed at which the dish on the table is consumed is slow, or the time for holding the bridge or fork is short, the operation unit 140 In the case of an accepted order, an instruction to intentionally delay the provision timing is generated and included in the order information. Conversely, when it is determined that the eating and drinking speed is fast, the provision timing of the goods related to this order is rushed (flag information indicating that priority is given to orders of other users is generated. The provision timing is already applied. The determination may be made based on the type of cooking food and user attributes.

  Regarding feedback information, utterance content (voice data) or video (facial expression etc.) itself may be included in the feedback information, or the information generation unit 110 generates information obtained by processing the voice data and image data. May be. For example, in association with the order information, the order ID, the order time, and the satisfaction index (for example, information of “2” in five stages) are generated.

The storage unit 160 is a storage device such as a hard disk or a semiconductor memory, and in addition to an OS program for realizing the operation of the terminal 100 to be described later, it generates a screen for displaying menus and receiving orders (user interface). ) Is stored.
In addition, user attributes are stored in the storage unit 160. The user attributes may include information such as the user's age and sex, as well as the user's relationship with the product to the service (order content history, store visit history, frequency of times, etc.).

In addition, the storage unit 160 stores image and sound data that is acquired by the photographing unit 180 and the voice input unit 190 and reflects the scene, or an analysis result generated by the analysis unit 130 (contents of the user's utterances). Is stored). Note that the data generated based on the photographing unit 180 and the voice input unit 190 is likely to contain information related to the user's privacy. It is preferable that the information is processed into information that is not specified and stored above.

  The output unit 120 includes a display unit 121, an audio output unit 122, and a communication unit 123, and outputs information reflecting the analysis result of the analysis unit 130 supplied from the information generation unit 110. Specifically, the display unit 121 and the voice output unit 122 notify the user, and the communication unit 123 provides information to the server 200 or the employee terminal 300.

  The display unit 121 includes a liquid crystal panel and a drive circuit, and is realized as the touch panel 103. The display unit 121 notifies the user of information using characters or images. The audio output unit 122 includes a microphone and an A / D conversion circuit, and is realized as the speaker 102. The voice output unit 122 notifies the user of information by voice. The display unit 121 and the audio output unit 122 may notify the same information or different information. In the former case, one of the display unit 121 and the audio output unit 122 may be omitted.

  The communication unit 123 is realized as a communication module for transmitting information to the server 200 in accordance with a predetermined wireless communication standard such as IEEE, and transmits order information, reception information, and feedback information to the server 200. Information transmitted to server 200 is transmitted in association with information for identifying each terminal 100 that is the transmission source of the information, and stored in server 200.

  FIG. 6 shows a screen SC <b> 1 displayed on the touch panel 103 by the output unit 120. In this example, the terminal 100 is installed in a conveyor belt sushi restaurant and has a function of receiving an order by a user operation. Specifically, the screen SC1 includes an object OB10, an object OB11, an object OB12, an object OB13, an object OB30, an object OB31, an object OB40, an object OB41, and an object OB42. A user touches each object to select a desired object.

  The object OB10, the object OB11, the object OB12, and the object OB13 are for selecting the type of product. The object OB20 displays a list of products (in this example, sushi) and allows the user to select what he wants to order. The object OB30 displays the product selected by the object OB20. When the object OB31 is selected, the order of the product displayed on the object OB30 is confirmed.

The object OB40 and the object OB41 are objects (soft buttons) for performing screen transition. Object OB42 is selected when it is desired to call an employee.
When placing an order, the user selects an object displayed in the object OB20 and drags it into the object OB30. When the user selects the object OB31, order information is generated by the information generation unit 110, and is transmitted via the communication unit 123. To the server 200. When the object OB42 is selected, an employee call instruction is generated by the information generation unit 110 and transmitted to the server 200 via the communication unit 123.

  FIG. 4 is a functional block diagram of the server 200. The server 200 includes a communication unit 210, a control unit 220, and a storage unit 240. The communication unit 210 is a communication interface for transmitting and receiving information between each terminal 100 and each employee terminal 300. The storage unit 240 is an information storage device such as a hard disk or a semiconductor memory, and stores a program for realizing the operation of the server 200 described later.

The control unit 220 includes a staff management unit 221, an order management unit 222, and a feedback management unit 223. When the notification information received from the terminal 100 includes information indicating that a call has been received from the user or other information indicating that an employee needs to respond to the staff management unit 221, An instruction is transmitted to at least one or more employee terminals 300 corresponding to the notification information among 300-1, 300-2, and 300-3.

  Upon receiving the order information from the terminal 100, the staff management unit 221 transmits the order information to the employee terminal 300-3. Note that the staff management unit 221 may rewrite the information regarding the order timing included in the order information in consideration of the status of orders received from other tables.

  The feedback management unit 223 transmits to the employee terminal 300 information about the content of the order received from the terminal 100, information on the timing of provision, and information regarding the employee call. Specifically, the order information is transmitted to the employee terminal 300-3, and the response necessity information is transmitted to at least one of the employee terminal 300-1 and the employee terminal 300-2.

  FIG. 5 shows an operation example of the terminal 100. The terminal 100 intermittently performs image acquisition (S502) and audio acquisition (S504) at a predetermined timing, and analyzes the acquired image data and audio data (S506). Note that the audio acquisition timing and the video acquisition timing do not need to match. It is only necessary that the acquired image and sound are correlated in time.

  Based on the result of the analysis in the analysis unit 130, the information generation unit 110 determines whether or not order support is necessary, such as whether the user is troublesome in the operation of order input (S508), and determines that it is necessary (S508). : YES), a screen for supporting the order operation is generated (S510).

  FIG. 7 is a screen SC2 displayed on the touch panel 103 in S510. On the screen SC2, the objects OB50 to OB53 are displayed so as to overlap the normal screen. The objects OB50 to 53, for example, browse the same page and select the same item, but have not been dragged to the object OB30, or are staring at the screen for a predetermined period or more while speaking “tuna”. Is displayed when it is estimated that tuna would be ordered.

  When the user who understands the contents of the object OB50 selects the object OB51, the order of tuna is confirmed. As a result, the order of the tuna is confirmed without performing an operation on the object OB30 and the object OB31. Specifically, the information generation unit 110 generates order information indicating that a tuna order has been ordered.

  When the user selects the object OB52, a screen (not shown) for explaining the operation is displayed. When the user selects the object OB53, the display of the objects OB50 to 53 is terminated, and the display content returns to the screen of FIG.

  When order support is not necessary (S508: NO), the information generation unit 110 determines whether or not it is appropriate to propose a product (S512). When it is determined to be appropriate (S512: YES), the information generation unit 110 generates the proposal content, and the proposal content is displayed on the touch panel 103 (S514).

  For example, when the beverage in the glass placed on the table is low, or when a plurality of items other than the soup are consecutively used, the information generation unit 110 receives the order even if a predetermined time has elapsed since the previous order. If it is not done, or if it is estimated that the user wants a drink, such as when a thirsty product such as dried fish is ordered, it is determined that a drink should be proposed.

FIG. 8 shows a screen SC <b> 3 displayed on the screen of the touch panel 103. On the screen SC3, an object OB54, an object OB55, an object OB56, and an object OB57 are displayed in an overlapping manner. This example shows a case where it is determined that a drink should be proposed.
When a user who is interested in the proposal contents of the object OB54 selects the object OB55, a menu (not shown) is displayed. When the user who has decided the item to be ordered selects the object OB56, the terminal 100 receives the utterance content, determines that the item included in the utterance content is an order, and confirms the order. When a user who is not interested in this proposal selects the object OB57, the objects OB54 to OB57 are deleted, and the display content returns to the previous screen.

Or the information generation part 110 may determine the goods to propose based on the tendency information and / or the preference information described in the user attribute until now.
FIG. 9 shows a screen SC4 of contents displayed on the screen of the terminal 100. On the screen SC4, the object OB58, the object OB59, and the object OB60 are displayed in an overlapping manner. When a user who is interested in the message of the object OB58 selects the object OB59, a list of recommended products (not shown) is displayed. When a user who is not interested in this message selects the object OB60, the objects OB58 to OB60 are deleted, and the display content returns to the previous screen.
Alternatively, the presence of a recommended product is not notified, but recommended content may be presented as shown in FIG. 10 (object OB61).

  If the information generation unit 110 determines that the proposal is not appropriate at present (S512: NO), whether or not the content of the utterance acquired by the voice input unit 190 describes an impression about the provided product or service, etc. Is determined (S516). Specifically, the information generation unit 110 determines whether or not a speech presumed to be a customer service attitude, an impression of an ordered dish, or the like is made from the utterance content. When it is determined that it is an impression, etc., about providing the user with voice and images reflecting the impression acquired by the terminal 100 to the operator of the customer service support system 10 or the store management side where the terminal 100 is set. Is requested (S518).

  When the user consents (S520: YES), the information generation unit 110 temporarily displays the utterance content stored in the storage unit 160 at the timing of acquiring the utterance and the user who spoke (or the same scene as the user). The information is transmitted to the server 200 in association with the attribute of the shared user. The server 200 that has received the utterance content stores it in the storage unit 240 (S522).

  A screen SC6 displayed on the terminal 100 in S518 is shown in FIG. On the screen SC6, the object OB62, the object OB63, and the object OB64 are displayed in an overlapping manner. When the user who confirms the content of the object OB62 and approves the impression on the screen SC6 selects the object OB63, the information generation unit 110 displays the sound or video data itself indicating the impression or the sound or video data. Is read out from the storage unit 160 generated from, and feedback information is generated and transmitted to the server 200 via the communication unit 123. When the user who has not accepted the provision of the impression selects the object OB64 (S520: NO), the objects OB62 to OB64 are deleted, and the screen returns to the previous screen.

When an impression or the like has not been detected (S516: NO), the information generation unit 110 subsequently determines whether or not an employee needs to be handled (S524). For example, the information generation unit 110 may respond to an employee when there is an expression that the employee wants to deal with in the utterance content, or when a voice that is loud between users or a fighting action is detected. Determine that it is necessary.
When it is determined that a response is required (S524: YES), the information generation unit 110 generates response necessity information and transmits it to the server 200 via the communication unit 123. If it is determined that no response is required (S524: NO), the process returns to S502.

  FIG. 12 shows a screen SC7 displayed on the terminal 100 when it is determined that a response is necessary. On the screen SC7, the object OB65, the object OB66, and the object OB67 are displayed in an overlapping manner. When the user selects the object OB6, generation of response necessity information and provision to the server 200 are executed (S526).

When the request content for the employee is specifically specified by the information generation unit 110, a screen SC8 as shown in FIG. 13 may be displayed. When the object OB66 is selected on the screen SC8, information for instructing to clean up the table is transmitted to the employee terminal 300-1 or the employee terminal 300-2 via the server 200. When the object OB68 is selected, the objects OB68 to OB70 are deleted and the screen returns to the previous screen.
Thereafter, S502 to S526 are repeated.

According to the above embodiment, the scene is analyzed from the viewpoint of audio and video. And based on the result of this analysis, a customer's demand for goods and services can be estimated, and a service in accordance with the estimated demand can be provided. If this request includes information about customer service, the necessity for reception is automatically determined, and an instruction is sent to the employee only when necessary. As a result, it is not necessary to excessively arrange the customer service in order to maintain the service quality.
In addition, according to the above-described embodiment, after obtaining the prior consent of the customer, an impression related to the customer's service is acquired. At this time, since the request for consent is executed at a timing determined to be appropriate based on the scene, the possibility of harming the customer's mood is reduced, and the possibility that the questionnaire can be cooperated is increased.

  Note that it is not necessary to execute all the determination steps S508, S512, S516, and S524 in FIG. 6, and at least one of them may be executed. Moreover, the timing and order of executing each determination step are arbitrary.

The terminal 100 may determine whether the user is ready to place an order, such as when the user is seated, and may notify the user when it is determined that the user is ready.
Specifically, the analysis unit 130 detects that a user has arrived based on the acquired video data, and in addition, determines attribute information (number of people, sex, age, etc.) of the user. The information generation unit 110 controls the display unit 121 and the audio output unit based on the content of the determination. For example, either one of waitress images and waiter images stored in advance is selected and displayed according to the gender of the user.
Then, depending on the user's age, a welcome message such as “Come on. Please tell me when you place an order. The touch panel is fine.” Or, please talk to this terminal when the order is decided. Anyway! "Is selected, and the synthesized voice of the selected message is emitted. Thereby, it is possible to give the user a sense of security that the existence of the user is surely grasped by the service provider. And even if it is the first visit, the order operation can be performed smoothly. Also, the guidance method can be changed according to the user's attributes.
In addition, when there are a plurality of users having different user attributes such as age and sex in the shooting target, for example, one attribute may be determined based on the commonality or superiority of the attribute values (for example, male 3 One attribute may be determined by performing statistical processing (for example, calculating an average value of age) or the like if the user attribute is male for a group of two people and two women.

Further, when the terminal 100 does not detect the user, the display unit 121 and the audio output unit 122 are used.
The display unit 121 and the audio output unit 122 may be activated when the user operates in a power saving mode in which a function necessary for receiving an order is not realized. In this case, it is not necessary for the user to perform an operation for operating the terminal 100.

  The server 200 accumulates the order information in the storage unit 240, and based on the accumulated information, the control unit 220 generates information on the order tendency for each season or each day of the week. Or a staffing plan may be determined. This is expected to contribute to improving the freshness of the food provided, reducing waste, preventing orders from being sold out, and improving the efficiency of staffing (increase or decrease in part-time jobs).

  In the above-described embodiment, the specification of the order contents is mainly performed by a touch operation, and the order by voice is an auxiliary role. However, the order instruction by voice is mainly accepted, and input using the touch panel 103 is performed only when necessary. You may be prompted. Alternatively, only one of reception of an order by touch operation and reception of an order by voice input may be received.

Alternatively, the user may designate either or both of the touch operation and the voice input. For example, an object for accepting designation for enabling either or both of an order by touch operation and an order by voice input is displayed on the screen, and the user is allowed to select or switch the desired order method. For example, when the terminal 100 performs an operation mainly for ordering by voice, if the meaning of the utterance content (order content) cannot be heard by the analysis unit 130, the user is prompted to utter again. Here, if the order contents cannot be determined even if the utterance is acquired multiple times due to factors such as bad tongue, or the uttered word is not registered in the audio signal processing unit, The generation unit 110 uses the display unit 121 and the audio output unit 122 to prompt the user to input order details using the touch panel 103. When the information generation unit 110 determines that the order reception by voice is not successful, the information generation unit 110 transmits response information to the server 200 and instructs the employee to direct the order directly to the user. Also good.
As a result, it is expected that user satisfaction will be improved.

  The service to which the present invention is applied is not limited to the provision of food and drink. For example, it may be the sale of goods. Further, the service providing place is not limited to the inside of the store, but may be, for example, a passenger plane or a train. For example, the terminal 100 may be provided on the front surface of each seat such as an aircraft or a vehicle (the back surface of the backrest of the front seat) and provided to a food service provided by a crew member in the cabin or in the vehicle.

  The terminal 100 may be provided with a function for paying the consideration (fee) for receiving the service. For example, a card reading mechanism for a credit card is provided in the terminal 100, and a user is allowed to select one of a plurality of payment methods, and a screen for guiding a cash payment is generated for a user who wishes to make a cash payment. A screen for guiding the card insertion operation to the desired user is generated.

The timing for inducing or supporting the payment process may be determined based on the scene information. For example, if the order is not entered even after a predetermined time has passed since the last received order, if the movement of chopsticks and forks has not been confirmed for a certain period of time, the last ordered dish such as dessert has been ordered If the utterance suggesting the end of the order is confirmed, such as “full or full” or “Is it over now” or “Is it to account?”, The information generation unit 110 determines that the meal has ended. To do. Alternatively, when the analysis unit 130 detects that the user has stood up from the chair, it may be determined that the meal has ended. In this case, for example, when no order item is provided, when a dish remains on the table, or when a user's carry-on item such as a bag is detected, it is a temporary absence from the meal. May be determined.

  When it is determined that the meal has ended, the information generation unit 110 displays an image of a waitress or waiter on the screen and emits a synthesized voice such as “Thank you for coming today. The output unit 120 is controlled to do so. Alternatively, a message such as “How about the last tea?” May be displayed on the display unit 121. Thereby, the service provider side can inform the user that the order has been recognized.

  Subsequently, the terminal 100 performs processing for guiding to accounting. For example, when the user agrees to provide tea or when the tea is provided, the display unit 121 and / or the audio output unit 122 “when paying, the card reader of this table can be used for payment at the cash register. A message for supporting payment processing such as “It is possible” is output by video or audio.

  When payment at the cash register is selected, for example, the terminal 100 notifies “Please tell the table number at the cash register”, and the order contents and the amount charged by the terminal 100 associated with the table number are sent to the server. The voucher is acquired from 200, a slip is generated based on the acquired information, and presented to the user. Alternatively, the terminal 100 notifies “Please check out at the cashier”, extracts an image of the user's face or information for identifying the user from the image, and identifies the identification information of the terminal 100 Correspondingly, it is wirelessly transmitted to a cash register terminal (a cash register or other payment terminal provided in the store and operated by an employee; not shown).

Then, when this user visits the cash register and determines whether or not the image taken by the camera of the cash register terminal installed at the cash register matches that of the user received at the cash register terminal. The server 200 is inquired of the server 200 for the identifier of the terminal 100 received in association with the user authentication information, and the order content (content of the provided service) and the billing amount that can be stored in the server 200 in association with the terminal 100 are obtained. Obtain from the server 200.
This eliminates the need for the employee to take the slip to the table and the user to take the slip to the cash register at the time of accounting.

Further, some of the functions of the terminal 100 illustrated in FIG. For example, the server 200 is provided with the function of the information generation unit 110. In this case, the information generated by the analysis unit 130 is sequentially transmitted to the server 200, and order information, user support information, response necessity information and feedback information are generated in 200, and the terminal 100 receives user support information from the server 200. It will be received and notified to the user.
In short, in an information processing system including one or more information processing apparatuses, based on the captured image, the step of capturing a scene around the user, the step of acquiring the content spoken by the user, and the captured image The step of analyzing the acquired voice and the step of outputting the analysis result may be executed.

DESCRIPTION OF SYMBOLS 100 ... Terminal, 101 ... Microphone camera, 102 ... Speaker, 103 ... Touch panel, 180 ... Shooting unit, 181 ... Image processing unit, 190 ... Audio input unit, 191, ..Audio signal processing unit, 130 ... analysis unit, 140 ... operation unit, 150 ... timer unit, 110 ... information generation unit, 160 ... storage unit, 120 ... output unit, 121: Display unit, 122: Audio output unit, 123 ... Communication unit, 200 ... Server, 210 ... Communication unit, 240 ... Storage unit, 220 ... Control unit, 221 ... Staff management unit, 223 ... Feedback management unit, 222 ... Order management unit

In one aspect, the present invention provides an image acquisition unit that acquires an image showing a scene around the user, a voice acquisition unit that acquires a voice related to the user's utterance and identifies the content of the utterance , and the acquisition An information processing apparatus is provided that includes an analysis unit that determines an intention of the user's utterance based on an operation of the user indicated by the displayed image, and an output unit that outputs information according to the determination result .

In one aspect, the present invention provides an image acquisition unit that acquires an image showing a scene around the user, a voice acquisition unit that acquires a voice related to the user's utterance and identifies the content of the utterance, and the acquisition Based on the action of the user indicated by the displayed image, the intention of the user is determined , and based on the result of the determination, the necessity information or the necessity information of the response indicating the necessity or contents of the employee to the user is determined. There is provided an information processing apparatus having analysis means for generating and output means for outputting the information on necessity of reception to an employee terminal .

Claims (11)

Image acquisition means for acquiring an image showing a scene around the user;
Voice acquisition means for acquiring voice related to the user's utterance;
Analyzing means for analyzing the utterance based on the acquired image;
An information processing apparatus comprising: output means for outputting a result of the analysis. The image acquisition means shoots a place where a product is placed or a place where a service is provided to the user;
The information processing apparatus according to claim 1. The analysis means determines whether or not the user's utterance means the content of an order based on the captured user action.
The information processing apparatus according to claim 1 or 2. Further comprising means for notifying the user,
The notification means requests the user to approve the content of the determined order;
The information processing apparatus according to claim 3. The analysis means determines whether or not the acquired utterance means the content of an order according to the progress of the user's eating and drinking determined based on the captured image.
The information processing apparatus according to any one of claims 1 to 4. The analysis means further considers the meaning of the acquired uttered word, and determines whether the acquired utterance means order content,
The information processing apparatus according to any one of claims 1 to 5. Means for notifying the user;
Storage means for storing the content of the utterance, and
The notification means requests the user to consent to store the impression in the storage means when the analysis means determines that the utterance is an impression of food or drink or customer service,
The information processing apparatus according to any one of claims 4 to 6. The output means notifies the employee of an instruction to execute customer service corresponding to the result of the analysis;
The information processing apparatus according to any one of claims 1 to 7. It further comprises display means for displaying an image corresponding to the scene.
The information processing apparatus according to claim 1. The display means displays a product or service corresponding to the scene.
The information processing apparatus according to claim 9. On the computer,
Shooting a scene around the user;
Obtaining the content spoken by the user;
Analyzing the acquired sound based on the captured image;
A program for executing the step of outputting the result of the analysis.

Images