JP7111042B2 - Information processing device, presentation system, and information processing program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, a presentation system, and an information processing program.

In recent years, systems have been developed that assist workers in learning the details of their work. For example, Patent Literature 1 discloses a work training system that displays differences in line of sight or work motion between a model worker and a learning worker based on image information.

JP 2018-180090

On the other hand, in customer service work, etc., there is a problem that the above-described conventional technique using only image information cannot appropriately support learning of the work content.

One aspect of the present invention is to realize an information processing apparatus capable of assisting a work learner's learning related to customer service work.

In order to solve the above problems, an information processing apparatus according to an aspect of the present invention includes a face information acquisition unit that acquires face information including at least part of face information of a work learner who learns customer service work, a speech information acquisition unit that acquires speech information including information of an utterance by the task learner; and a difference information derivation unit for deriving second difference information indicating a difference between the audio information acquired by the audio information acquisition unit and the second reference information; and the first difference information and a presentation information generation unit that generates presentation information corresponding to at least one of the second difference information.

According to one aspect of the present invention, it is possible to realize an information processing apparatus capable of assisting a work learner's learning related to customer service work.

1 is a block diagram illustrating components of a presentation system according to one embodiment of the invention; FIG. FIG. 4 is a diagram showing an example of an image displayed by the display of the head mounted display according to one embodiment of the present invention; It is a table|surface which shows an example of a learner's visual recognition target which the visual recognition target specific|specification part or the decision-making information specific|specification part which concerns on one Embodiment of this invention identified, an utterance, an operation|movement, and a degree of concentration. It is a table|surface which shows an example of the data of a veteran customer service person previously memorize|stored in the memory|storage part which concerns on one Embodiment of this invention. 4 is a table showing an example of difference information data derived by a difference information derivation unit according to an embodiment of the present invention; 4 is a flow chart showing processing of the presentation system according to one embodiment of the present invention. It is a figure which shows an example of the presentation information which the presentation information production|generation part which concerns on one Embodiment of this invention produced|generated. It is a figure which shows an example of the presentation information which the presentation information production|generation part which concerns on one Embodiment of this invention produced|generated. 1 is a block diagram illustrating components of a presentation system according to one embodiment of the invention; FIG. 4 is a flow chart showing processing of the presentation system according to one embodiment of the present invention. It is a figure which shows an example of the information regarding a visual recognition target object.

An embodiment of the present invention will be described in detail below. If the configuration in a specific item (embodiment) below is the same as the configuration described in other items, the description may be omitted. For convenience of explanation, members having the same functions as members shown in each item are denoted by the same reference numerals, and explanations thereof are omitted as appropriate.

[Embodiment 1]
1. Presentation System A presentation system 800 according to the present embodiment provides learning support to a work learner who learns customer service work. As a form of the learning support device, for example, there is an experiential learning device using VR or the like. The presentation system 800 according to the present embodiment includes information on a work learner (for example, a new part-time worker; hereinafter sometimes simply referred to as a learner) and a pre-stored model worker (for example, a veteran customer). By presenting the difference (also called "difference information") from the information (also called "reference information") to the learner as presentation information, it supports independent and experiential learning for the work learner. It is possible to In the following, a case where the presentation system 800 is used in a situation where a learner who works at a restaurant or the like where customer service is required learns customer service work will be described as an example.

The learner's information and reference information include, for example, the object to be visually recognized, the degree of concentration on the object to be visually recognized, the content of the utterance, the tone of voice, the action, and the like.

FIG. 1 is a block diagram illustrating components of a presentation system according to one embodiment of the invention. As shown in FIG. 1 , a presentation system 800 according to this embodiment includes an information processing device 100 , a head mounted display 200 , a wearable interface 300 , an information management server 400 and a display section 500 . With such a configuration, the presentation system 800 according to this embodiment can provide learning support to the learner.

<Head-mounted display>
As shown in FIG. 1 , the head-mounted display 200 includes a camera 210 (also referred to as an “imaging unit”) that captures at least a portion of the face of the learner, and a microphone 220 that collects the learner’s uttered voice (“sound collecting unit”). ), a speaker 240 and a display 270 (both of which may be referred to as a “presentation unit”), a motion sensor 230 , a control unit 250 and a communication unit 260 .

The control unit 250 of the head mounted display 200 comprehensively controls each unit of the head mounted display 200 . Each function of the control unit 250 and all functions included in the head mounted display 200 may be implemented by the CPU executing a program stored in a storage unit (not shown) or the like.

The communication unit 260 communicates with the information processing device 100 under the control of the control unit 250 . The form of communication is not particularly limited, and may be communication using Bluetooth (registered trademark), WiFi (registered trademark), other short-range wireless communication technologies, or other communication.

The camera 210 acquires an image of the learner's face as face information. Camera 210 may acquire an image of at least a portion of the learner's body at the same time as the facial image. The microphone 220 acquires the learner's voice as audio information. The motion sensor 230 is, for example, an acceleration sensor, and acquires the learner's motion as motion information.

Speaker 240 outputs sound input to head mounted display 200 via communication unit 260 . Display 270 outputs an image input to head mounted display 200 via communication unit 260 . The image output by the display 270 may display a virtual space, and an example of the display on the display 270 is the image shown in FIG. FIG. 2 is a diagram showing an example of an image displayed by the display of the head mounted display according to one embodiment of the present invention. An image 200A shown in FIG. 2 shows the inside of a restaurant seen from the viewpoint of a learner who is a store clerk, and the image 200A shows customers, their hands, other store employees, tables, and the like.

In addition, FIG. 3 shows the visual recognition target of the learner identified by the visual recognition target identification unit or the decision making information identification unit, utterance (utterance content and tone of voice), action and degree of concentration (blink, pupil state, gaze retention time and 10 is a table showing an example of facial expression).

<Wearable interface>
As shown in FIG. 1, presentation system 800 includes wearable interface 300 . In one example, the wearable interface 300 includes a motion sensor 310 and a button 320 to input operations performed by the learner to the information processing device 100 . An example of the operation performed by the learner includes an operation of raising or lowering the motion sensor 310, an operation of pressing the button 320, and the like.

In one example, the presentation system 800 can include more detailed information about the movement of the learner's hand in the learner's movement information acquired by the movement information acquisition unit 115 in the information processing apparatus 100 . Examples of the learner's hand movements in this embodiment include the movement of holding a cup of water, placing it on the table, and taking a meal order.

Information such as the learner's hand movement input via the wearable interface 300 can be reflected in the image output by the display 270 of the head-mounted display 200 as the learner's hand movement.

A known sensor and button can be used as the motion sensor 310 and the button 320, and the motion sensor 310 is, for example, an acceleration sensor.

<Information processing device>
As shown in FIG. 1, the information processing apparatus 100 in this embodiment includes a control section 110, a storage section 140 and a communication section .

The control unit 110 of the information processing apparatus 100 according to the present embodiment includes a face information acquisition unit 111 that acquires face information including at least part of information on the face of the task learner, and a voice information including information of an utterance by the task learner. and information stored in the storage unit 140 and the state of at least a part of the face indicated by the face information acquired by the face information acquisition unit 111 (also referred to as “first reference information”). Difference information (also referred to as “first difference information”) indicating the difference between the audio information acquired by the audio information acquisition unit 114 and information stored in the storage unit 140 (also referred to as “second reference information”) a difference information derivation unit 118 for deriving difference information (also referred to as “second difference information”) indicating a difference between the difference information and the presentation information generation unit 119 for generating presentation information according to the difference information; .

The information processing device 100 is implemented in, for example, a terminal device (eg, smart phone, tablet, personal computer, television receiver, etc.) connectable to a local network or a global network.

[Control part]
As shown in FIG. 1, the control unit 110 of the information processing apparatus 100 according to the present embodiment includes a face information acquisition unit 111, a state detection unit 112, a visual target identification unit 113, a voice information acquisition unit 114, a motion information acquisition unit 115, A decision-making information identification unit 116 , a learner information acquisition unit 117 , a difference information derivation unit 118 and a presentation information generation unit 119 are provided. The control unit 110 comprehensively controls each unit of the information processing apparatus 100 . Each function of the control unit 110 and all functions included in the information processing apparatus 100 may be implemented by the CPU executing a program stored in a storage unit (not shown) or the like.

(Face information acquisition unit)
The face information acquisition unit 111 acquires face information including information of at least a part of the face of the learner at each time point. The face information acquisition unit 111 can acquire the learner's face information, for example, from an image acquired from the camera 210 included in the head mounted display 200 . A learner's face information is information which shows the feature-value of a face. The face feature quantity refers to, for example, position information indicating the position of each part of the face (eg, eyes, nose, mouth, eyebrows, etc.), shape information indicating the shape, size information indicating the size, and the like.

In particular, eye information is particularly useful because it enables the learner to identify an object visually recognized. Eye information includes, for example, end points of the inner and outer corners of the eye, edges of the iris and pupil, and the like. Further, the face information acquisition unit 111 may appropriately perform correction processing such as noise reduction and edge enhancement on the image acquired from the imaging unit. Face information acquisition section 111 transmits the extracted face information to state detection section 112 .

(Status detector)
The state detection unit 112 detects the state of the learner from the learner's face information acquired by the face information acquisition unit 111 . For example, the state detection unit 112 detects an object that the learner is visually recognizing at each time point (“viewing target” in FIG. 3), the number of blinks per minute at each time point (“blinking” in FIG. 3), each time point The opening and closing of the pupil (“pupil state” in FIG. 3), the time during which the visual recognition target is viewed (“visual gaze retention time” in FIG. 3), or the facial expression (facial expression in FIG. 3) are detected.

The state detection unit 112 detects the learner's state based on the face information extracted by the face information acquisition unit 111 . After detecting the state of the learner, the state detection unit 112 transmits the detection result to the visual recognition target identification unit 113 .

For example, the state detection unit 112 obtains position information indicating the position of each part of the face (for example, eyes, nose, mouth, cheeks, eyebrows, etc.), which is the feature amount of the face acquired by the face information acquisition unit 111, and the shape. With reference to the shape information and the size information indicating the size, the learner's state includes, for example, the learner's visual target, pupil state, number of blinks, eyebrow movement, cheek movement, eyelid movement , lip movement and/or jaw movement.

By including the state detection unit in this way, the information processing apparatus can suitably detect the state of the learner.

Although the method for detecting the object to be viewed is not particularly limited, the information processing apparatus 100 is provided with a point light source (not shown), and the imaging unit captures a corneal reflection image of the light from the point light source for a predetermined period of time. A method of detecting an object to be visually recognized by a person can be mentioned. The type of the point light source is not particularly limited, and includes visible light and infrared light. For example, by using an infrared LED, it is possible to detect the visual recognition target without giving discomfort to the learner.

As a learner's state, the degree of concentration can be mentioned in addition to the visual recognition target. In general, when humans are concentrating, they blink at low and stable intervals, and their pupils tend to dilate. It is possible to evaluate the degree of concentration of a person. For example, when the number of blinks is detected for a predetermined period of time, and blinking is performed at stable intervals within the predetermined period of time, it can be said that there is a high possibility that the learner is gazing at a certain object. Further, if the size of the pupil is detected for a predetermined period of time and the pupil size is long within the predetermined period of time, it can be said that there is a high possibility that the learner is gazing at a certain object.

The method of detecting the number of blinks is not particularly limited, but for example, a method of irradiating the learner's eyes with infrared light and detecting the difference in the amount of reflected infrared light between when the eyes are open and when the eyes are closed. etc.

A method for detecting the state of the pupil is not particularly limited, but for example, a method of detecting a circular pupil from an eye image using Hough transform can be used. A threshold may be set for pupil size, and a pupil size greater than or equal to the threshold may be evaluated as "open", and a pupil size less than the threshold may be evaluated as "closed".

Moreover, it is also possible to detect that the visual recognition target does not change for a predetermined period of time or more as an evaluation of the degree of concentration. For example, the above-described technology can be used to detect an object to be visually recognized by the learner, and measure the time during which the learner visually recognizes the object (“visual fixation time” in FIG. 3).

The state detection unit 112 detects at least one of the learner's visual recognition object, the state of the pupil, the number of blinks, eyebrow movement, eyelid movement, cheek movement, nose movement, lip movement, and jaw movement. However, it is preferable to combine them. By combining the detection methods in this manner, the state detection unit 112 can suitably evaluate the learner's degree of concentration when visually recognizing a certain object.

In addition to the state of the eyes, for example, eyebrow movements such as lifting the inside of the eyebrows or raising the outside, eyelid movements such as raising the upper eyelids and tightening the eyelids, and nose movements such as wrinkling the nose. , lip movements such as lifting the upper lip and pursing the lips, cheek movements such as lifting the cheeks, and jaw movements such as lowering the chin. A state of a learner may be a combination of states of a plurality of parts of the face.

The state detection unit 112 may calculate the learner's degree of concentration by determining the learner's mood from the learner's facial expression. The state related to the mood of the learner, for example, refers to the feature amount of the face as described above, and the state of the learner includes, for example, the learner's visible object, the state of the pupil, the number of blinks, the movement of the eyebrows, At least one of cheek movement, eyelid movement, lip movement, and jaw movement can be detected, and determination can be made based on the detected state of each part of the learner's face.

As a method of judging the degree of concentration of the learner from facial expressions, for example, when the eyebrows move upward, it can be judged that the learner is concentrating because he/she is gazing more at the visual recognition target. Further, for example, when the cheeks move upward while visually recognizing a person's face, it can be determined that the user is concentrating on the visually recognized target, assuming that the user is making a facial expression for the other party.

Moreover, when detecting the learner's facial expression, the state detection unit 112 can also calculate information about the learner's facial expression by machine learning. Although the specific configuration of the learning process for acquiring information about facial expressions does not limit the present embodiment, for example, any one of the following machine learning techniques or a combination thereof can be used.

・Support Vector Machine (SVM)
・Clustering
・Inductive Logic Programming (ILP)
・Genetic Algorithms (GP)
・Bayesian Network (BN)
・Neural Network (NN)
When using a neural network, the data may be preprocessed and used as input to the neural network. For such processing, techniques such as data argumentation can be used in addition to one-dimensional or multi-dimensional arraying of data.

Moreover, when using a neural network, a convolutional neural network (CNN: Convolutional Neural Network) including convolution processing may be used. More specifically, one or more layers included in the neural network include a convolution layer that performs a convolution operation, and a filter operation (product-sum operation) is performed on input data input to the layer. may be configured. Further, when performing filter calculation, processing such as padding may be used in combination, or an appropriately set stride width may be employed.

As the neural network, a multi-layered or super multi-layered neural network with tens to thousands of layers may be used.

(Visual recognition target identification unit)
The visual recognition target identification unit 113 refers to the face information acquired by the face information acquisition unit 111 or the detection result acquired from the state detection unit 112 to identify the target visually recognized by the learner. The object to be visually recognized (visible object) may be a person or an object.

In addition, the visual target identification unit 113 identifies the degree of concentration of the work learner based on the detection result obtained from the state detection unit 112 .

In this way, the information processing apparatus 100 can accurately determine the target visually recognized by the learner and the degree of concentration by including the visual recognition target identification unit 113 .

Specific processing of the visual target identification unit 113 will be described. The visual recognition target identification unit 113 identifies the visual recognition target by determining the position coordinates ahead of the learner's line of sight from the detection result acquired from the state detection unit 112 . For example, as shown in FIG. 3, the visual recognition target identification unit 113 identifies that the coordinates of the position ahead of the learner's line of sight are within the coordinates of the customer's feet. In addition to the line-of-sight information, the state of the pupil, the number of blinks, the movement of the eyebrows, the movement of the eyelids, the movement of the cheeks, the movement of the nose, the movement of the lips, and the movement of the jaw, as described in (state detection unit). By referring to the motion detection result, it is possible to more preferably identify which object the learner is concentrating on.

The visual recognition target identification unit 113 may identify which object the learner is concentrating on visualizing using the machine learning technique described in (state detection unit).

The visual target identification unit 113 transmits the identification result to the difference information derivation unit 118 .

(Voice information acquisition unit)
The voice information acquisition unit 114 acquires voice information including information on utterances made by the learner at each point in time. The voice information acquisition unit 114 can acquire the learner's voice information from the microphone of the head mounted display 200, for example. As the voice information of the learner, for example, voice frequency, voice volume, etc. are acquired. A known speech recognition technique can be used to specify the speech information.

The voice information acquired by the voice information acquisition unit 114 is input to the decision making information identification unit 116 .

(Motion information acquisition unit)
The motion information acquisition unit 115 acquires motion information indicating the motion of at least part of the body of the work learner at each time point. The motion information acquisition unit 115 acquires from, for example, the motion sensor 230 of the head mounted display 200, or the motion sensor 310 or button 320 of the wearable interface 300, as shown in FIG. In addition, it is obtained from a photographed image of the learning body photographed by the camera 210 of the head-mounted display 200 .

The motion information acquired by the motion information acquiring unit 115 is input to the decision making information specifying unit 116 .

(Decision-making information identification unit)
The decision-making information specifying unit 116 specifies decision-making information indicating the intention of the learner at each time based on the voice information acquired by the voice information acquisition unit 114 and the motion information acquired by the motion information acquisition unit 115 . In the present embodiment, the decision-making information includes, for example, the learner's utterance content, voice tone, and action content.

As an example, the decision-making information identifying unit 116 identifies the utterance content and the tone of voice uttered by the learner. For example, as shown in FIG. 3, the decision-making information identifying unit 116 identifies the utterance content as text indicating the utterance content, such as "Welcome" or "What would you like to order?" Tones are specified as either high, medium, or low as an example.

Also, the decision-making information specifying unit 116 refers to the motion information acquired by the motion information acquiring unit 115 to specify the motion performed by the learner at each time point. For example, as shown in FIG. 3, the decision-making information specifying unit 116 specifies an action such as "bow" or "put water on".

Note that the decision-making information specifying unit 116 may specify the decision-making information using the machine learning method described in (state detection unit).

The decision making information specified by the decision making information specifying unit 116 is input to the difference information deriving unit 118 .

(Difference information derivation part)
The difference information derivation unit 118 refers to the visual target acquired from the visual target identification unit 113, the decision-making information obtained from the decision-making information identification unit 116, and the degree of concentration, and compares it with the information stored in the storage unit 140. Derive the difference as difference information. An example of derivation of difference information will be described with reference to FIGS.

FIG. 3 shows learner data specified by the visual recognition target specifying unit 113 and the decision-making information specifying unit 116 . FIG. 3 shows, as an example of the learner's data, the timing at which an object is visually recognized, the object to be visually recognized, the utterance content, the tone of voice, the action, and the degree of concentration.

FIG. 4 is a table showing an example of data of experienced customers stored in the storage unit 140 in advance. FIG. 4 shows, as an example of data of a veteran customer, the timing at which an object is visually recognized, the object to be visually recognized, the utterance content, the tone of voice, the action, and the degree of concentration. "First reference information", "second reference information", and "third reference information" in FIG. 4 respectively correspond to the visual recognition target, utterance, and action of the experienced customer service person.

In one example, the information processing apparatus 100 collects a series of decision-making (utterances, actions, etc.) made within a predetermined time from the timing when the learner or the veteran customer visually recognizes the object as one unit ("event ”). As shown in FIGS. 3-5, in one example, events include timing, sightings, utterances, actions, and concentration, and each event is identifiable by an ID. As shown in FIGS. 3 to 5, an event for a learner and an event for a veteran customer can be associated using an event ID as an example. In this specification, for example, as shown in FIG. 3, the data in the event ID 1 in the learner's data is performed within a certain period of time after the visual recognition target and the object visually recognized at the timing of 00;00;03. including decisions made by

In addition, the "timing" data in FIGS. 3 and 4 is, as an example, data indicating the point in time when the object is visually recognized. ) is completed, or a combination of a plurality of these may be used as data (not shown). Furthermore, in one example, the difference information derivation unit 118 refers to the timing information acquired from the visual target identification unit 113 and the decision making information identification unit 116, respectively, to determine the progress from visual recognition to decision making (utterance or action). The time may be calculated and included as data (not shown).

The difference information derivation unit 118 derives difference information for each event ID based on the learner's data and the veteran customer's data shown in FIGS. 3 and 4, respectively. FIG. 5 is a table showing an example of difference information data derived by the difference information derivation unit 118 . In FIG. 5, the difference information is information derived for each event ID, and the difference in the timing at which the object was viewed, the target to be viewed, the content of the utterance, the tone of voice, the movement, and the degree of concentration at the corresponding event ID. show.

“First difference information”, “second difference information”, and “third difference information” in FIG. 5 respectively correspond to the differences in visual recognition target, utterance, and action between the learner and the veteran.

The difference information is the information of "deviation" between the data of the learner shown in FIG. 3 and the data of the veteran customer shown in FIG. Quantitative information indicated by the value of -experienced customer service), or information on whether or not the data matches the data of the veteran customer service (qualitative information indicated by "O/X" in the table). In one example, as shown in FIG. 5, when the data of the learner and the data of the veteran customer do not match, the difference information derived by the difference information derivation unit 118 is the data of the veteran customer (" correct answer”).

In one example, according to the event ID1 in FIG. 3 indicating the learner's data, at the timing "00;00;03", the visual recognition target identification unit 113 determines that the learner's visual recognition target is "the customer's feet". are identified. Further, according to the information of the veteran customer, which is the information stored in advance in the storage unit 140, indicated by the event ID 1 in FIG. "eyes". Based on both data, the difference information deriving unit 118 determines that the visual recognition target of the learner does not match the veteran customer service person, and as shown in event ID 1 in FIG. to derive Further, the difference information derivation unit 118 derives the "customer's eyes", which are the visual recognition objects of the experienced customer service, as the correct answer.

Further, as shown in event ID 1 in FIG. 3 , at timing “00;00;03”, the decision-making information specifying unit 116 detects that the learner utters “Welcome” and performs a “bow” motion. that is specified. Further, according to the information pre-stored in the storage unit 140, as indicated by event ID 1 in FIG. ” operation. Based on both data, the difference information derivation unit 118 determines that the learner's utterance content and actions match those of the veteran customer, and as shown in event ID 1 in FIG. "○" is derived as information.

Then, as shown in event ID 1 in FIG. 3, at timing "00;00;03", for example, the decision-making information identifying unit 116 identifies that the learner's voice tone is "medium". . According to the information pre-stored in the storage unit 140, as indicated by the event ID 1 in FIG. 4, the voice tone of the veteran customer at timing "00;00;03" is "high". Based on both data, the difference information derivation unit 118 determines that the tone of voice of the learner is lower than that of the veteran customer, and as shown in event ID 1 in FIG. -" is derived.

Furthermore, at the timing “00;00;03”, the visual recognition target specifying unit 113 specifies that the learner blinks (times/minute) is “5”, as indicated by the event ID 1 in FIG. 3 . . According to the information pre-stored in the storage unit 140, as indicated by the event ID 1 in FIG. 4, the blink of the experienced customer at the timing "00;00;03" is "10". Based on both data, the difference information derivation unit 118 determines that the learner's blink is 5 less than the veteran customer's blink, and as shown in event ID 1 in FIG. derive

At the timing “00;00;05”, the visual recognition target specifying unit 113 specifies that the state of the pupil of the learner is “closed”, as indicated by event ID2 in FIG. 3 . Further, as indicated by event ID2 in FIG. 4, according to information pre-stored in the storage unit 140, the veteran customer's blink at timing "00;00;05" is "open". Based on both data, the difference information derivation unit 118 determines that the pupil condition of the learner does not match that of the veteran customer, and as shown in event ID 2 in FIG. x” is derived.

At the timing "00;00;05", the visual recognition target identification unit 113 identifies that the learner's gaze retention time (seconds) is "0.0", as indicated by event ID2 in FIG. . According to the information stored in advance in the storage unit 140, as indicated by the event ID 2 in FIG. 4, the veteran customer's gaze retention time at the timing "00;00;05" is "0.8". . Based on both data, the difference information derivation unit 118 determines that the learner's gaze retention time is 0.8 less than that of the veteran customer service, and as shown in event ID 2 in FIG. to derive "-0.8".

Also, at the timing “00;00;05”, the visual recognition target specifying unit 113 detects that the movement of the eyebrows and the movement of the cheeks are not detected in the expression of the learner, as indicated by the event ID2 in FIG. It is specified as "-". On the other hand, as shown in event ID2 in FIG. 4, according to the information pre-stored in the storage unit 140, the expression of the veteran customer service at timing "00; Cheek movement is "upward". In this case, the difference information derivation unit 118 determines that the eyebrow movement of the learner matches that of the experienced customer based on both data, and as shown in event ID 2 in FIG. "○" is derived as the difference information regarding. Further, the difference information derivation unit 118 determines that the learner's cheek movement does not match the veteran customer's cheek movement based on both data, and, as indicated by event ID 2 in FIG. "x" is derived as

Further, according to the visual recognition target identification unit 113 and the decision-making information identification unit 116, the learner visually recognizes the "customer's table" at the timing "00;00;40" as indicated by the event ID3 in FIG. , "What would you like to order?" On the other hand, as indicated by event ID 3 in FIG. 4, according to the information pre-stored in the storage unit 140, the customer visually recognizes the "customer's table", says "What would you like to order?" The timing for performing the operation of "putting water" is "00;00;33". Based on both data, the difference information derivation unit 118 determines that the group of data of the learner is 00; "+00;00;07" is derived as the difference information for

According to the information stored in advance in the storage unit 140, as indicated by event ID 4 in FIG. I would like to confirm your order." On the other hand, as shown in FIG. 3, neither the visual recognition target identification unit 113 nor the decision-making information identification unit 116 identifies the visual recognition target and the decision-making information of the learner at the timing “00;00;47”. Also, regardless of the timing, none of the visual recognition of "my hand", the utterance of "I will confirm the order", and the action of "reciting the details of the order" are specified. In such a case, the difference information derivation unit 118 determines that there is no data corresponding to the veteran customer's data in the learner's data based on both data, and as indicated by 600 in FIG. As information, "no data", "x", and correct information are derived.

From the table in FIG. 5, it is determined that learners tend to be late in taking orders because they do not look customers in the eye. Also, from the table in FIG. 5, it is determined that the learner has not confirmed the order with the customer.

The difference information derivation unit 118 may derive the difference information using the machine learning method described in (state detection unit).

The difference information derived by the difference information derivation unit 118 is input to the presentation information generation unit 119 .

(Student Information Acquisition Department)
The learner information acquisition unit 117 acquires learner information managed by the information management server 400 as learner information, for example. As an example, the learner information includes attribute information indicating attributes of the learner, and learner identification information for identifying other learners and the target learner. The learner's attribute information includes, for example, the learner's age, sex, work history, and the learning period so far using the presentation system 800 . Further, the learner identification information is, for example, a learner's ID or a learner's e-mail address.

In this way, the learner information includes the attribute information and the learner identification information, so that the presentation information generation unit 119, which will be described later, provides effective presentation for each learner according to the attributes of each learner. Can present information. Details will be described in the item of (presentation information generation unit).

The learner information acquisition unit 117 inputs the learner information acquired from the information management server 400 to the presentation information generation unit 119 .

(Presentation information generator)
The presentation information generating unit 119 uses the difference information derived by the difference information deriving unit 118 (the visual target and the degree of concentration identified from the face information by the visual target identifying unit 113, the utterance and movement identified by the decision-making information identifying unit 116 from the voice information). derived based on the action specified from the information) to generate the presentation information. In one example, the presentation information generated by the presentation information generator 119 is an image.

The presentation information generating unit 119 generates presentation information in which the target visually recognized by the experienced customer is highlighted when the target visually recognized by the learner is different from the target visually recognized by the experienced customer. FIG. 8 is a diagram showing an example of the presentation information generated by the presentation information generation unit 119. As shown in FIG. An image 200C shown in FIG. 8 is an image in which an object to be visually recognized by the learner is highlighted, and an arrow pointer 200C1 is added as a symbol to the image 200A shown in FIG. In FIG. 8, arrow pointer 200C1 is in the shape of an arrow, but may be of any shape and color.

Other examples of highlighting the visual target include framing the visual target, changing the background color around the visual target, displaying the visual target larger than other images, and displaying the visual target. For example, an image is displayed as a pop-up window.

The presentation information generated by the presentation information generation unit 119 may always include highlighting. good.

Also, the presentation information generation unit 119 may calculate a score according to the difference information derived by the difference information derivation unit 118 and include the calculated score in the presentation information.

FIG. 7 is a diagram showing an example of the presentation information generated by the presentation information generation unit 119. As shown in FIG. An image 200B shown in FIG. 7 includes an instruction window 200B1 and a score window 200B2 as text.

The instruction window 200B1 in the image 200B is a window that displays text that guides the learner's utterance content and actions. The contents of the guide are based on the data of experienced customers stored in the storage unit 140 in advance. The text, as shown in FIG. 7, may instruct the learner on all of the utterances and actions to be performed, for example, "The customer looked up from the menu, what would you like to do? , etc., to give a hint to the learner.

A score window 200B2 in the image 200B is a window that displays a score indicating the proficiency level of the learner. As a method of calculating the score, for example, (i) the method of adding or subtracting points is divided between the difference information determined by matching or non-matching and the difference information determining the degree of deviation, and a predetermined score is calculated for the mismatching item. A method of deducting points and changing the number of points to be deducted according to the degree of deviation for items that are deviated, and (ii) not only the difference information determined by the degree of deviation, but also the difference information determined by match or mismatch in advance. If it does not deviate from the defined range (for example, the veteran customer's visual recognition target is "customer's eyes", whereas the learner's visual recognition target is "customer's nose") For example, the content is "What would you like to order?" and the content of the learner's utterance is "I would like to ask for your order." methods, etc., but are not particularly limited.

In one example, the presentation information generated by the presentation information generator 119 includes audio (not shown). The voice may be a voice or a sound effect. In one example, the voice reads aloud the text displayed in instruction window 200B1. Examples of sound effects include sounds that indicate correct or incorrect answers ("ping-pong" or "boo") according to the content of the difference information, and the highlighting of the visual recognition target that draws the learner's attention. Prompting sound etc. are mentioned.

The presentation information generation unit 119 refers to the learner information acquired by the learner information acquisition unit 117 when generating the presentation information. By referring to the learner information, the presentation information generation unit 119 can present presentation information that enables each learner to learn effectively according to the attribute of each learner.

For example, if the learner in the present embodiment has never experienced a hospitality business, the presentation information generation unit 119 generates presentation information that emphasizes the visual recognition target, or creates a guide with more specific content. and generate presentation information including Also, for example, when the learner's previous learning period is longer than a predetermined period, the presentation information generation unit 119 generates presentation information including a score indicating the learner's proficiency level calculated under stricter conditions. can be done. Furthermore, in one example, the presentation information generation unit 119 can generate presentation information including the voice of the learner's favorite anime character or the like.

The presentation information generation unit 119 transmits the generated presentation information to the head mounted display 200 . In addition, presentation information generating section 119 transmits the presentation information to display section 500 .

<Display part>
As shown in FIG. 1 , the presentation system 800 has a display section 500 . The display unit 500 displays the presentation information generated by the presentation information generation unit 119 mainly for instructors who instruct learners. The image displayed by the display unit 500 may include the table showing the learner's data shown in FIG. 3 and the table showing the difference information shown in FIG. A table showing the data may be included. Also, the image displayed by the display unit 500 may include part of the image displayed by the display 270 of the head mounted display 200 .

2. Processing Example of Presentation System FIG. 6 is a flow chart showing processing of the presentation system 800 . Processing of the presentation system 800 will be described based on FIG.

First, use of the presentation system is started (step S100) to start processing. The process proceeds to step S (hereinafter, “step” is omitted) 102 .

In S102, the learner information acquisition unit 117 in the control unit 110 of the information processing apparatus 100 acquires learner information (learner information acquisition step). The details of the processing are as described in (Learner Information Acquisition Unit). The learner information acquisition unit 117 transmits the learner information to the presentation information generation unit 119, and proceeds to S104.

In S104, the face information acquisition unit 111 acquires from the camera 210 in the head mounted display 200 of the learner and extracts each part of the face from the face image (face information acquisition step). The details of the processing are as described in (Face Information Acquisition Unit). The face information acquisition unit 111 transmits the extracted result to the state detection unit 112, and proceeds to S106.

In S106, the state detection unit 112 detects the learner's visual recognition target, the blink state, the state of the pupil, and the state of each part of the face (state detection step). The details of the processing are as described in (state detector). The detection result is transmitted to the visual recognition target identification unit 113, and the process proceeds to S108.

In S<b>108 , the visual target identifying unit 113 identifies the visual target visually recognized by the learner (visual target identification step). The details of the processing are as described in (Visual recognition target identification unit). At this time, the visual recognition target identification unit 113 may identify the degree of concentration during visual recognition at the same time as identifying the visual recognition target. The visual target identifying unit 113 transmits the identified result to the difference information deriving unit 118, and proceeds to S110.

In S110, at least one of the voice information acquisition unit 114 and the motion information acquisition unit 115 acquires the learner's decision-making information from the microphone 220 and the motion sensor 310 included in the head-mounted display 200 (decision-making information acquisition step). The details of the processing are as described in (Voice Information Acquisition Unit) and (Motion Information Acquisition Unit). Proceed to S112.

In S112, at least one of the voice information acquisition unit 114 and the motion information acquisition unit 115 refers to whether voice information, motion information, or the like (hereinafter sometimes referred to as “decision-making information”) has been acquired to determine whether or not the decision is made. Determine the presence or absence of determination information. If there is decision-making information, the voice information acquisition unit 114 and the motion information acquisition unit 115 transmit the acquired decision-making information to the decision-making information specifying unit 116, and the process proceeds to S113. If there is no decision-making information, the process proceeds to S114.

In S113, the decision-making information identification unit 116 identifies decision-making information such as the learner's utterance content, tone of voice, and action (decision-making information identification step). The details of the processing are as described in (decision-making information specifying unit). The decision making information specifying unit 116 transmits the specified result to the difference information deriving unit 118, and proceeds to S114.

In S114, the difference information derivation unit 118 acquires reference information such as a visual recognition target and decision-making information of the experienced customer stored in advance in the storage unit 140 (reference information acquisition step). Proceed to S116.

In S116, the difference information deriving unit 118 refers to the visual target identified by the visual target identifying unit 113, the decision making information identified by the decision information identifying unit 116, and the reference information acquired from the storage unit 140, and determines the difference. Derive information (difference information derivation step). The details of the processing are as described in (difference information derivation unit). The difference information derivation unit 118 transmits the derived difference information to the presentation information generation unit 119, and proceeds to S118.

In S118, the presentation information generation unit 119 refers to the difference information derived by the difference information derivation unit 118 and the learner information acquired from the learner information acquisition unit 117 to generate presentation information (presentation information generation step). . The details of the processing are as described in (Presentation Information Generation Unit). The control unit 110 of the information processing apparatus 100 causes the communication unit 130 to transmit the presentation information generated by the presentation information generation unit 119 to the head mounted display 200 and the display unit 500, and proceeds to S120.

In S120, the control unit 250 of the head mounted display 200 causes the display 270 or the like to display the presentation information generated by the presentation information generation unit 119 (presentation information display step), proceeds to S122, and ends the processing. At this time, the control unit 250 of the head mounted display 200 may output the presentation information generated by the presentation information generation unit 119 to the speaker 240 or the like as audio (not shown).

7 and the arrow pointer 200C1 in FIG. 8 may not necessarily be displayed in real time during learning of the customer service work in progress. The presentation information generated in the learning may be displayed at the time of learning the customer service work from the next time onward. Further, the information processing apparatus 100 may have a function of recording an image displayed on the display 270 or the like in the storage unit 140, and may be configured to display each piece of presentation information when the image is reproduced later. Moreover, it is the same also in Embodiment 2 mentioned later.

[Embodiment 2]
1. Presentation System The presentation system 800a according to the present embodiment has functions equivalent to those of the presentation system 800 shown in FIG. Present to learners. As a result, it is possible to more preferably support independent and experiential learning for the work learner. In the following, as in the first embodiment, a case where the presentation system 800a is used in a situation where a learner who works at a restaurant or the like where customer service is required learns customer service work will be described as an example.

FIG. 9 is a block diagram illustrating components of a presentation system according to one embodiment of the invention. As shown in FIG. 9, in the presentation system 800a according to the present embodiment, the control unit 110a of the presentation system 800 shown in FIG. 1 further includes a difference analysis unit 120 that analyzes the difference information. In addition, the presentation information generation unit 119a according to the present embodiment generates presentation information by referring to the judgment result input from the difference analysis unit 120 in addition to the difference information input from the difference information derivation unit 118. FIG. Further, the storage unit 140a according to the present embodiment has a judgment rule database 141, which is a database regarding judgment rules. As another aspect, the determination rule database 141 may be configured not to be included in the storage unit 140 but to be stored separately in a memory (not shown) or the like.

(Difference Analysis Department)
The difference analysis unit 120 performs determination processing by applying a determination rule read from the determination rule database 141 to at least one of the first to third difference information derived by the difference information derivation unit 118 .

Among the difference information shown in FIGS. 3 to 5, all of the information on the "difference" between the data of the learner and the data of the veteran customer and the information on whether or not the data of the veteran customer matches is presented to the learner. , it becomes rather difficult for the learner to understand which data to pay attention to. Therefore, if the difference between the two data is within a predetermined threshold or if a predetermined criterion is satisfied, it may be preferable not to present the difference to the learner. The judgment rule contained in the judgment rule database 141 may be interpreted as a data set that defines the above threshold or judgment criteria for each difference.

For example, at a certain timing, when the visual recognition target of the veteran customer service is the customer's eyes, and the determination rule indicates within a predetermined range including the customer's eyes, the difference analysis unit 120 Even if the visual recognition target is not exactly the customer's eyes, it may be determined that it is permissible as long as it is within the above-mentioned predetermined range, and that the difference in the visual recognition target is not presented to the learner. On the other hand, the difference analysis unit 120 may determine that the difference in the visual recognition target is presented to the learner when the visual recognition target of the learner is his or her hand in the above case. In addition, the above-described predetermined range that is allowed as not to be presented to the learner may be a plurality of discontinuous ranges.

Also, for example, at a certain timing, if the utterance of a veteran customer is "What would you like to order?" The difference analysis unit 120 may determine that the difference in the utterance content is not presented to the learner, assuming that the same content is shown. On the other hand, when the content of the learner's utterance is "I will confirm your order" in the above case, the difference analysis unit 120 determines that the difference in the content of the utterance is presented to the learner. good. Further, as in the example described above, the difference analysis unit 120 may determine that if a predetermined portion of the utterance content matches, the utterance content indicates the same content. Further, the judgment rule database 141 may include a data set indicating the learner's utterance content that is permitted with respect to the utterance content of the experienced customer. Further, a threshold may be set for whether or not to present to the learner not only the content of the utterance itself but also the pitch of the tone of voice.

Further, for example, at a certain timing, when the action of the experienced customer is "show the front", even if the action of the learner is "state the table number", the difference analysis unit 120 may be judged as not presenting the difference in the motion to the learner. On the other hand, in the above case, when the learner's action is "ask for an order" or the like, the difference analysis unit 120 may determine that the difference in the action should be presented to the learner. Further, the judgment rule database 141 may include a data set indicating actions of learners that are permitted in addition to certain actions of experienced customers.

Also, not limited to the above examples, for example, the judgment rule database 141 includes a threshold for the difference in the timing of each action between the two, and the difference analysis unit 120 provides the learner with the difference in timing exceeding the threshold. It may be judged as presented.

In addition, the difference analysis unit 120 collects information indicating which visual recognition target is visually recognized by the veteran customer service person and the learner at each time, and information regarding the time at which each visual recognition target is visually recognized within a predetermined time. Generate. An example of each of the above information will be described later.

2. Processing Example of Presentation System FIG. 10 is a flowchart showing processing of the presentation system 800a. Processing of the presentation system 800a will be described based on FIG.

In the processing based on the flowchart of FIG. 10, the same processing as in the first embodiment is executed from S100 to S116. After the processing in S116 is performed, the process proceeds to S117.

In S117, the difference analysis unit 120 performs determination processing by applying the determination rule read from the determination rule database 141 to at least one of the first to third difference information derived by the difference information derivation unit 118. . The details of the processing are as described in (difference analysis section). In addition, the difference analysis unit 120 collects information indicating which visual recognition target is visually recognized by the veteran customer service person and the learner at each time, and information regarding the time at which each visual recognition target is visually recognized within a predetermined time. Generate. The difference analysis unit 120 transmits the determination result of the determination process and each of the above information to the presentation information generation unit 119a, and then proceeds to S118'.

In S<b>118 ′, the presentation information generation unit 119 a refers to the determination result by the difference analysis unit 120 to generate presentation information. In other words, the presentation information generation unit 119a generates presentation information for presenting the difference information determined by the difference analysis unit 120 to be presented to the learner. In addition, the presentation information generation unit 119a may include information about the visual recognition object input from the difference analysis unit 120, and the information illustrated in FIG. 11 in the presentation information. FIG. 11A is a diagram showing which visual recognition target the learner has visually recognized at each time. Further, in FIG. 11, O1 to O5 respectively indicate visual recognition objects, and for example, O1 corresponds to "customer's eyes". Further, FIG. 11B is a diagram showing the ratio of the time during which the learner visually recognized each visual target, which is information related to the time during which each visual target was visually recognized within a predetermined time. In addition, the presentation information generation unit 119a may generate the information illustrated in FIG. 11 for the visually recognized object that was visually recognized by the experienced customer service person. The presentation information generation unit 119a may also perform the same processing as the presentation information generation unit 119 in S118 of the first embodiment. Proceed to S120'.

In S120', the control unit 250 of the head mounted display 200 causes the display 270 or the like to display the presentation information generated by the presentation information generation unit 119a (presentation information display step), proceeds to S122, and ends the processing. At this time, the control unit 250 of the head mounted display 200 may output the presentation information generated by the presentation information generation unit 119a to the speaker 240 or the like as audio (not shown). Note that all of the presentation information displayed on the display 270 or the like may not be displayed on one screen, and for example, information displayed may be switchable according to a user's instruction. In addition, in S118′, when the presentation information generation unit 119a generates the information illustrated in FIG. 11 for the visual recognition object that the veteran customer has visually recognized, the control unit 250 makes the veteran customer and the learner The corresponding information illustrated in FIG. 11 may be displayed on the display 270 or the like for comparison. As a result, the learner can easily comprehend the visual recognition target that the learner should pay more attention to by comparing with the data of the veteran customer.

[Example of realization by software]
The control block of the information processing device 100 (100a) may be implemented by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be implemented by software.

In the latter case, the information processing apparatus 100 is provided with a computer that executes instructions of a program, which is software that implements each function. This computer includes, for example, at least one processor (control device) and at least one computer-readable recording medium storing the program. In the computer, the processor reads the program from the recording medium and executes it, thereby achieving the object of the present invention. As the processor, for example, a CPU (Central Processing Unit) can be used. As the recording medium, a "non-temporary tangible medium" such as a ROM (Read Only Memory), a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. In addition, a RAM (Random Access Memory) or the like for developing the above program may be further provided. Also, the program may be supplied to the computer via any transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. Note that one aspect of the present invention can also be implemented in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

The information processing apparatus according to each aspect of the present invention may be implemented by a computer. In this case, the information processing apparatus is implemented by the computer by operating the computer as each part (software element) included in the information processing apparatus. A control program for an information processing apparatus realized by a computer and a computer-readable recording medium recording it are also included in the scope of the present invention.

The present invention is not limited to the above-described embodiments, but can be modified in various ways within the scope of the claims, and can be obtained by appropriately combining technical means disclosed in different embodiments. is also included in the technical scope of the present invention. Furthermore, new technical features can be formed by combining the technical means disclosed in each embodiment.

100, 100a information processing apparatus 110, 110a control unit 111 face information acquisition unit 112 state detection unit 113 visual recognition target identification unit 114 voice information acquisition unit 115 motion information acquisition unit 116 decision-making information identification unit 117 learner information acquisition unit 118 difference information Derivation unit 119, 119a Presentation information generation unit 120 Difference analysis unit 130 Communication unit 140, 140a Storage unit 141 Judgment rule database 200 Head mounted display 210 Camera 220 Microphone 230 Motion sensor 240 Speaker 250 Control unit 260 Communication unit 270 Display 300 Wearable interface 310 motion sensor 320 button 400 information management server 500 display unit

Claims (12)

a face information acquisition unit that acquires face information including at least part of face information of a work learner who learns customer service work;
a voice information acquisition unit that acquires voice information including information on the utterance by the working learner;
First difference information indicating a difference between the state of at least a part of the face indicated by the face information acquired by the face information acquisition unit and first reference information including information on the state of at least a part of the model worker's face and a difference information derivation unit for deriving second difference information indicating a difference between the voice information acquired by the voice information acquisition unit and second reference information including information about the voice of the model worker ;
An information processing apparatus, comprising: a presentation information generation unit that generates presentation information corresponding to at least one of the first difference information and the second difference information. Further comprising a visual target identifying unit that identifies a target visually recognized by the work learner by referring to the face information,
2. The information processing apparatus according to claim 1, wherein the difference information derivation unit derives the first difference information by referring to the visual target identified by the visual target identification unit. The visual recognition target identification unit further identifies the degree of concentration of the work learner on the target visually recognized by the work learner,
3. The information processing apparatus according to claim 2, wherein the difference information derivation unit derives the first difference information by referring to the degree of concentration specified by the visual recognition target specifying unit. When the target visually recognized by the work learner is different from the target indicated by the first reference information, the presentation information generation unit generates presentation information in which the target indicated by the first reference information is highlighted. 4. The information processing apparatus according to claim 2, wherein: 5. The information processing apparatus according to any one of claims 1 to 4, wherein the presentation information generation unit generates presentation information according to the difference in sound indicated by the second difference information. 6. The presentation information generator according to any one of claims 1 to 5, wherein the presentation information generating unit calculates scores according to the first difference information and the second difference information, and includes the calculated scores in the presentation information. 1. The information processing apparatus according to 1. further comprising a motion information acquiring unit that acquires motion information indicating motion of at least part of the body of the work learner,
The difference information deriving unit derives third difference information indicating a difference between the motion information and third reference information,
7. The information processing apparatus according to any one of claims 1 to 6, wherein the presentation information generated by the presentation information generation unit includes information corresponding to the third difference information. a difference analysis unit that performs determination processing by applying a determination rule to at least one of the first difference information, the second difference information, and the third difference information;
8. The information processing apparatus according to any one of claims 1 to 7, wherein the presentation information generation unit generates the presentation information by referring to the determination result by the difference analysis unit. The difference analysis unit generates information indicating which visual target is being visually recognized at each time,
9. The information processing apparatus according to claim 8, wherein the presentation information generating unit includes information indicating which viewing target is being viewed at each time in the presentation information. The difference analysis unit generates information regarding the time at which each visual object was viewed within a predetermined time,
10. The information processing apparatus according to claim 8, wherein the presentation information generation unit includes in the presentation information information about a time at which each visual recognition target is visually recognized within the predetermined time. A presentation system comprising the information processing device according to any one of claims 1 to 10 and a head-mounted display,
The head mounted display is
an imaging unit that captures at least a portion of the work learner's face;
a sound collecting unit that collects the uttered voice of the working learner;
and a presentation unit that presents the presentation information. A program for causing a computer to function as the information processing apparatus according to claim 1 .

Images