JP2016093279A - Control apparatus, control apparatus operation method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To shorten a time for preparation and execution required for reminiscence therapy.SOLUTION: A control apparatus 1, that is a control apparatus to control an image presentation apparatus 2 and a robot 3 used by a user U, switches the operation mode of the control apparatus 1 among a reminiscence acceleration mode, a speech mode, and a display image speech mode on the basis of a speech by the user U and the direction of the user U's attention, and controls the image presentation apparatus 2 and the robot 3 in association.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image presentation apparatus and a robot control technology that are intended to reduce the time required for preparation and execution necessary for reminiscence therapy by automation.

  Conventionally, when performing reminiscence therapy, it is possible to ask appropriate questions to elderly people who are receiving therapy, present pictures that encourage conversations by elderly people, and appropriately speak to elderly people. It was necessary to produce a state in which elderly people can continue to interact with each other by hitting and competing. For this reason, there are many things to implement, such as the formation of human resources (therapists) with skills to implement reminiscence therapy, ensuring the operation of therapists to implement reminiscence therapy, and preparing photos to encourage conversation among the elderly. Needed human resources.

  In the prior art, as one of the methods for supporting reminiscence therapy, reminiscence therapy support using a memory video and a remote videophone as in Non-Patent Document 1 has been proposed. In the method of Non-Patent Document 1, reminiscence therapy is performed while a therapist operates a memory video from a remote place with a slideshow of old photos of the elderly, so that the therapist can recall from a remote location without directly facing him. The therapy is realized. However, in order to create a memory video necessary for this method, it is necessary to select photos that are related to the elderly in advance, import the photos to the PC, and then import the location name, person name, shooting date, etc. of the photos as annotation data . For this reason, the creation of a memory video for each individual who performs reminiscence therapy has not been easy, and has been one of the challenges in implementation. In addition, reminiscence therapy by remote videophone can reduce the time required for movement because it is no longer necessary for the therapist to visit the elderly's house, while therapist needs to perform therapy at a remote location when reminiscence therapy is performed. There was still a challenge in terms of requiring human operation.

  In Patent Document 1, for each user who receives reminiscence therapy, a time attribute indicating “when”, a spatial attribute indicating “where”, a target attribute indicating “what”, and “what happened” A method of performing reminiscence therapy by generating a question sentence from an event database by registering each value of the behavior attribute to be shown as an event database and by asking the user a question has been proposed. However, in this method, it is necessary to register each user's action in the event database, and for automatic acquisition, it is necessary to take a method such as attaching various sensors to the user or the environment. Further, even in the case of automatic acquisition, it is generally not easy to automatically acquire the target attribute indicating “what” and the behavior attribute indicating “how”. Therefore, it is necessary to prepare facilities such as a sensor device for automatic event extraction in the user or environment, or it is necessary to register daily events for each user by a caregiver, and there remains a problem.

  Further, in the prior art, an invention such as [Patent Document 2] has been proposed in which an image to be displayed on an image display device is switched in accordance with the content of dialogue between humans. However, when these conventional techniques are applied to reminiscence therapy, there is an advantage that image stimuli can be provided to elderly people according to conversation without preparing image stimuli that facilitate recollection in advance. Needs to be accompanied by the elderly, and when the elderly wants to expand the story on the displayed image, the image display will be switched, so the attendant or the elderly themselves must operate the device, One of them learned how to use the device, and each had to operate appropriately so as not to stop the flow of conversation.

JP 2010-92358 A JP 2009-294866 A

Realization and evaluation of a system that enables remote conversation and reminiscence to elderly people using TV phone and content sharing. Journal of Human Interface Society 9 (2), 111-122, 2007. A study on speech segment detection using Dirichlet prior distribution. IEICE technical report. Language comprehension and communication 109 (355), 65-70, 2009. An automatic extraction method for proper terms contained in Japanese sentences. National Conference Proceedings 41st Late 1990, 227-228, 1990

  This invention is made | formed in view of said subject, The place made into the objective is to reduce the prior preparation required for reminiscence therapy, and the time of implementation by automation.

  In order to solve the above-described problems, a first aspect of the present invention is an image presentation device used by a user and a control device for controlling a robot, based on the utterance by the user and the direction of attention directed by the user. The operation mode of the control device is switched between a recollection promotion mode, a dialogue mode, and a display image dialogue mode, and the image presentation device and the robot are controlled in conjunction with each other.

  For example, in the recollection promotion mode, the robot is controlled so as to face the user, and the robot is controlled to speak a question sentence for causing the user to start recollection. The operation of controlling the image presentation device to switch and display the image searched by the search word string obtained from the user's utterance, and controlling the robot to face the user and listening to the user Is controlled so that the robot executes, and in the display image dialogue mode, the image presentation device is controlled to continuously display an image searched by a search word string obtained from the user's utterance, The robot is controlled so that the robot faces the image presentation device that displays the continuously displayed image, and the utterance text corresponding to the search word string is Tsu door is controlled so as to speak.

  For example, when the user's utterance is detected in the recollection promotion mode, the mode changes to the dialogue mode, and in the dialogue mode, the user starts to pay attention to the image presentation device. When detected, the display image dialogue mode is changed, while when the user's utterance is detected not to exceed a certain time, the recollection promotion mode is changed. When the display image dialogue mode is selected, the user is changed to the display image dialogue mode. Transitions to the dialogue mode when detecting that the user has finished paying attention to the image presentation device, while transitioning to the recollection promotion mode when it is detected that the user's utterance has not exceeded a predetermined time. .

  For example, in the dialog mode, at least one of the first condition that the length of the section in which the dialog by the user is stopped is longer than a predetermined first threshold, and the user's face or line of sight is the When the user satisfies the second condition that the length of the section that is suitable for the image presentation device and the conversation by the user continues is longer than a predetermined second threshold, the user It determines with having started paying attention to a presentation apparatus.

  For example, in the display image dialogue mode, the length of time after detecting that the state where at least one of the user's face or line of sight is facing the image presentation device has ended is greater than a predetermined threshold value. When the long condition is satisfied, it is determined that the user has finished paying attention to the image presentation device.

  A second aspect of the present invention is an operation method of a control device for controlling an image presentation device and a robot used by a user, wherein the control device is based on the direction of the utterance by the user and the direction of attention directed by the user. The operation mode of the control device is switched among a recall promotion mode, a dialogue mode, and a display image dialogue mode, and the image presentation device and the robot are controlled in conjunction with each other.

   The third aspect of the present invention is a computer program for causing a computer to function as the control device according to the first aspect of the present invention.

  ADVANTAGE OF THE INVENTION According to this invention, the advance preparation required for reminiscence therapy and the time of implementation can be reduced by automation.

It is a block diagram which shows schematic structure of the control apparatus which concerns on this Embodiment. It is a figure which shows arrangement | positioning of the control apparatus 1 and another apparatus. It is a figure which shows the operation example in recollection promotion mode. It is a 1st figure which shows the operation example in dialog mode. It is a 2nd figure which shows the operation example in dialog mode. It is a 3rd figure which shows the operation example in dialog mode. It is a figure which shows the operation example in display image dialog mode. It is a figure which shows the example of a transition of an operation mode. It is a transition diagram of an operation mode. It is a figure which shows the variable used for the detection of the user U having started paying attention to the image presentation apparatus. It is a figure which shows the variable used for the detection of the user U having finished paying attention to the image presentation apparatus. It is a figure which shows the structure of the information of the recollection start question memory | storage device 108. FIG. FIG. 4 is a diagram illustrating a configuration of information in a collected image storage device 109. It is a figure which shows the structure of the information of the present operation mode memory | storage device 111. FIG. It is a figure which shows the structure of the information of the attention object result storage device. 4 is a flowchart of processing of a speech recognizer 101. 4 is a flowchart of processing of a search word extractor 102. 10 is a flowchart of processing of a caution target determiner 105. 5 is a flowchart of processing of an operation mode determination unit 106. It is a flowchart of a process of the dialog related image search device 103. 5 is a flowchart of processing of a robot motion generator 107. 5 is a flowchart of processing of a display image operation device 104.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  In the present embodiment, a method is proposed that switches display contents appropriately according to attention with little advance preparation and without the need of a therapist when performing reminiscence therapy.

  FIG. 1 is a block diagram showing a schematic configuration of a control device 1 according to the present embodiment. FIG. 2 is a diagram illustrating an arrangement of the control device 1 and other devices.

  The control device 1 is a device that controls the image presentation device 2 and the robot 3 used by the user U, and recalls the operation mode of the control device 1 based on the utterance by the user U and the direction of attention directed by the user U. The mode is switched between the promotion mode, the dialogue mode, and the display image dialogue mode, and the image presentation device 2 and the robot 3 are controlled in conjunction with each other.

  In order to perform the control, the control device 1 is connected to the microphone 4, the camera 5 on the image presentation device side, the camera 6 on the robot side, and the external image search device 7 via a cable or a communication line (such as the Internet). The It is assumed that the camera 5 is disposed at a position sufficiently close to the image presentation device 2. It is assumed that the camera 6 is disposed at a position sufficiently close to the robot 3. The microphone 4 has directivity and does not measure the voice uttered by the robot 3. The external image search device 7 is not limited to one.

  The microphone 4 acquires the voice of the user U. The cameras 5 and 6 are for measuring the face / line-of-sight direction of the user U, and are a Web camera, a camera capable of depth measurement, and the like. The robot 3 is capable of voice output and face orientation adjustment. Note that the control device 1, the image presentation device 2, the microphone 4, and the cameras 5 and 6 may be included in the robot 3. The cameras 5 and 6 may be used for measuring one of the face direction and the line-of-sight direction. The voice of the robot 3 may be output from another device.

  The control device 1 searches the external image search device 7 for a speech recognizer 101 that performs speech recognition on the acquired user speech, a search word extractor 102 that extracts a search word from the speech recognition result, and the external image search device 7. Dialog-related image search unit 103 that performs image search by word and obtains a result, display image operation unit 104 that displays an image of the search result on image presentation device 2, and attention target determination unit 105 that determines the attention target of user U An operation mode determination unit 106 that changes the operations of the image presentation device 2 and the robot 3 according to the input, a robot operation generator 107 that operates the robot 3, and a recollection start question that the robot 3 refers to when making a question A sentence storage device 108, a collected image storage device 109 for storing images acquired from the external image search device 7, and a caution target result storage device 110 for storing caution target results It stores the current operation mode and a current operating mode storage unit 111.

  The speech recognizer 101 to the robot motion generator 107 are software modules, and the others are storage devices (hardware) in the control device 1. The search word extractor 102 has a function of extracting proper nouns from sentences of speech recognition results, for example.

  The control device 1 operates by switching the operation mode to any one of the recall mode, the interaction mode, and the display image interaction mode, and changes the operations of the image presentation device 2 and the robot 3 according to each operation mode.

  FIG. 3 is a diagram illustrating an operation example in the recollection promotion mode. 4, 5, and 6 are diagrams illustrating an operation example in the interactive mode. FIG. 7 is a diagram illustrating an operation example in the display image dialogue mode. FIG. 8 is a diagram illustrating an example of transition of the operation mode. FIG. 9 is a transition diagram of the operation mode.

  In the recollection promotion mode, the control device 1 controls the robot 3 so that the face of the robot 3 faces the user U, and controls the robot 3 to speak a question sentence for causing the user to start recollection.

  The recollection promotion mode continues unless the user U's utterance is detected, and the robot 3 detects the presence of the user's U face and line of sight and utters a question that triggers recollection to the user U (see FIG. 3 (1)).

  In the recollection promotion mode, for example, the robot 3 throws a question dialogue such as “Tell me about the place you enjoyed most when you were traveling as a child” to the user U. When the user U utters a response to the question of the robot 3, the operation mode is changed to the dialogue mode.

  In the interactive mode, the control device 1 controls the image presentation device 2 to switch and display the image searched by the search word string obtained by the utterance of the user U, and controls the face of the robot 3 to face the user U. Then, the robot 3 is controlled so as to execute the operation listening to the user.

  The dialogue mode continues as long as the user U's utterance is detected, and a search word is extracted from the user U's utterance sentence (FIG. 4 (2)), and an image search is performed based on the search word (FIG. 5 (4)). )), The image of the search result is displayed on the image presentation device 2 for a certain period of time. When a plurality of images are obtained as a search result from one search word, the images of the plurality of search results are switched in a certain time (FIG. 6 (5)). When a plurality of search words are extracted from the spoken sentence, an image search is performed for each of them, and images of a plurality of search results obtained by the plurality of search words are switched and displayed in a certain time (FIG. 6 (5)). .

  Further, in the dialogue mode, the robot 3 performs a listening operation composed of nodding, competing, and parrot returning of the recognized word in response to the utterance of the user U ((3) in FIG. 4).

  In the dialogue mode, when it is detected that the user U has stopped speaking for a certain time or longer, the operation mode is returned to the recall promotion mode (FIG. 8 (10)). In the interactive mode, when it is detected that the user U has paid attention to the image presentation device 2, the operation mode is shifted to the display image interactive mode.

  In the display image dialogue mode, the control device 1 controls the image presentation device 2 to continuously display the image searched by the search word string obtained by the utterance of the user U, and displays the continuously displayed image. The robot 3 is controlled so that the face of the robot 3 faces the image presentation device 2 to be controlled, and the robot 3 is controlled to speak the utterance text corresponding to the search word string.

  The display image interaction mode continues as long as the user U's attention to the display image is detected. In the display image dialogue mode, the image switching of the image presentation device 2 is stopped, and the image presentation at the timing when it is determined that the attention is directed to the image presentation device 2 is maintained (FIG. 7 (6)).

  In the display image dialogue mode, the robot 3 also performs speech and operation according to the display image. In the utterance and operation according to the display image, the robot 3 is directed to the image presentation device 2 with the direction of the line of sight (face and the entire robot), and the search word “N” of the image currently displayed on the robot 3 is used. A question sentence such as “I want to hear more about this story?” Is generated and the robot 3 is uttered.

  In the display image dialogue mode, when the presentation content is stopped, the speech operation is switched according to the presentation content ((7) in FIG. 7).

  Further, in the display image dialogue mode, when the end of attention of the user U to the image presentation device 2 is detected, the operation mode is returned to the dialogue mode, and image switching is resumed (FIG. 8 (8)). In the dialogue mode, listening operations such as nodding, companionship, and parrot return are performed (FIG. 8 (9)).

  Further, in the display image dialogue mode, when it is detected that the conversation of the user U has been stopped for a certain period of time, the operation mode is returned to the recall promotion mode, and a question that triggers the recall is uttered (FIG. 8 (10)). .

  As described above, in the recollection promotion mode, when the user U's utterance is detected, the mode transitions to the dialogue mode.

  Further, in the interactive mode, when it is detected that the user U has started to pay attention to the image presentation device 2, the display mode is changed to the display image interactive mode, and it is detected that the user U has not spoken for a certain time or more. If you do, transition to recollection promotion mode.

  Further, in the display image dialogue mode, when it is detected that the user U has finished paying attention to the image presentation device 2, the user U is detected that the utterance of the user U is not longer than a predetermined time while transitioning to the dialogue mode. If you do, transition to recollection promotion mode.

  The control device 1 has an algorithm for determining that the attention object of the user U is directed to the display image, in addition to each operation mode and mode transition described above. In general, it is known that a person performs an act of removing his / her line of sight from a conversation partner in order to think about the content of the conversation, adjust the intimacy with the conversation partner, and maintain a conversation number during the conversation. For this reason, if the sight line direction is simply deviated from the robot 3 and directed to the image presentation device 2, the display image is displayed even though the user U is not paying attention to the display image. The possibility of misjudging that attention has been directed to is increased. Therefore, an algorithm for improving the accuracy of detecting the attention transition to the display image of the user U is used.

  FIG. 10 is a diagram illustrating variables used to detect that the user U has started to pay attention to the image presentation device 2.

In this algorithm, two threshold values (threshold values) and two variables are used. The dialogue switching section judgment threshold t _Tnext and the display image attention transition judgment threshold t _toV are real numbers of 0 or more, and one of the variables, the conversation stop section length t _Tstop, is from the dialog detection stop until the next dialog detection starts. The length t _VandT of the face / gaze direction detection / dialogue direction continuation detection to the display image from the start of dialogue, which is the other variable, is the face to the display image after the start of dialogue is detected. -This is the length of the interval until either gaze detection or dialogue detection stops. That is, t _VandT is the length of a section in which the face / line-of-sight direction of the user U faces the image presentation device 2 and the dialogue by the user U continues. T _Tnext and t _Tstop are used to detect the start of dialogue. When t _Tnext <t _Tstop , the last time of t _Tstop is set as the dialog start time. Subsequently, t _VandT is detected from the dialogue start time. When t _toV <t _VandT is satisfied, it is determined that the user U has started to _pay attention to the image presentation device 2.

  In addition to the determination algorithm when detecting that the user U has started to pay attention to the image presentation device 2 as described above, the control device 1 similarly relates to the user U's attention target determination device. Has an algorithm for determination when it is detected that the user has finished paying attention to the image presentation device 2.

  FIG. 11 is a diagram illustrating variables used for detecting that the user U has finished paying attention to the image presentation device 2.

This algorithm uses one threshold value (threshold) and one variable. The display image attention end determination threshold t _noV is a real number greater than or _equal to 0, and the time t _Vstop after the detection of the face / gaze on the variable display image is stopped, that is, the face / gaze direction of the user U is the image. the length of time _{t Vstop} state facing the presentation device 2 from detection that it has _finished, if it meets _{t noV <t Vstop} against _{t nov,} user U attention to the image display device 2 It is determined that the direction is finished.

Here, data exchanged in the control device 1 shown in FIG. 1 will be described.
(1) The voice input is a stream of sound waveform data associated with the time measured by the microphone 4.
(2) Face / line-of-sight direction input is the face / line-of-sight direction for each of a maximum of one camera obtained from images acquired by the cameras 5 and 6, the face horizontal direction angle θ _{face_x} , and the face vertical direction The angle θ _{face_y} , the line-of-sight horizontal direction angle θ eye — _x , and the line of sight vertical angle θ _{eye — y} When the face of the user U and the line of sight are both facing the direction of the camera and are horizontal with the camera optical axis, θ _{face_x} = 0, θ _{face_y} = 0, θ _{eye_x} = 0, θ _{eye_y} = 0, It takes a value from -90 to 90 depending on the angle. As a method of taking a person's face and line-of-sight direction, for example, OMRON's Okao-Vision can be cited.
(3) The attention object result is information in which any one of the three character strings “attention to image presentation device”, “attention to robot”, and “not applicable to attention object” is associated with the time.
(4) The attention object result log is information in which one or more attention object results are collected.
(5) The operation mode result and the previous operation mode result are character string information of “display image dialog mode”, “interaction mode”, or “recollection promotion mode”.
(6) The question sentence character string is a character string describing conversation contents.
(7) The reference information update is command information for rewriting the last call time of the specified ID in the recollection start question message storage device 108.
(8) The speech character string is character string information generated from the voice input result.
(9) The search word string is character string information of proper nouns.
(10) The image data is an image file representing a photograph or illustration.
(11) The image search word pair is a set of an image related word and image data.
(12) Collected image information is a set of image display time, image storage time, image-related word, and image data.
(13) The image information update command is a command for rewriting and updating the current display flag and the image display time in the collected image storage device 109.
(14) The image display command is a command for causing the image presentation device 2 to display the image data.
(15) The robot control information is composed of motion instructions to be executed by the robot 3 and utterance texts. The robot 3 has an action of turning the face toward the image presentation device 2, an action of turning the face toward the user, and an action of shaking the head vertically. It is set in advance and can be executed by specifying any one of the robot control information, and when the utterance text of the robot control information is received, speech synthesis is performed on the text and voice output can be executed. To do.

  FIG. 12 is a diagram showing a configuration of information in the recollection start question message storage device 108.

  The recollection start question storage device 108 stores ID, final call time, and recollection start question for each of a plurality of recollection start questions sent to the user U so as to cause the user U to start recollection. With records to contain. ID is numerical data unique to each recollection start question sentence. The last call time is composed of time information, and indicates the time when each recollection start question sentence is last referred to from the recollection start question sentence storage device 108. In a recall start question sentence that has never been called, the last call time is null. The recollection start question text is text information of dialogue as shown in the example.

  FIG. 13 is a diagram illustrating a configuration of information in the collected image storage device 109.

  The collected image storage device 109 stores collected image information including ID, current display flag, image display time, image related word, and image data as identification information for each image data. The current display flag represents an image currently displayed on the image presentation device 2, and only the current display flag of one collected image information is true. The image display time represents the time when the current display flag becomes true. The image recording time represents the time when the image search word pair is registered in the collected image storage device 109. The image-related word is a character string that matches the image search word.

  FIG. 14 is a diagram illustrating a configuration of information in the current operation mode storage device 111.

  The current operation mode storage device 111 stores a current operation mode indicating the current operation mode. In the figure, the current operation mode indicates the display image interaction mode, but there are also cases where the interaction mode and the recall promotion mode are indicated.

  FIG. 15 is a diagram illustrating a configuration of information in the attention object result storage device 110.

  The attention object result storage device 110 stores information including the attention object result and the update time for each result of detecting the object of attention by the user U in time series. The attention object result is “attention to the image presentation apparatus” when the attention object is the image presentation apparatus 2, and “attention to the robot” when the attention object is the robot 3. If there is no target, “Not applicable for caution”. The update time indicates the time when each attention object result is added to the attention object result storage device 110.

  FIG. 16 is a flowchart of processing of the speech recognizer 101.

  As long as the voice recognizer 101 receives voice input, the voice recognizer 101 repeatedly performs the following processing.

  In S101, a voice input is received. If a voice input has been received, the process proceeds to S102.

  In S102, it is determined whether or not the speech section is continued in the voice input acquired in S101. Various methods are generally known as a method for identifying a person's utterance section in sound, and for example, the method of Non-Patent Document 2 can be mentioned. If there is a part that is divided as an utterance section in the voice input, the part from the voice input start to the part is sent to S103, the remaining voice input is held as a queue, and the held voice input is used for the next voice input acquisition. Information is added in the following form, and the process proceeds to S103 assuming that there is no continuation of the utterance section. If there is no portion delimited as the utterance section, all voice inputs are held as a queue, and the process returns to S101 to wait for reception of additional voice inputs.

  In S103, the speech input sent from S102 is converted to text by speech recognition, and if the text is converted, the process proceeds to S104.

  In S104, it is determined whether or not the speech recognition result includes a character string. If the text data contains no character string, the process returns to S101. If a character string is included, the process proceeds to S105.

  In S105, the speech recognition result (text string) is transmitted to the search word extractor 102 as an utterance character string. When the utterance character string is transmitted, the process is terminated.

  FIG. 17 is a flowchart of the processing of the search word extractor 102.

  When the search word extractor 102 receives the uttered character string from the speech recognizer 101, the search word extractor 102 starts the following processing.

  In S201, the utterance character string is received from the speech recognizer 101. When reception is completed, the process proceeds to S202.

  In S202, it is determined whether or not the utterance character string received in S201 includes a proper noun. Various methods for extracting proper nouns from text have been proposed, and Non-Patent Document 3 and the like can be used. If one or more proper nouns are included, the process proceeds to S203. If no proper noun is included, the process returns to S201.

  In S203, one of the proper nouns included in the utterance character string is selected at random, and the selected proper noun is transmitted as a search word string to the dialogue related image search unit 103. When transmission is completed, the process ends.

  FIG. 18 is a flowchart of the processing of the attention target determination unit 105.

In S301, the face / gaze direction inputs θ _{face_x} , θ _{face_y} , θ _{eye_x} , θ _{eye_y} are received from the camera 5 on the image presentation device side. These values are 0 degrees when both the face and line of sight face the camera 5. θ _{face_x} and θ _{eye_x} represent _{how much} the face direction and the line-of-sight direction are inclined in the horizontal direction from the front of the camera 5, respectively, and θ _{face_y} and θ _{eye_y} are the face direction and the line-of-sight direction perpendicular to the front of the camera 5, respectively. Indicates how many times it is tilted. When reception is completed, the process proceeds to S302.
In S302, it is determined whether or not the face / line-of-sight direction of the user is facing the image presentation device 2. For the determination, the thresholds for determining whether or not the line of sight and the face are facing the direction of the camera from which the face / line-of-sight direction is acquired are set to the values of _{θTHREAHOLD_x and θTHREAHOLD_y} . These values are positive real numbers from 0 to 90. Two conditional expressions

If both are satisfied, it is determined that the user's face / gaze direction is facing the image presentation device, and the process proceeds to S303. If either or both of the two conditional expressions are not satisfied, the process proceeds to S304. Only one of the two conditional expressions may be used. That is, the direction of the user's face is the image presentation device 2.
It may be determined only whether or not the user is facing, or only whether or not the direction of the user's line of sight is facing the image presentation device 2 may be determined.

  In S303, “attention to the image presentation device” is added to the attention object result storage device 110 as the attention object result. At the time of addition, the time at the time of addition is used as the update time. The process ends when the addition is made.

In S304, the user's face / gaze direction input θ ′ _{face_x} , θ ′ _{face_y} , θ ′ _{eye_x} , θ ′ _{eye_y} is received from the camera 6 on the user side. These values are 0 degrees when both the face and line of sight face the camera 6. θ ′ _{face_x} and θ ′ _{eye_x} represent how much the face direction and the line-of-sight direction are inclined in the horizontal direction from the front of the camera 6, respectively, and θ ′ _{face_y} and θ ′ _{eye_y} are the face directions in the vertical direction from the front of the camera 6, respectively. And how many times the line-of-sight direction is tilted. When reception is completed, the process proceeds to S305.

  In S305, it is determined whether or not the face / line-of-sight direction of the user is facing the robot 3. Here, two conditional expressions

It is determined whether or not the above is satisfied. If both of the two conditional expressions are satisfied, it is determined that the user's face / gaze direction is facing the robot 3, and the process proceeds to S306. If either or both of the two conditional expressions are not satisfied, the process proceeds to S307. Only one of the two conditional expressions may be used. That is, it may be determined only whether or not the direction of the user's face is facing the image presentation device 2, and only whether or not the direction of the user's line of sight is facing the image presentation device 2 may be determined.

  In S306, “attention to robot” is added to the attention object result storage device 110 as the attention object result. At the time of addition, the time at the time of addition is used as the update time. The process ends when the addition is made.

  In S307, “not applicable to caution target” is added to the caution target result storage device 110 as the caution target result. At the time of addition, the time at the time of addition is used as the update time. The process ends when the addition is made.

  FIG. 19 is a flowchart of the process of the operation mode determination unit 106.

  The operation mode determination unit 106 repeatedly executes the following processing when there is a voice input.

  In S401, a new voice input is received from the microphone 4 for a predetermined time. When reception is completed, the process proceeds to S402.

  In S402, the attention object determination result log is read from the attention object result storage device 110. When the acquisition is completed, the process proceeds to S403.

In S403, the operation mode stored in the current operation mode storage device 111 is read. Hereinafter, this operation mode is referred to as a previous operation mode result. After reading the previous operation mode result, the process proceeds to S404.
In S404, it is determined whether or not there is an utterance section in the voice input for a certain time received from the microphone 4. If there is no utterance section, the process proceeds to S411. If there is, the process proceeds to S405.

  In S405, it is determined whether or not the operation mode of the previous operation mode result received in S403 is the display image dialogue mode. If it is in the display image interactive mode, the process proceeds to S407. If it is not the display image interactive mode, the process proceeds to S406.

  In S406, it is determined in S403 whether or not the operation mode of the previous operation mode result is the interactive mode. If it is in the interactive mode, the process proceeds to S408. If not in the interactive mode, the process proceeds to S410.

In S407, the end of the user's attention to the image presentation device is detected based on the attention determination result log. Of the update times of the respective attention target results in the attention determination result log, the update time of the latest time that is the attention target result is attention to the image presentation device is t ₀ . At this time, if the time at the time of this calculation is t _now , the time t _Vstop after the detection of the face / gaze on the most recent display image is stopped as the difference between the time t _now and t ₀ , t _Vstop = t _now −t ₀ It is calculated as follows. If t _noV <t _Vstop is satisfied with respect to the display image attention end determination threshold t _noV , it is determined that the user's attention to the display image has ended. If it is determined that the attention has ended, the process proceeds to S408. Otherwise, the process proceeds to S409.
In S408, the user's attention start to the image presentation device is detected based on the attention determination result log and the voice input for a certain period of time.

First, an utterance section in a voice input for a certain period of time is extracted, a place where there has been a change from a non-speech section to a utterance section is extracted, and each time is set to t _{st_i} (where i is the place where the extracted part is a voice input) (An integer value indicating what point is seen from the start) If no extraction is performed, the process proceeds to S410. Then, the length of the non-speech section immediately before the time _{t ST_i} and _{t Stop_i} respectively in the speech input. When each t _{stop — i} is compared with the dialogue switching interval determination threshold t _Tnext , the time t _{st —} i of the portion i where the change from the non-speech interval _where t _Tnext <t _{stop —} i is _satisfied to the utterance interval is _defined as T _{st —} j (j is extracted) Integer value indicating whether a portion satisfying t _Tnext <t _{stop — i} corresponds to what point from the start of voice input). If there is no place corresponding to T _{st — j} , the process proceeds to S410.

Next, from the attention determination result log, the attention object result having the latest update time at the time before T _{st — j} is _set as the starting point D _j0 , and the attention object result having the latest update result in the attention determination result log is sorted in chronological order. _{j0, D} j1, ..., and _{D jn.} Furthermore, for D _j0 , D _j1 ,..., D _jn , whether or not the condition that the result of attention is “attention to the screen display device” is satisfied is started by _checking from D _j0 to the oldest update time. _Let D _jk be the log that _does not satisfy. At this time, if all the conditions are satisfied, k = n, and if none satisfy the conditions, k = 0. Further, for each time T _{st_j} where there is a change from the non-speaking section to the speaking section, the time when the next changing from the speaking section to the non-speaking section is _{Tet_j} . At this time, the T _{Et_j} the end time of the audio input when the audio input is terminated without changing the next speech segment of T _{St_j} the non-spoken section. Finally when the update time _{T jk} of _{D jk} to _the newer time at _{T jk} and _{T Et_j} and _{T J_end,} a _{t toV <T st_j -T} j_end the display image note transition determination threshold _{t Tov} extraction If there is a part j, the process proceeds to S409. If there is no corresponding extraction location, the process proceeds to S410.

  In S409, the current operation mode of the current operation mode storage device 111 is updated to the display image dialogue mode, and the operation mode result is transmitted to the robot operation generator 107 and the display image operation unit 104. When the update and transmission are completed, the process ends.

  In S410, the current operation mode of the current operation mode storage device 111 is updated to the interactive mode, and the operation mode result is transmitted to the robot operation generator 107 and the display image operation unit 104. When the update and transmission are completed, the process ends.

  In S411, the current operation mode of the current operation mode storage device 111 is updated to the recollection promotion mode, the operation mode result is transmitted to the robot operation generator 107 and the display image operation unit 104, and the process is terminated when the update and transmission are completed. .

  FIG. 20 is a flowchart of the process of the dialogue related image search unit 103.

  The dialogue related image search unit 103 executes the following process when receiving the search word string.

In S501, a search word string is received from the search word extractor 102. When reception is completed, the process proceeds to S502.
In S502, the search word string is transmitted to the external image search device 7. When the transmission is completed, the process proceeds to S503.

  In S503, the image data of the image searched by the search word string is received from the external image search device 7. When reception is completed, the process proceeds to S504.

  In S504, each image data received from the dialogue related image search unit 103 and the search word string are stored in the collected image storage device 109 as a pair (image search word pair). When saving is completed, the process is terminated.

  FIG. 21 is a flowchart of the process of the robot motion generator 107.

  When receiving the operation mode result, the robot operation generator 107 executes the following processing.

  In S601, the operation mode result is received from the operation mode determination unit 106. If reception is completed, the process proceeds to S602.

  In S602, it is determined whether or not the operation mode result received in S601 is the display image dialogue mode. If the result is the display image interactive mode, the process proceeds to S605. Otherwise, the process proceeds to S603.

  In S603, robot control information for controlling the robot 3 so that the face of the robot 3 faces the user U is transmitted to the robot 3. When the transmission is completed, the process proceeds to S604. Note that the robot 3 itself or the line of sight of the robot 3 may face the user U.

  In S604, it is determined whether or not the operation mode result received in S601 is the interactive mode. If the operation mode result is the interactive mode, the process proceeds to S611. Otherwise, the process proceeds to S615.

  In S605, robot control information for controlling the robot 3 is transmitted to the robot 3 so that the face of the robot 3 faces the image presentation device 2. When the transmission is completed, the process proceeds to S606. The robot 3 itself or the line of sight of the robot 3 may be directed toward the image presentation device 2.

  In S606, a voice input is received from the microphone 4 for a predetermined time. When the reception of the voice input is completed, the process proceeds to S607.

In S607, first, processing for obtaining an utterance section during voice input is executed. Further, a threshold t _{keep_talk} representing a time for determining continuation of speech expressed by a positive real number is set in advance, and if there is a non-speech interval of t _{keep_talk} or more from the end of the received voice input, the process _proceeds to S609. Proceed. In other cases, the process proceeds to S608.

  In S608, robot control information for causing the robot 3 to swing the head vertically (nodding) is transmitted to the robot 3. When the transmission is completed, the process ends.

  In S609, the display image search word is received from the display image operation unit 104, and when the reception is completed, the process proceeds to S610.

In S610, an utterance text is created using the display image search word Word _{now_v} received in S609. Utterance text can be created by applying display image search words to a fixed sentence. For example, “Hey, I _'m very interested in _{Wordnow_v} , so I want you to tell me a little more.” . When the utterance text creation is completed, the robot control information including the utterance text for operating the robot 3 is transmitted to the robot 3 so that the utterance text is synthesized and output. When the transmission is completed, the process ends.

  In S611, a voice input is received from the microphone 4 for a predetermined time. When the reception of the voice input is completed, the process proceeds to S612.

In S612, as in S607, first, processing for obtaining an utterance section during voice input is executed. Further, similarly to S607, a threshold t _{keep_talk indicating the} time for determining the continuation of utterance is set in advance, and if there is a non-speech interval of t _{keep_talk} or more from the end of the received voice input, the process proceeds to S613. Otherwise, the process proceeds to S614.

  In S613, robot control information for causing the robot 3 to swing the head vertically (nodding) is transmitted to the robot 3. When the transmission is completed, the process ends.

  In S 614, robot control information including the utterance text for uttering the robot 3 and synthesizing and outputting the utterance text is transmitted to the robot 3. As the utterance text, a predetermined text such as “So, then?” Is used. When the transmission is completed, the process ends.

  In S615, first, the recollection start question sentence is read out from the recollection start question sentence storage device 108 as a question sentence character string. In that case, refer to the last call time and read the recollection start question sentence of null. If there is no null thing, the recollection start question sentence having the oldest last call time in the recollection start question sentence storage device 108 is read out. When read, the reference information update is transmitted to the recollection start question sentence storage device 108 so that the last call time of the ID of the recollection start question sentence is changed to the current time value. When the transmission is completed, the process proceeds to S616.

  In S616, the robot control information including the question sentence character string for causing the robot 3 to synthesize and output (speak) the question sentence character string read in S615 as the utterance text is transmitted to the robot 3. When the transmission is completed, the process ends.

  FIG. 22 is a flowchart of processing of the display image operation unit 104.

  When the display image operation unit 104 receives the operation mode result, the display image operation unit 104 executes the following processing.

  In S701, the operation mode result is received from the operation mode determiner 106. If reception is completed, the process proceeds to S702.

  In S702, it is determined whether or not the operation mode result received in S701 is the display image dialogue mode. If the operation mode result is the display image interaction mode, the process proceeds to S703. Otherwise, the process proceeds to S705.

  In S703, the collected image storage device 109 is referred to, and the collected image information whose current display flag is true is read. When the collected image information is read, the process proceeds to S704.

  In S704, the image related word included in the collected image information read in S703 is transmitted to the robot motion generator 107 as a display image search word. When the transmission is completed, the process ends.

In S705, it is determined whether or not the operation mode result received in S701 is the interactive mode. If the operation mode result is the interactive mode, the process proceeds to S706. Otherwise, the process ends.
In S706, the collected image storage device 109 is referred to, and the collected image information whose current display flag is true is read. When the reading is finished, the process proceeds to S707.

  In S707, the image display time of the collected image information is compared with the current time, and it is determined whether or not the current time has passed a certain time from the image display time. If the predetermined time has elapsed, the process proceeds to S708. If not, the process proceeds to S713.

  In S708, the collected image storage device 109 is referenced to search for collected image information whose image recording time is within a predetermined time from the current time and whose image display time is null. If the search ends, the process proceeds to S709.

  In S709, it is determined whether there is one or more collected image information satisfying the condition of S708. If there is no acquired collected image information, the process proceeds to S713. In other cases, the process proceeds to S710.

  In S710, one is randomly selected from the collected image information acquired in S708. When the selection is completed, the process proceeds to S711.

  In S711, the image data of the collected image information selected in S710 is transmitted to the robot motion generator 107 as an image display command. When the transmission is completed, the process proceeds to S712.

  In S712, first, the information in which the current display flag of the collected image storage device 109 is true is changed to false, and an image information update command is executed to change the current display flag of the collected image information selected in S710 to true. Transmit to the storage device 109. Furthermore, an image information update command for changing the image display time of the collected image information selected in S710 to the current time is transmitted. When the transmission is completed, the process ends.

  In S713, the image related words of the collected image information read out in S706 are transmitted to the robot motion generator 107 as display image search words. When the transmission is completed, the process ends.

  As described above, the control device 1 according to the present embodiment is a control device that controls the image presentation device 2 and the robot 3 that are used by the user U. Based on the above, the operation mode of the control device 1 is switched among the recall mode, the dialogue mode, and the display image dialogue mode, and the image presentation device 2 and the robot 3 are controlled in conjunction with each other.

  Specifically, in the recollection promotion mode, the robot 3 is controlled to face the user U, and the robot U is controlled to speak a question sentence for causing the user U to start recollection. Controls the image presentation device 2 to switch and display the image searched by the search word string obtained from the utterance of the user U, controls the robot 3 to face the user U, and listens to the user U. The operation is controlled so that the robot 3 executes, and in the display image dialogue mode, the image presentation device 2 is controlled to continuously display the image searched by the search word string obtained from the utterance of the user U, Control is performed so that the robot 3 faces the image presentation apparatus 2 that displays images continuously displayed, and control is performed so that the robot 3 speaks the utterance text corresponding to the search word string.

  More specifically, when the user U's utterance is detected in the recollection promotion mode, the mode transitions to the dialogue mode, and in the dialogue mode, the user U starts to pay attention to the image presentation device. When it detects, it changes to display image dialog mode, On the other hand, when it detects that there is no utterance of user U for a fixed time or more, it changes to recollection promotion mode. When it is detected that attention to the device 2 is finished, the mode transitions to the dialogue mode. On the other hand, when it is detected that the user U has not spoken for a certain period of time, the mode changes to the recollection promotion mode.

More specifically, in the interactive mode, the first condition that the length (t _Tstop ) of the section in which the dialogue by the user U is stopped is longer than a predetermined first threshold value (t _Tnext ); A length (t _VandT ) of a section in which at least one of the face or line of sight of the user U faces the image presentation device 2 and the dialogue by the user U continues is a predetermined second threshold value (t _toV ). If the second condition of longer is satisfied, it is determined that the user U has started to pay attention to the image presentation device 2.

Further, in the display image interaction mode, the length of time (t _Vstop ) after detecting that the state where at least one of the user U's face or line of sight is facing the image presentation device 2 is completed is determined in advance. When the condition that it is longer than the threshold (t _noV ) is satisfied, it is determined that the user U has finished paying attention to the image presentation device 2.

  According to the control device 1, it is possible to reduce the preparation time required for reminiscence therapy and the implementation time by one or a combination of these techniques.

  The computer program for causing the computer to function as the control device 1 can be recorded on a computer-readable recording medium such as a semiconductor memory, a magnetic disk, an optical disk, a magneto-optical disk, a magnetic tape, and a communication network such as the Internet. And can be widely distributed.

DESCRIPTION OF SYMBOLS 1 Control apparatus 2 Image presentation apparatus 3 Robot 4 Microphone 5, 6 Camera 7 External image search apparatus 101 Speech recognizer 102 Search word extractor 103 Dialog-related image search unit 104 Display image operation unit 105 Attention object determination unit 106 Operation mode determination unit 107 robot motion generator 108 recollection start question sentence storage device 109 collected image storage device 110 attention object result storage device

Claims (7)

A control device for controlling an image presentation device and a robot used by a user,
Based on the utterance by the user and the direction of attention directed by the user, the operation mode of the control device is switched between the recall mode, the dialogue mode, and the display image dialogue mode, and the image presentation device and the robot are interlocked. A control apparatus comprising: means for controlling. When in the recollection promotion mode, the robot is controlled to face the user, and the robot is controlled to speak a question sentence for starting the recollection.
In the interactive mode, control is performed so that the image presentation device switches and displays an image searched based on a search word string obtained from the user's utterance, and the robot is controlled to face the user. Control the robot to perform the action of listening to
In the display image dialogue mode, the image presentation device is controlled to continuously display an image searched by a search word string obtained from the user's utterance, and the continuously displayed image is displayed. The control device according to claim 1, wherein the robot is controlled so that the robot faces the image presentation device, and the utterance text corresponding to the search word string is controlled to speak. In the recollection promotion mode, if the user's utterance is detected, transition to the dialogue mode,
In the interactive mode, when it is detected that the user has started to pay attention to the image presentation device, the display image interactive mode is transitioned to while the user's utterance is not detected for a certain period of time. If you do, transition to the recall mode,
In the display image dialogue mode, when it is detected that the user has finished paying attention to the image presentation device, the user makes a transition to the dialogue mode while detecting that the user's utterance has not exceeded a predetermined time. When it does, it changes to the recollection promotion mode. The control device according to claim 1 or 2 characterized by things. In the interactive mode,
The first condition that the length of the section in which the user's dialogue is stopped is longer than a predetermined first threshold, and at least one of the user's face or line of sight is suitable for the image presentation device. And when the 2nd condition that the length of the section where the dialog by the user is continuing is longer than the predetermined 2nd threshold is satisfied, the user pays attention to the image presentation device. The control device according to claim 3, wherein it is determined that the operation is started. In the display image interaction mode,
When the condition that the length of time after detecting that the state in which at least one of the user's face or line of sight is facing the image presentation device has ended is longer than a predetermined threshold is satisfied, The control device according to claim 3, wherein it is determined that the user has finished paying attention to the image presentation device. An operation method of an image presentation device used by a user and a control device for controlling a robot,
The control device switches the operation mode of the control device among a recollection promotion mode, a dialogue mode and a display image dialogue mode based on the utterance by the user and the direction of attention directed by the user, the image presentation device and the An operation method of a control device characterized by controlling the robot in conjunction with each other. The computer program for functioning a computer as a control apparatus in any one of Claims 1 thru | or 5.

Images