JP6683974B2 - Coaching support device and program - Google Patents

Description

  The present invention relates to a coaching support device and program.

  Conventionally, various techniques for assisting a caregiver who cares for a care receiver have been proposed. For example, Patent Document 1 discloses a system that notifies a caregiver of an action proposed for the patient based on a signal of a sensor that detects patient characteristics such as a body position and a temperature.

JP, 2018-8069, A

  However, although the above-described conventional system can assist the caregiver by notifying the suggested action, it is difficult to improve the skill of the caregiver himself. Therefore, it is desired to realize a mechanism to support coaching for caregivers.

  An object of the present invention is to provide a mechanism that supports coaching for caregivers.

One aspect of the present invention is
A coaching support device for supporting coaching for a caregiver who cares for a cared person,
A means for analyzing a photographed image of the caregiver caring for the cared person,
A means for setting guidance information for the caregiver based on an analysis result of the photographed image,
It is a coaching support device.

  According to the present invention, it is possible to provide a mechanism for supporting coaching for a caregiver.

It is a figure which shows an example of a structure of the coaching assistance system of this embodiment. It is a block diagram which shows schematically the structure of the coaching assistance apparatus of this embodiment. It is an explanatory view of the outline of this embodiment. It is a figure which shows the data structure of the photography data management table of this embodiment. It is a figure which shows the data structure of the guidance information reference table of this embodiment. It is a figure which shows the data structure of the guidance information management table of this embodiment. It is a flowchart of a process of the coaching assistance apparatus of FIG. It is a figure which illustrates the 1st photography video image | photographed with a wearable camera. It is a figure which illustrates the 2nd photography image picturized by a ceiling camera. It is a figure which illustrates the guidance screen for a caregiver to confirm the guidance information set.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(1) Configuration of Coaching Support System The configuration of the coaching support system of this embodiment will be described. FIG. 1 is a diagram showing an example of the configuration of the coaching support system of this embodiment.

As shown in FIG. 1, the coaching support system 1 of the present embodiment is arranged at a care site (for example, a hospital room) where a care giver 30 cares for a care receiver 32.
The coaching support system 1 includes a coaching support device 10, a fixed camera 20, a wearable camera 22, and a pin microphone 24.

  The fixed camera 20 is arranged, for example, on the ceiling of a nursing care site. The fixed camera 20 is configured to take an image of the caregiver 30 caring for the care receiver 32 from above the caregiver 30 and the care receiver 32.

  The wearable camera 22 is worn by the caregiver 30. The wearable camera 22 is configured to capture an image corresponding to the field of view of the caregiver 30. The wearable camera 22 is, for example, a glasses-type camera.

  The pin microphone 24 is worn by the caregiver 30. In the example of FIG. 1, it is attached to the chest of the caregiver 30. The pin microphone 24 is configured to record a voice at a care site.

  The coaching support device 10 may be arranged at the care site or outside the care site.

(1-1) Configuration of Coaching Support Device The configuration of the coaching support device 10 of this embodiment will be described. FIG. 2 is a configuration diagram schematically showing the configuration of the coaching assistance device of this embodiment.

The coaching support device 10 has a function of supporting coaching for the caregiver 30.
As shown in FIG. 2, the coaching support device 10 includes a processor 11, a memory 12, an input / output interface (hereinafter referred to as “input / output I / F”) 13, and a communication interface (hereinafter referred to as “communication I / F”). 14 and a storage 15.

The processor 11 is configured to read the program stored in the storage 15 into the memory 12 and execute the instructions included in the program.
The processor 11 is, for example, at least one of a CPU (Central Processing Unit) and a GPU (Graphical Processing Unit).

The memory 12 is configured to temporarily store data and programs.
The memory 12 is, for example, a DRAM (Dynamic Random Access Memory).

The input / output I / F 13 is configured to exchange information with the user.
The input / output I / F 13 includes, for example, at least one of the following.
Information input device (keyboard, mouse and touch panel as an example)
・ Voice input device (microphone as an example)
Image input device (camera as an example)
Image output device (display, for example)
・ Audio output device (speaker as an example)

The communication I / F 14 is configured to control wired or wireless communication.
The communication I / F 14 is, for example, at least one of the following.
Hardware (network adapter as an example)
・ Various communication software

The storage 15 is configured to store various information (for example, data and programs). For example, as shown in FIG. 1, a shooting data management table 151 (FIG. 4) for managing information on shooting data of how the caregiver 30 cares for the care receiver 32, and guidance information based on the analysis result of the shooting data. A guidance information reference table 152 (FIG. 5) that is referred to for setting the guidance information, and a guidance information management table 153 (FIG. 6) for managing the guidance information that is set based on the analysis result of the photographing data. .
The storage 15 includes, for example, at least one of the following.
・ Magnetic disk ・ Flash memory

(2) Outline of Embodiment An outline of this embodiment will be described. FIG. 3 is an explanatory diagram of the outline of the present embodiment.

  As shown in FIG. 3, the image of the caregiver 30 caring for the care receiver 32 is captured by the image input device. As a result, a photographed image of nursing care can be obtained.

The coaching support device 10 analyzes the captured video.
The coaching support device 10 sets guidance information for the caregiver 30 based on the analysis result of the captured video and stores it in the storage 15. The guidance information includes information on advice regarding caregiving.
That is, the storage 15 automatically stores the guidance information based on the analysis result of the captured image of the caregiver 30 caring for the care receiver 32.

  According to the present embodiment, coaching for the caregiver 30 based on the actual state of care can be easily realized.

(3) Data Structure The data structure of this embodiment will be described. Each data table is stored in the storage 15.

(3-1) Shooting Data Management Table The data structure of the shooting data management table 151 of this embodiment will be described. FIG. 4 is a diagram showing the data structure of the shooting data management table of this embodiment.

  The shooting data management table 151 of FIG. 4 includes a “shooting ID” field, a “wearable camera shooting video file” field, a “fixed camera shooting video file” field, and a “pin microphone recording audio file” field. Each field is associated with each other.

  In the “shooting ID” field, a shooting ID for identifying individual shooting is stored.

  The file of the first captured image 40 captured by the wearable camera 22 is stored in the “wearable camera captured image file” field.

  The file of the second captured image 50 captured by the fixed camera 20 is stored in the “fixed camera captured image file” field.

  The “pin microphone recorded voice file” field stores a file of recorded voice recorded by the pin microphone 24.

(3-2) Guidance Information Reference Table The data structure of the guidance information reference table 152 of this embodiment will be described. FIG. 5 is a diagram showing a data structure of the guidance information reference table of this embodiment.

  The shooting data management table 151 of FIG. 5 includes an “analysis result” field and a “teaching information” field. Each field is associated with each other.

  The “analysis result” field stores information about the result of image analysis of the first captured video 40 and the second captured video 50.

  The “guidance information” field stores guidance information corresponding to the result of image analysis of the first captured video 40 and the second captured video 50.

For example, the analysis result “no eye contact” is associated with instruction information “Let's make eye contact during conversation.”
The analysis result “with eye contact and distance between heads exceeding 20 cm” is associated with instruction information “with eye contact within 20 cm”.
The instructional information of "excellent!" Is associated with the analysis result "there is eye contact and the head distance is within 20 cm" of the captured image.

The analysis result “no eye contact” corresponds to a case where no eye contact between the caregiver 30 and the care receiver 32 is detected in the first captured image 40 when the caregiver utterance is detected.
The analysis result “with eye contact” corresponds to the case where eye contact is detected in the first captured image 40.
The analysis result “within 20 cm” corresponds to the case where the head-to-head distance determined from the second captured image 50 is within 20 cm.

(3-3) Guidance Information Management Table The data structure of the guidance information management table of this embodiment will be described. FIG. 6 is a diagram showing the data structure of the guidance information management table of this embodiment.

  The guidance information management table 153 of FIG. 6 includes a “shooting ID” field, a “playback timing” field, and a “teaching information” field. Each field is associated with each other.

  A shooting ID is stored in the “shooting ID” field.

  In the “playback timing” field, information (eg, playback timing (as an example, hours, minutes, and seconds ), And at least one of the frame numbers assigned in time series to each image frame).

  Guidance information is stored in the “guidance information” field.

(4) Process of Coaching Support Device The process of the coaching support device 10 of this embodiment will be described. FIG. 7 is a flowchart of the process of the coaching support device of FIG. FIG. 8 is a diagram illustrating a first captured image captured by the wearable camera. FIG. 9 is a diagram illustrating a second captured image captured by the ceiling camera. FIG. 10: is a figure which illustrates the guidance screen for a caregiver to confirm the guidance information set.

  The process of FIG. 7 is executed in response to an instruction from an operator (for example, an administrator) of the coaching support device 10 after the shooting at the care site is completed and the shot video and the recorded sound are prepared.

As shown in FIG. 7, the coaching support device 10 executes the analysis (ST100) of the recorded voice at the shooting site.
Specifically, the processor 11 stores the recording voice file acquired by the pin microphone 24 in the photographing data management table 151 (FIG. 4) in association with the photographing ID and the pin microphone recording voice file.
The processor 11 analyzes a recorded voice acquired by the pin microphone 24 by using a learned model (hereinafter referred to as a “voice analysis model”) for analyzing a recorded voice, so that a caregiver's utterance (for example, care Utterances including dialogue when starting a specific work in).
The learned model is constructed by machine learning using a large amount of recorded voices at a nursing care site as training data. The input of the recorded voice analysis model is the recorded voice. The output of the recorded voice analysis model is information for detecting a caregiver utterance.

The processor 11 specifies the reproduction timing of the recorded voice when the caregiver utterance is detected.
The analysis of the recorded voice is repeated until a predetermined utterance by the caregiver 30 (hereinafter referred to as “caregiver utterance”) is detected (NO in ST110).

  When the caregiver utterance is detected (YES in ST110), coaching assistance device 10 executes an analysis of the captured image (ST120).

The processor 11 stores the file of the first captured image 40 acquired by the wearable camera 22 in the captured data management table 151 (FIG. 4). The file of the first captured image 40 is associated with the captured ID associated with the sound recording in step ST100.
As shown in FIG. 8, the first captured image 40 typically includes the face of the cared person 32.

  The processor 11 detects an eye contact from each image frame of the first captured image 40 by using a learned model (hereinafter referred to as “first image analysis model”) for analyzing the first captured image 40. The first image analysis model is constructed by machine learning using a large number of first captured images 40 of a nursing care site as training data. The input of the first image analysis model is the first captured image 40. The output of the first image analysis model is information for detecting the state of eye contact.

The first image analysis model includes, for example, an algorithm (for example, ResNet (Residual Network)) for estimating the face direction of the care receiver 32.
In this case, the processor 11 detects any of the following.
The face of the caregiver 30 and the face of the care receiver 32 face each other (that is, eye contact is established).
The caregiver 30 face is not facing the care recipient 32 face.
-The face of the cared person 32 does not face the face of the cared person 30.
-The face of the caregiver 30 and the face of the care receiver 32 are not facing each other.

  The processor 11 identifies the reproduction timing of the first captured image 40 when the eye contact occurs.

  The processor 11 stores the file of the second captured image 50 acquired by the fixed camera 20 in the captured data management table 151 (FIG. 4). The file of the second captured image 50 is associated with the captured ID associated with the sound recording in step ST100.

As shown in FIG. 9, the processor 11 uses the learned model for analyzing the second captured image 50 (hereinafter referred to as “second image analysis model”) to form each image frame forming the second captured image 50. From the above, the parietal region 52 of the caregiver 30 (for example, a rectangular region surrounding the pixel indicating the parietal feature amount) and the face region 54 of the care receiver 32 (for example, the pixel indicating the facial feature amount). And a rectangular area that surrounds).
The processor 11 determines the distance between the head of the care giver 30 and the head of the care receiver 32 (hereinafter referred to as “inter-head distance”) based on the positional relationship between the region 52 of the crown and the face region 54. ) Is determined for each reproduction timing.

The second image analysis model is constructed by machine learning using a large number of second captured images 50 of a nursing care site as training data.
The second image analysis model includes, for example, the following algorithm.
An algorithm for classifying the caregiver 30 and the cared person 32 (for example, the nearest neighbor method)
-Algorithm for detecting top of head and face (for example, SSD (Single Shot MultiboxDetector))
-Algorithm for determining head-to-head distance (for example, linear regression algorithm)

  After step ST120, the coaching assistance device 10 sets instruction information (ST130).

The processor 11 refers to the guidance information reference table 152 (FIG. 5) to determine the guidance information corresponding to the analysis results of steps ST100 and ST120.
The processor 11 adds a new record to the guidance information management table 153.
The following information is stored in each field of the new record.
A new shooting ID is stored in the "shooting ID" field.
The "playback timing" field stores the playback timing of the image frame referred to when determining the guidance information.
The determined guidance information is stored in the "teaching information" field.

  As shown in FIG. 7, the processing (ST100 to ST120) of analyzing a captured image and setting teaching information in response to the detection of a caregiver utterance in such recorded voice is repeated until the recorded voice ends. (NO in ST130).

As shown in FIG. 10, the guidance screen 60 includes a first reproduction area 62, a second reproduction area 64, and a guidance information display area 66.
The first reproduction area 62 is an area for reproducing the first captured image 40.
The second reproduction area 64 is an area for reproducing the second captured image 50.
The guidance information display area 66 is an area for displaying guidance information.

In the first reproduction area 62 and the second reproduction area 64, the first captured video 40 stored in the “wearable camera captured video file” field of the captured data management table 151 (FIG. 4) and the “fixed camera captured video file”. The second captured image 50 stored in the field is reproduced in synchronization with the reproduction timing.
The recorded sound stored in the “pin microphone recorded sound file” field is reproduced in synchronization with the reproduction timing of the first captured video 40 and the second captured video 50.
In the guidance information display area 66, the guidance information stored in the guidance information management table 153 (FIG. 6) is displayed according to the reproduction timing associated with the guidance information. For example, the guidance information displayed in the guidance information display area 66 disappears when a predetermined time has elapsed from the associated reproduction timing.
The caregiver 30 can confirm the guidance information set by the coaching support device 10 while watching the caregiver 32 to care for himself / herself via the guidance screen 60.

  Since the coaching support device 10 according to the present embodiment described above sets the guidance information for the caregiver based on the analysis result of the captured video and the recorded sound of the caregiver caring for the caregiver, Coaching for a caregiver based on the situation can be easily realized.

(5) Summary This embodiment will be summarized.

The first aspect of the present embodiment is
A coaching support device 10 for supporting coaching for a caregiver 30 who cares for a cared person 32,
A means (for example, the processor 11 that executes step ST120) that analyzes a captured video (for example, at least one of the first captured video 40 and the second captured video 50) of how the caregiver 30 cares for the care receiver 32. Prepare,
A means for setting guidance information for the caregiver 30 based on the analysis result of the photographed image (for example, the processor 11 executing step ST130),
The coaching support device 10.

  According to the first aspect, the guidance information for the caregiver 30 is automatically set based on the analysis result of the captured image of the caregiver 30 caring for the care receiver 32. As a result, coaching for the caregiver 30 based on the actual state of care can be easily realized.

The second aspect of this embodiment is
The analyzing unit detects the eye contact between the care receiver 32 and the care giver 30 based on the first captured image 40 corresponding to the field of view of the care giver 30, and the care giver 30 cares for the care receiver 32. The head-to-head distance between the head of the care-receiver 32 and the head of the care-giver 30 is determined based on the second photographed image 50 obtained by photographing the situation from above,
The setting means sets the guidance information based on the head-to-head distance when eye contact is detected,
The coaching support device 10.

  According to the second aspect, it becomes possible to provide guidance regarding the head-to-head distance when eye contact is detected.

The third aspect of the present embodiment is
The means of analysis is
Detecting a face area and a head area included in the second captured image 50,
The face area is determined to be the head of the care-receiver 32,
The area of the crown is determined to be the head of the caregiver 30,
The coaching support device 10.

  According to the third aspect, the heads of the cared person 32 and the carer 30 can be easily detected.

The fourth aspect of the present embodiment is
In addition to the captured video, the analyzing means analyzes the recorded sound of the caregiver 30 caring for the care receiver 32,
The setting means sets the teaching information based on the analysis result of the recorded sound in addition to the analysis result of the captured image.
It is a coaching support device.

  According to the fourth aspect, it is possible to provide guidance based on the utterance of the caregiver 30.

The fifth aspect of the present embodiment is
The analyzing unit detects a predetermined utterance of the caregiver 30 based on the recorded voice, and detects between the caregiver 32 and the caregiver 30 based on the first captured image 40 corresponding to the field of view of the caregiver 30. Detect eye contact,
The setting means sets the instruction information based on the detection result of the eye contact when the predetermined utterance of the caregiver 30 is detected,
It is a coaching support device.

  According to the fifth aspect, it is possible to provide guidance regarding eye contact when a predetermined utterance of the caregiver 30 is detected.

The sixth aspect of the present embodiment is
The means for setting is to set the instruction information in association with the reproduction timing in the captured video,
The coaching support device 10.

  According to the sixth aspect, the caregiver 30 can confirm the guidance information in the related scene while reproducing the captured video.

A seventh aspect of the present embodiment is
A coaching support program for causing a computer (for example, the processor 11) to function as each of the above means.

(6) Other Embodiments Other embodiments will be described.

In the above-described embodiment, when a predetermined utterance by the caregiver is detected in the recorded voice, the guidance information is set based on the presence / absence of eye contact and the head-to-head distance. Is not limited to this.
Instead of such a rule-based mechanism, setting of the guidance information itself can be realized by using a learned model. In this case, the learned model is generated by, for example, machine learning using a large amount of shooting data (shooting video and / or recorded voice) in which appropriate guidance information is manually set as training data.

In the above-described embodiment, the fixed camera 20 provided on the ceiling of the living room (hospital room, etc.) and the second captured image 50 captured by the wearable camera 22 worn by the caregiver are used. The installation mode of is not limited to these.
In the present embodiment, for example, when only one of the fixed camera 20 and the wearable camera 22 is used, or another camera installed to have another field of view (that is, the fixed camera 20 and the wearable camera) is used. It is also applicable when using a captured image acquired by a camera other than the camera 22).

In the above-described embodiment, after the imaging of the care site is completed and the imaging data is prepared, the guidance information is set based on the analysis result of the imaging data, but the present embodiment is not limited to this.
The present embodiment is also applicable to a case where imaging data of a nursing care site is analyzed in real time and guidance information is set based on the analysis result in real time.
In this case, as shown in FIG. 1, it is preferable to arrange the coaching support device 10 at the care site. Thereby, the caregiver 30 can care for the care receiver 32 while referring to the guidance information set by the coaching support device 10.

  Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited to the above embodiments. Further, the above-described embodiment can be variously modified and changed without departing from the gist of the present invention. Further, the above-described embodiments and modified examples can be combined.

1: Coaching support system 10: Coaching support device 11: Processor 12: Memory 13: Input / output I / F
14: Communication I / F
15: Storage 20: Fixed camera 22: Wearable camera 24: Pin microphone

Claims (6)

A coaching support device for supporting coaching for a caregiver who cares for a cared person,
A means for analyzing a photographed image of the caregiver caring for the cared person,
A means for setting guidance information for the caregiver based on an analysis result of the photographed image,
The means for analyzing is
Detecting eye contact between the cared person and the caregiver based on a first captured image corresponding to the field of view of the caregiver,
The head-to-head distance between the head of the cared person and the head of the cared person is determined based on a second video image of the cared person taking care of the cared person from above,
The setting means sets the teaching information based on the head-to-head distance when the eye contact is detected,
Coaching support device. The means for analyzing is
Detecting a face area and a crown area included in the second captured image,
The area of the face is determined to be the care recipient's head,
The area of the crown is determined to be the caregiver's head,
The coaching support device according to claim 1 . Wherein the means for analysis, in addition to the captured image, to analyze the recording sound voice of a state in which the caregiver caring for the person cared for,
It said means for setting, in addition to the analysis result of the captured image, to set the guidance information based on the analysis results of the recorded speech,
The coaching support device according to claim 1 . The analyzing unit detects a predetermined utterance of the caregiver based on the recorded voice, and detects between the cared person and the caregiver based on a first captured image corresponding to the field of view of the caregiver. Eye contact of
The setting means sets the guidance information based on a detection result of the eye contact when a predetermined utterance of the caregiver is detected,
The coaching support device according to claim 3 . The setting means sets the instruction information in association with the reproduction timing in the captured video.
Any coaching support device according to claim 1-4. The computer program for functioning as one of the means of the claims 1-5.

Images