WO2020179128A1

WO2020179128A1 - Learning assist system, learning assist device, and program

Info

Publication number: WO2020179128A1
Application number: PCT/JP2019/042527
Authority: WO
Inventors: 力也田尻; 真理子味八木; 昌樹高野; さゆり橋爪; 純一桑田
Original assignee: 株式会社日立製作所
Priority date: 2019-03-06
Filing date: 2019-10-30
Publication date: 2020-09-10
Also published as: JP2020144233A; JP7116696B2

Abstract

The purpose of the present invention is to provide a learning assist system with which it is possible for a learner to efficiently learn technology.　The learning assist system comprises: a display unit 19 fitted to a learner 102; an imaging unit 15 fitted to the learner 102, for imaging the visual field video of the learner 102; a storage unit for storing an example moving-image that is the moving-image of work motion of an instructor that serves as an example of motion of the learner 102; and an arithmetic unit 16. The arithmetic unit 16 causes the example moving-image to be displayed by the display unit 19 in superimposition on the visual field video imaged by the imaging unit 15, and causes the display content of the example moving-image to be dynamically changed in accordance with the feature of work motion of the learner 102 included in the visual field video.

Description

Learning support system, learning support device and program

The present invention relates to a learning support system, a learning support device, and a program used when learning a technique.

Traditionally, the transmission of technology in various technical fields has been carried out by actual training and work by technical learners under the guidance of instructors.

❖ In order to convey advanced skills to learners, it may be necessary to convey tips that are difficult to express in words, but it is not easy for teachers to convey those tips to learners efficiently. In recent years, in order to efficiently convey the technique acquired by the instructor to the learner, there has been proposed an apparatus or method for converting the technique of the instructor into data and storing it, and displaying the stored technical data to the learner (for example, Patent Document 1).

Patent Document 1 discloses a method of extracting and saving the movement of the instructor such as the position, movement amount, and moving speed of the instructor's hand under various environments. With this method, the instructor is asked to perform the same work each time while changing the environment, and the instructor's movements are measured each time by motion capture, etc., and the changes in the environment and movements are stored in association with each other. ..

In addition, a technology has been proposed in which the measured motion information of the instructor is imaged, and the image is superimposed on the actual work scene by augmented reality (AR) technology using a glasses-type wearable device (smart glasses) or the like. (For example, Patent Document 2).

For example, in the work support device of Patent Document 2, the information processing unit compares the surface shape of the object to be worked measured by the imaging unit with the surface shape of the object stored in advance, and determines each of the work target objects. After calculating the amount to be thermally deformed at the point, calculate the heating position and the heating amount to obtain the deformation amount, and superimpose the image showing the heating position on the image of the work target object on the head mounted display. it's shown.

JP-A-2003-281287 JP, 2009-69954, A

The conventional learning support system that displays the technology of the instructor, which has been converted into data, to the learner, such as Patent Document 2, is suitable for both learners having different proficiency levels and learners having different characteristics such as the dominant hand. To display the technology. In such a system, learners with different proficiency levels and characteristics may not be able to learn efficiently.

The present invention aims to provide a learning support system that enables a learner to efficiently learn a technique.

In order to solve the above problems, the present invention provides a display unit mounted on a learner, an imaging unit mounted on the learner to capture a visual field image of the learner, and an instructor serving as a model for the learner's operation. A storage unit that stores a model moving image, which is a moving image of a work operation, and a calculation unit are provided. The calculation unit superimposes the model video on the visual field image captured by the imaging unit and displays it on the display unit, and dynamically displays the model video according to the characteristics of the learner's work operation included in the visual field image. Change the content.

According to the present invention, since the displayed model video dynamically changes according to the learner's work operation, the learner can perform training to learn the technique at a pace suitable for each. This allows the learner to efficiently acquire the technique.

Explanatory drawing which shows the structure of the learning support system of 1st Embodiment. The block diagram which shows the structure of the control system 101 of the learning support system of 1st Embodiment. The flowchart which shows the flow of the learning training of 1st Embodiment. The example of a screen in case the difference of a position is displayed on the display part 19. The example of a screen in case the difference of a position is displayed on the display part 19. A display example of the position difference. A display example of the position difference. A display example of the position difference. The display example which shows the difference of the operation element of a learner and an instructor. A display example showing a difference in pressure between a learner and a teacher. FIG. 3 is a block diagram showing details of the configuration of the glasses-type device 201. Example of display change of operation element with reduced difference. The block diagram which shows the structural example of the control system 101 of a learning support system. The flowchart which shows the flow of the whole learning support system of 1st Embodiment. The figure which shows the input / output sequence of the control system 101, the glasses-type device 201, and the sensing device 204. (A), (b) UI example displayed on the display when inputting basic information and training type. Explanatory drawing of difference judgment, proficiency level evaluation, and reflection. An example of a log stored in the information storage unit 17. A flowchart showing the flow in the level judgment mode. Explanatory drawing which shows the structure of the learning support system of 2nd Embodiment. FIG. 3 is a block diagram showing a configuration of a glasses-type device worn by a leader. The block diagram which shows the structure of the control system 101 of the learning support system of 2nd Embodiment. The flowchart which shows the flow of operation|movement of the learning support system of 2nd Embodiment. The block diagram which shows the structure of the learning support system of 3rd Embodiment. The flowchart which shows the flow of operation|movement of the learning assistance system of 3rd Embodiment. The flowchart which shows the operation flow of the learning support system of the modification of 3rd Embodiment. Explanatory drawing which shows the structure of the learning support system of 4th Embodiment. The display example of the display part 19 of the learning support system of 4th Embodiment.

The learning support system according to the embodiment of the present invention will be described below.

<<Learning Support System of First Embodiment>>
As the learning support system of the first embodiment, an example of a system for eyeglass-type devices in which a learner learns how to operate a probe in an ultrasonic imaging apparatus will be described. As shown in FIG. 1, this learning support system includes glasses-type device 201 worn by learner 102 and glasses connected to glasses-type device 201 via a network (a wireless LAN (Local Area Network) as an example) 30. And a control system 101 for controlling the function of the molding device 201. The glasses-type device 201 includes a display unit 19 and an imaging unit (camera) 15 that captures a field image of the learner 102.

As shown in FIG. 2, the control system 101 includes a computing unit 16 that performs various computations in this learning support system based on information received from the glasses-type device 201, an information storage unit 17 that stores various information, Equipped with. The arithmetic unit 16 superimposes a model video (see, for example, FIG. 4), which is a video of the work motion of the instructor, which serves as a model of the work motion of the learner 102, on the visual field image captured by the camera 15, and displays it on the display unit 19. In addition to the display, the display content of the model moving image is dynamically changed according to the characteristics of the work operation of the learner 102 included in the visual field image. As will be described in detail later, the information storage unit 17 stores at least model data including a model video and position information of the instructor's hand.

The calculation unit 16 executes various calculations by executing the learning support program stored in advance in the information storage unit 17. Specifically, the calculation unit 16 superimposes a model moving image on the image capture unit 161 that captures the field of view image of the learner 102 from the camera 15 and the field of view image of the learner 102, and outputs the superimposed processed image to the display unit 19. The superimposition display unit 162, the difference calculation unit 163 that calculates the difference between the work motion of the instructor included in the model video and the work motion of the learner 102 included in the visual field image, and the difference calculated by the difference calculation unit 163. It is provided with a difference determination unit 16A for determining whether or not is equal to or greater than a predetermined threshold, and a display adjustment unit 16C for changing the display content of the model moving image displayed based on the difference calculated by the difference calculation unit 163.

The flow of the training operation by the learning support system of the first embodiment will be described with reference to FIG. In this flow, the work operation when the learner 102 carries out the training while holding the probe 202 in his/her hand will be described.

The flow training operation of FIG. 3 is controlled and executed by the control system 101. First, the camera 15 captures the visual field image of the learner 102. The image capturing unit 161 captures the learner's view image captured by the camera 15 via the network 30 (step S41). Next, the superimposition display unit 162 reads out the model image stored in the information storage unit 17, and the images of the learner's hands and the probe 202 included in the learner's visual field image show the image of the instructor included in the model video. The positions are adjusted such that the predetermined reference points of the hand and the probe overlap, and the model moving image with the adjusted positions is displayed on the display unit 19 (step S42). Next, the difference calculation unit 163 calculates the difference between the position of the probe 202 held by the learner and the position of the probe held by the instructor, as shown in FIG. Specifically, first, the difference calculation unit 163 calculates the position of the center 202b of the surface of the learner's probe 202 that comes into contact with the surface of the inspection target from the learner's visual field image. Further, the difference calculation unit 163 reads the position of the center 202c of the surface of the probe held by the instructor that touches the inspection target from the model data stored in the information storage unit 17. Then, the difference calculation unit 163 calculates the difference between the position of the center 202c of the probe held by the instructor and the position of the center 202b of the probe 202 held by the learner (step S43).

Next, the difference determination unit 16A determines whether or not the difference obtained by the difference calculation unit 163 in step S43 is equal to or greater than a predetermined threshold value, and if the difference is equal to or greater than the threshold value, the result (difference determination result) is displayed. The data is output to the adjusting unit 16C (step S44). The display adjustment unit 16C performs image processing to shift the position of the probe center 202b of the model video to the position of the center 202c of the probe 202 of the learner, and processes the processed model video via the network 30 to the glasses-type device 201. Output to. The model moving image received by the glasses-type device 201 is displayed on the display unit 19 (step S45).

This allows the learner to see an image in which the positions of the instructor and the probe of the instructor overlap, so that the orientation and angle of the hand and probe can be adjusted to the orientation and angle of the instructor's hand and probe. , You can efficiently learn how to move your hands and probes.

When the model video processed in step S45 is displayed to the learner, the flow returns to step S41. The learning support system repeats the above flow until the learner's training is completed. If the difference calculated by the difference calculation unit 163 in step S44 is less than the threshold value, the flow returns to step S41.

The display adjustment unit 16C aligns the hand position of the model video with the hand position of the learner's visual field image with the center 202b of the learner's probe 202 and the center 202c of the instructor's probe as reference points. However, other reference points may be used. For example, the display adjustment unit 16C adjusts the display position of the model video by detecting the position of a part of the body of the learner, such as aligning the position of the fingertip of the learner 102 with the position of the fingertip of the instructor of the model video. May be.

[Display to inform learner of position difference]
Hereinafter, another display adjustment example performed by the display adjustment unit 16C when the difference between the positions of the learner and the teacher calculated by the difference determination unit 16A is equal to or larger than the threshold will be described.

The display adjustment unit 16C can display the difference in position between the center 202b of the probe 202 held by the learner and the center 202c of the probe held by the instructor by the arrow 102d as shown in FIG. The learner can recognize how much his or her movement deviates from the model by looking at the size and inclination of the arrow 102d. Further, the learner can easily bring the position of the probe 202 held by the learner closer to the position of the probe held by the instructor while looking at the arrow 102d.

Also, the position of the instructor's elbow may be included in the model video, and the position information of the elbow may be stored in the information storage unit 17. In this case, the difference calculation unit 163 can detect the position of the learner's elbow from the view image of the learner and calculate the difference between the position of the learner's elbow and the position of the instructor's elbow. When the difference between the position of the learner's elbow and the position of the instructor's elbow is equal to or greater than the threshold value, the display adjustment unit 16C causes the learner to select, for example, “lift the elbow more!” as shown in FIG. It is possible to specifically display in sentences whether or not to move the body. The display adjustment unit 16C may display a hint by displaying a sign such as the arrow 19b on the body part to be moved.

Further, the information storage unit 17 may be configured to store a skeleton diagram 19c showing the positions of the bones and joints of the instructor's hand 102b. In this case, as shown in FIG. 6, the display adjustment unit 16C can display the skeleton diagram 19c in an overlapping manner on the image of the instructor's hand 102b. By displaying the skeleton diagram 19c, the learner can easily correct the position of his or her hand while observing the position of the joint of the instructor's hand 102b displayed.

Further, as shown in FIG. 7, the information storage unit 17 has the shape of a hand or arm to be worked, and a color distribution diagram showing the magnitude of the force to be applied to each part of the hand at each time point of the work in color. 19d may be stored. In this case, the difference calculation unit 163 detects the position and shape of the learner's hand and arm from the learner's visual field image, and the display adjustment unit 16C displays the color distribution map 19d on the learner's hand 102c in an overlapping manner. You can For example, such as thermography, warm colors are superimposed on the parts of the body where the learner should apply a large amount of force to the muscles, and cool colors are superimposed on the parts of the body where the learner should apply a small amount of force. , It is possible to let the learner recognize the level of effort.

Further, as shown in FIG. 8, the display adjusting unit 16C detects a region where the positions of the probe 202 of the instructor and the probe 202 of the learner are overlapped with each other, and the overlapping region is white. The probe of the instructor and the probe 202 can be displayed in different colors for the region where the displacement occurs. In this way, the region where the position of the learner and the instructor are displaced can be displayed in different colors between the learner and the instructor, so that the difference between the positions of the learner and the instructor is emphasized. .. Therefore, the learner can easily recognize the direction in which his/her hand should be moved.

Further, when the position of the probe 202 held by the learner deviates from the position of the probe held by the instructor, the display adjustment unit 16C may be configured to rewind and reproduce the model video until the timing when the position shift starts. At this time, the model moving image to be played again may be divided into short time segments and the model moving image may be repeatedly reproduced for each segment. As a result, the learner can practice the work motions he/she is not good at many times, so that the learning effect can be enhanced.

Note that the glasses-type device 201 may be equipped with a vibration sensor. In that case, when the position of the probe 202 held by the learner and the position of the probe held by the instructor deviate from each other, the vibration sensor mounted on the glasses-type device 201 gives the learner force feedback such as vibration and impact. May be.

By changing the display contents as described above, it is possible to notify the learner of the positional deviation, so that the learner can easily grasp the correct probe or hand position and correct the positional deviation.

[Display adjustment type]
<Operating elements other than position>
In the above description, an example in which the difference calculation unit 163 calculates the difference (motion element) between the position of the learner and the teacher's probe or hand is explained, but the difference calculation unit 163 performs the work between the learner and the teacher. As the motion element for calculating the motion difference, one or more motion elements such as “speed, pressure, motion locus, gazing point, sound” may be detected in addition to “position”. These motion elements of the instructor are included in the model data and stored in the information storage unit 17. A method of calculating the difference between the learner and the instructor in each motion element and a method of displaying the difference will be described with reference to FIG.

The difference calculation unit 163 calculates the speed at which the learner moves the hand from the image captured by the speed camera 15, and the calculated speed of the learner's hand and the hand of the instructor stored in the information storage unit 17. The difference from the moving speed of is calculated from the difference in the position of the hands of the learner and the instructor. When the difference determination unit 16A determines that the difference between the learner's speed and the instructor's speed is greater than or equal to a predetermined value, the display adjustment unit 16C adjusts the reproduction speed of the model video to match the learner's speed. .. For example, when the work operation of the learner 102 is slower than the predetermined value by more than the work operation of the instructor, the display adjustment unit 16C slows down the reproduction speed of the model moving image so as to match the speed of the learner 102. The speed of the learner and the speed of the instructor are displayed by the difference in the color temperature of the displayed image, or by the barometer or numerical value, so that the learner can recognize the difference between his speed and the speed of the instructor. You may do so.

-Pressure Information about the pressure when the instructor presses the probe against the inspection target (hereinafter referred to as the instructor's pressure) is stored in the information storage unit 17 in advance. The superposition display unit 162 reads out the pressure information of the instructor from the information storage unit 17, and a circle 103b2 indicating the magnitude of the pressure in terms of the diameter is superposed and displayed on the visual field image of the learner as shown in FIG. .. Further, the difference calculation unit 163 calculates the pressure with which the learner presses the probe 202 on the inspection target from the change in the color of the hand, fingertip, or nail when the learner presses the probe 202 on the inspection target. When the difference determination unit 16A determines that the difference between the pressure of the learner's probe 202 and the pressure of the instructor is equal to or greater than a predetermined value, the display adjustment unit 16C determines whether the pressure of the instructor and the pressure of the instructor. The difference is displayed by, for example, the size of the circle and the difference in the color of the circle. The difference between the pressure of the instructor and the pressure of the learner is emphasized by filling the area surrounded by the circle 103b1 indicating the pressure by the learner 102 and the circle 103b2 indicating the pressure by the instructor as shown in FIG. Can be displayed. By moving the probe so as to reduce the filled area of this circle, the learner can be trained to press the test object with a pressure close to the pressure of the instructor.

-Operation locus Information on the path (operation locus) for the instructor to move the probe is stored in advance in the information storage unit 17. The superimposition display unit 162 reads out the information on the motion locus from the information storage unit 17, and superimposes and displays the flow line indicating the motion locus on the field image of the learner. At that time, as shown in FIG. 9, the color of the motion locus of the instructor is a past locus (past motion locus) 103c or a future locus (future motion locus) relative to the current position of the learner's probe 202. Different colors may be displayed depending on 103d. In addition, the difference calculation unit 163 calculates the movement locus of the probe 202 of the learner from the visual field image of the learner captured by the camera 15, and calculates the movement locus of the probe 202 of the learner and the movement locus of the probe of the instructor. Find the difference. When the difference determination unit 16A determines that the difference is equal to or larger than the predetermined value, the display adjustment unit 16C sets the motion locus of the learner's probe 202 to the

motion loci

103c and 103d of the instructor's probe that are already displayed. The learner may be made to recognize the difference between the motion trajectory of the instructor and the motion trajectory of the learner by superimposing and displaying the flow line shown.

-Gaze point Information of points (gaze point) that the instructor is paying attention to during work is stored in advance in the information storage unit 17. The superimposed display unit 162 reads the gazing point information from the information storage unit 17, and superimposes and displays the point 103a indicating the gazing point on the learner's visual field image. The glasses-type device 201 includes a line-of-sight sensor 20 as shown in FIG. 11, which will be described later. The line-of-sight sensor 20 detects the gazing point of the learner. The difference calculation unit 163 calculates the difference between the gazing point of the learner and the gazing point 103a of the instructor stored in the information storage unit 17. When the difference determination unit 16A determines that the difference between the learner and the gaze point 103a of the instructor is equal to or greater than the threshold value, the display adjustment unit 16C highlights the gaze point 103a of the instructor by blinking or the like. In addition to the gaze point 103a of the instructor, the display adjustment section 16C may display the gaze point of the learner detected by the line-of-sight sensor 20 on the display section 19 using a cross symbol or the like.

-Sound The glasses-type device 201 may be provided with a recording device that records a work sound. In this case, the sound recording device acquires the work sound of the learner by the sound recording device and records the work sound of the learner recorded by the difference calculation unit 163 and the work sound of the model moving image stored in the information storage unit 17. Calculate the difference. When the difference determination unit 16A determines that the difference between the work sounds of the learner and the instructor is equal to or greater than the threshold value, the display adjustment unit 16C displays that the sounds are deviated due to the waveform, the color, the character, and the like.

Note that for each action element, whichever is statistically likely to occur and the cause thereof may be stored in advance in the information storage unit 17. In that case, the difference calculation unit 163 may also be configured to obtain the cause of the difference between the work operation of the learner and the work operation of the instructor for the operation element in which the work operation of the learner and the work operation of the instructor are misaligned. Good. Further, the display adjustment unit 16C may cause the display unit 19 to display which of the action elements is statistically likely to occur and the cause thereof.

[Glasses device 201]
Here, the configuration of the glasses-type device 201 will be specifically described. As shown in FIG. 11, the glasses-type device 201 includes glasses, a camera 15, a display unit 19, a wireless communication device 13, a CPU 12, a memory 18, and a line-of-sight sensor 20 mounted on a part of the glasses. Will be done.

The camera 15 captures an image in a predetermined direction such as the direction of the user's line of sight, and measures the user's work motion.

The display unit 19 projects the model moving image adjusted by the display adjustment unit 16C under the control of the calculation unit 16 and the CPU 12 into the visual field of the user.

The display unit 19 may have a structure that projects an image on the retina of the user. A wearable display may be used instead of the glasses. In this case, the visual field image of the user captured by the camera 15 and the model moving image adjusted by the display adjustment unit 16C are superimposed and displayed on the wearable display.

The wireless communication device 13 communicates between the CPU 12 and the control system 101.

The CPU 12 controls the operation of each part of the glasses-type device 201 based on the information received from the calculation unit 16 of the control system 101. Specifically, the wireless communication device 13 is instructed to transmit the video data captured by the camera 15 to the video capture unit 161 and the information received from the superimposed display unit 162 and the display adjustment unit 16C is displayed in the display unit 19. Or send it to and display it on the video.

The image captured by the camera 15 and the calculation result of the CPU 12 are stored in the memory 18 as needed. Further, the memory 18 can also store the image to be displayed on the display unit 19.

The line-of-sight sensor 20 is a sensor that detects where the user's gazing point is. The line-of-sight sensor 20 transmits the detected gazing point data to the CPU 12.

[Training mode]
In the above description, the display adjustment unit 16C is described as adjusting the position of the probe held by the instructor to the position of the probe 202 held by the learner at any time. The information storage unit 17 includes a beginner mode for basic training that enables the trainer's hand to be aligned with the learner's pace at any time, and the trainer's hand only when the difference exceeds a predetermined value. A plurality of training modes (Table 1) may be stored, such as a normal mode in which the position of is corrected. Here, the training mode stored in the information storage unit 17 will be described.

In the beginner mode, the model video follows the work operation of the learner 102 during the reproduction of the model video, and the superimposed position and operation speed of the model video are dynamically changed. In this mode, since the model video is adjusted according to the learner's operation pace, the learner can closely observe the work of the instructor and learn the correct work pattern.

Normal mode is a training mode that emphasizes the speed of the instructor's work and plays the model video so that the learner can improve the quality of the work while getting used to it. In this mode, the model video is basically played at the work speed of the instructor, and only when the work operation of the learner 102 is delayed by a certain amount or more, the superimposed position and the playback speed of the model video match the learner 102. Controlled.

The processing in each training mode is as shown in the flow shown in FIG. 3, and in each training mode, the display adjustment unit 16C displays the display content of the model video according to the difference determination result by the difference determination unit 16A of the learner 102. It is changed dynamically according to the work operation. Specifically, the display adjustment unit 16C causes the learner 102 to display, based on the difference determination result, an action element whose difference is equal to or greater than a predetermined value in a manner that the learner 102 can easily recognize it than an action element whose difference is less than the predetermined value. As described above, the transmittance of each operation element of the model video is dynamically changed, and the presence/absence of display is dynamically switched.

Specifically, for example, as shown in FIG. 12A, a circle 103b2 indicating the pressure of the instructor, a gaze point 103a of the instructor, and

movement loci

103c and 103d are displayed, and the pressure and the gaze point of the learner are displayed. When the difference between the instructor's pressure and the gazing point changes so as to decrease, it means that the learner is working with an appropriate pressure and gazing point. As shown in 12(a) to 12(c), transparency control is performed to increase the transparency of the displayed pressure 103b2 and the gazing point 103a of the instructor. On the contrary, the display adjustment unit 16C displays the operation element, which has a low transparency and is emphasized, with respect to the operation element that changes so that the difference between the work operation of the learner and the work operation of the instructor increases.

By such transparency control, the display of the action elements that the learner is not good at (here, drawing the correct trajectory and moving the probe) is emphasized, so the learner conducts the training while paying attention to the action elements that he/she is not good at. be able to.

[Program structure and overall training flow]
Hereinafter, an example of the configuration of the technical training program executed by the learning support system of the present embodiment and the flow of the entire training will be described. The information storage unit 17 stores a plurality of modes as a learning support program as shown in Table 1, for example. Specifically, in addition to the above-mentioned beginner mode and normal mode, the level determination mode, the guidance mode, and the look-back mode are stored in the information storage unit 17.

When executing these modes, the calculation unit 16 of the control system 101, in addition to the video capturing unit 161, the superimposition display unit 162, the difference calculation unit 163, the difference determination unit 16A, and the display adjustment unit 16C, as illustrated in FIG. A level evaluation unit 16B that evaluates the skill level of the learner 102 based on the difference determination result by the difference determination unit 16A and a statistical calculation unit 16D that statistically calculates the skill data of the learner 102.

A known method can be used as a method in which the superimposed display unit 162 aligns and superimposes the learner's visual field image and the model moving image. For example, the superimposed display unit 162 is attached to or around the body (hand, etc.) of the learner 102, an object to be ultrasonically examined, a probe possessed by the learner, or the like, which is in the line of sight of the learner 102 during skill acquisition. The mark is used as a reference marker to determine the display position of the model video.

The function of the calculation unit 16 is realized as software mounted on the CPU or GPU. Further, some or all the functions of the arithmetic unit 16 can be realized by hardware such as ASIC (Application Specific Integrated Circuit) and FPGA (Field Programmable Gate Array).

In the information storage unit 17, in addition to a learning support program including a training mode for skill acquisition, basic information of the learner 102 (for example, differences in individual physical abilities such as age, dominant hand, habit, visual acuity, grip strength) and training are provided. The data table of the learner 102 including the model data regarding the work operation performed by the instructor according to the type of, the log of the technical data of the learner 102, the weakness list of the learner 102, and the like is stored. The model data stored in the information storage unit 17 includes a model video, a plurality of motion elements of the instructor's work motion (position, speed, pressure, motion locus, gazing point, and sound), basic information of the model motion video ( For example, it includes the dominant hand, the examination site to perform ultrasonic examination in the model video, the examination content, the body type of the patient), the display method of the difference when the difference between the movements of the instructor and the learner for each movement element, etc. .. Further, the information storage unit 17 stores the image captured by the camera 15 and the calculation result of the calculation unit 16 as necessary. Further, the information storage unit 17 can also store the video to be displayed on the display unit 19.

The following describes the overall flow of training by the learning support system with reference to FIGS. In this flow, a case will be described in which six motion elements of position, speed, pressure, motion locus, gazing point and sound are used as motion elements for determining the difference between the work motion of the learner and the work motion of the instructor.

Further, in this flow, a case where the learner holds the sensing device 204 (see FIG. 1) for training instead of the probe 202 will be described. When training is performed using the sensing device 204, in addition to the glasses-type device 201, the sensing device 204 is connected to the control system 101 via the network 30. The sensing device 204 can include a pressure sensor, a tilt sensor, a temperature sensor, a humidity sensor, an optical sensor, a haptic device (e.g., a vibrator) that gives haptic feedback to the learner 102, and the like. When the sensing device 204 is connected to the control system 101, the information acquired by the sensing device 204 can be used for the calculation by the calculation unit 16. Further, the multiple superimposition display unit 162 can convert information such as pressure and inclination detected by the sensing device 204 into an image that can be displayed on the display unit 19 and superimpose the information on the visual field image of the learner 102. When the sensing device 204 is equipped with a sensor that acquires information such as temperature, humidity, and light intensity that cannot be acquired from the camera 15, the information acquired by the sensor can be used for display control of the model video by the display adjustment unit 16C. it can. Even if the information can be acquired from the camera 15, the information may be acquired with higher accuracy than the camera 15 by using the sensing device 204. It is preferable to use, as the sensing device 204, a tool having the same shape as a tool used for training (for example, a probe of an ultrasonic imaging apparatus).

[Step S1]
First, the control system 101 receives an input of basic conditions required for training from a learner. Specifically, the control system 101 causes the display unit 19 to display a UI screen as shown in FIG. The learner inputs the learner's basic information and the type of training on the displayed UI screen. The input information is sent to the control system 101 via the network 30 and stored in the information storage unit 17. As a result, the content for executing the training program is activated and delivered to the glasses-type device 201.

[Step S2]
Next, the control system 101 distributes the content in the guidance mode to the glasses-type device 201, and the guidance mode is implemented. In the guidance mode, the learner performs a work operation according to the guide displayed on the display unit 19, so that the learner 102 should perform the work operation in the training mode to be executed later, confirm the displayed content, and perform training. You can learn how to use contents such as motivation. In this mode, the difference between the skill level of the learner 102, the work action of the learner and the work action of the instructor, how the proficiency level determination is displayed, the difference between the training modes, and the like are described. 102 is displayed. For example, when the work motion of the learner deviates from the work motion of the instructor during the training mode, the learner 102 can experience and learn how the information is displayed. The guidance mode may be performed after the level determination mode, or may be repeatedly performed until the learner learns how to use the content.

[Step S3]
Next, the control system 101 distributes the content in the level determination mode to the glasses-type device 201, and the level determination mode is executed. Although a specific flow of this mode will be described later, in this mode, the learner works while the model moving image is not reproduced. The difference determination unit 16A and the level evaluation unit 16B evaluate the current proficiency level of the learner 102, and set the difficulty level of the next training mode (beginner mode, normal mode, etc.) according to the proficiency level evaluation. .. The proficiency evaluation method is the same as the level evaluation method in each training mode.

[Step S4]
Next, the training is performed in the training mode set in the level determination mode. The specific flow in the training mode is as described in FIG. 3, and the difference calculation unit 163, based on the visual field image of the learner 102 obtained from the camera 15 and the sensing device 202, each operation element (position, speed, Regarding pressure, motion locus, gazing point, and sound), a difference between the motion of the learner 102 and the motion of the instructor included in the model video is calculated. At this time, as shown in FIG. 17, the difference calculation unit 163 allocates six operation elements for each unit time and obtains the difference between each operation element divided for each unit time. As shown in the spider chart of FIG. 17, the difference determination unit 16A determines that the difference calculated by the difference calculation unit 163 for each operation element is the allowable difference size (threshold value) predetermined for each operation element. It is determined whether it is 116 or more. When the difference between the learner and the instructor is equal to or greater than the threshold value, the display adjustment unit 16C adjusts the display content of the model moving image. It should be noted that the motion elements whose difference is less than or equal to the threshold value 116 (five motion elements other than sound in this figure) are motion elements that the learner was able to do well during the training, and motions whose difference exceeds the threshold value 116. Elements (sounds in this figure) are motion elements that the learner could not do well.

[Step S5]
When the training mode ends, the control system 101 activates the content for determining the skill level, and the level evaluation unit 16B determines the skill level of the learner under training. Specifically, as shown in the spider chart of FIG. 17, the level evaluation unit 16B uses the difference determination result by the difference determination unit 16A to determine which of the six operating elements has a difference of 116 or less. Evaluate the learner's proficiency according to the existence

The relationship between the number of motion elements that have become less than or equal to the threshold value 116 and the proficiency level can be set arbitrarily. For example, out of the six action elements, the number of action elements having a threshold value of 116 or less and the number of levels may be the same, and when the number of action elements having a threshold value of 116 or less is five, as shown in FIG. The evaluation unit 16B may determine the proficiency level of this learner as "level 5".

In addition, depending on the determined proficiency level, it is possible to change the learning mode and the content to be focused on in the learning to be performed next by the same learner 102. Further, for example, when the level of the learner exceeds a certain level such as the difference of all the motion elements becomes equal to or smaller than the threshold value 116 in the training result in the beginner mode, the next mode of training is advanced to the normal mode. You can

[Step S6]
Next, the proficiency level determined in step S5 is displayed to the learner by the glasses-type device 201, and the look-back mode is executed according to the proficiency level. In the look-back mode, the learner 102 is reproduced the work motion of the learner in the training mode of step S4, and the learner objectively grasps the current skill level. In this step, it is preferable to display to the learner how much the proficiency level has improved compared to the initial proficiency level determined in step S3.

For example, when the visual field image of the learner captured by the camera 15 is stored in the information storage unit 17 during the training mode, the superimposed display unit 162 reads the stored visual field image of the learner in this mode. A retrospective video with a model video superimposed on the field of view video is displayed. In particular, the work motion of the learner 102, which causes the motion element in which the difference becomes equal to or larger than the threshold value 116 (that the learner cannot do well), may be called and reproduced. This allows the learner to objectively grasp how different his/her current technique is from the model technique or where his/her weaknesses are.

In this look-back mode, the statistical calculation unit 16D generates a statistical graph (such as a histogram or a radar chart as shown in FIG. 17) based on the difference determination result of each operation element determined by the difference determination unit 16A, and learns with the statistical graph. The skill level of the person may be displayed. Further, instead of the statistical graph, for example, skill data and a list of weak points as shown in FIG. 18 may be displayed on the display unit 19 to be displayed to the learner. By displaying the statistical graph or the list, the learner 102 can easily confirm his/her own strengths or weaknesses. The list and the statistical graph are stored in the information storage unit 17.

The retrospective mode may be executed after the level determination mode, and is preferably executed after each training mode. By performing the look-back mode each time after performing the level determination mode or the training mode, the learner 102 can perform the next training while recognizing his/her current skill level.
Through the above steps, the training for the learner 102 to acquire the technique is carried out.

Here, a specific flow of the level determination mode of step S3 will be described with reference to FIGS. 15 and 19.

[Step S31]
When the level determination mode starts, the content of the level determination mode is displayed on the display unit 19 of the glasses-type device 201, and the camera 15 captures the view image of the learner 102 from the view image of the learner 102. The visual field image obtained from the camera 15 is transmitted to the control system 101. Further, the sensing device 202 measures the work operation of the learner 102 with each sensor, and transmits the measured data to the control system 101.

In the level judgment mode, the level judgment may be performed by the same work as the training mode performed later by the learner, or the level judgment may be performed by basic work simpler than the training mode. When the level is determined by the same work as the training mode, the number of modes stored in the information storage unit 17 can be reduced. On the other hand, when the level determination is performed by a basic work that is easier than the work performed in the training mode, the number of modes stored in the information storage unit 17 increases, but it is determined whether the level of the learner 102 has reached the basic level. be able to.

[Step S32]
Based on the information obtained from the camera 15, the difference calculation unit 163 calculates the difference between the work motion of the learner 102 and the work motion of the model (not displayed in this mode) for the six motion elements.

[Step S33]
The difference determination unit 16A determines whether or not the difference with the hand of the learner 102 is greater than or equal to a threshold in each action element, and transmits the determined difference determination result to the level evaluation unit 16B.

[Step S34]
The level evaluation unit 16B determines the proficiency level of the learner according to the difference determination result, and determines whether to perform the training in the normal mode or the beginner mode.

The level evaluation unit 16B determines to perform training in the normal mode when the level of the learner is equal to or higher than the predetermined level, and determines to perform training in the beginner mode when the level of the learner is equal to or lower than the predetermined level.

[Step S35]
The level evaluation unit 16B transmits the level determination result determined in step S34 to the glasses-type device 201, and the level of the learner is displayed. Further, similarly to the look-back mode performed after the execution of the training mode, it is preferable to execute the content that looks back on the work operation of the learner in the level determination mode.

[Step S36]
The display adjustment unit 16C constructs the training content according to the mode determined in step S34, and distributes the training content to the glasses-type device 201. After delivering the training content, the training flow by the learning support system returns to step S4 of FIG. 14, and the training mode is executed. The display adjusting unit 16C may change the reproduction speed of the model moving image or adjust the display by other methods in addition to the reproduction speed of the model moving image according to the training mode to be constructed.

[Other display adjustment examples]
An example of adjusting the display of the model moving image according to each mode and each operation element by using a method other than that described above is shown below.
For example, when it is found from the difference determination result of the difference determination unit 16A that the difference between the gazing point of the learner 102 and the gazing point of the instructor is large and deviates from the threshold value, the display adjusting unit 16C causes the display unit 19 to display. It is possible to make the learner 102 acquire the correct gaze point by displaying or highlighting the correct gaze point for a long time.

Also, in the beginner mode, I explained that adjustments are made to slow down the instructor's operation speed according to the learner's operation speed, but in the method in which the instructor's work operation follows the learner's work operation, the playback speed is set. There are other methods than dropping. For example, when the learner's motion is delayed by a certain amount or more, the display adjustment unit 16C may pause the model video and construct content to be rewound and played back so as to start over from the previous process. In addition, it was explained that the normal mode is a mode in which the learner trains according to the movement speed of the instructor, but even in the normal mode, if the learner's work movement is delayed by a certain amount or more, the display adjustment unit 16C is used as a model video. The learner's work may be able to catch up with the model video by pausing the playback of.

The display adjustment unit 16C may construct the content so that when the respective motion elements of the work motion of the learner match the motion elements of the instructor, the match is displayed to the learner. As the display method, feedback by display, sound, or vibration can be used. If the method of displaying a match is common to all motion elements, the learner can easily recognize that the matched motion element can be worked as a model.

The display adjustment unit 16C improves the deviation between the learner's motion element and the instructor's motion element, which has been observed for a certain period of time, by a predetermined value or more (when the learner 102's technique has improved by a certain level or more). Alternatively, when a certain series of work operations is cleared, the content may be constructed so as to give the learner 102 feedback that makes the learner feel improvement. For example, the display adjustment unit 16C can display to the learner that the level has been increased by feedback such as display, sound, and vibration, or by highlighting an improved operation element.

In addition, the display adjustment unit 16C constructs the content that gives the learner tactile feedback such as vibration and impact by the sensing device 202 in addition to the visual and auditory feedback by display and sound to the learner. You may make it have a feeling of actually working.

As described above, in the learning support system of the first embodiment, the display of the model video can be dynamically changed according to the work operation of the learner. Therefore, each learner, such as a learner with a different level or a learner with different characteristics, can train the technique at a learning pace that suits them, and it becomes easier to learn the skill.

In addition, in the learning support system of the first embodiment, subtle tips that are difficult to express in words and the like, which are necessary for learning the technique, are extracted as motion elements, and the difference between the techniques of the instructor and the learner for each motion element. Since the difference determination result is obtained and displayed, the learner can grasp, for each action element, the skill of the skill and the detailed technique, and how much the skill the user has. Since the difference determination result is displayed in a display method that the learner can intuitively learn, the learner can learn the technique by intuitive operation.

Further, in the learning support system of the first embodiment, information such as pressure and line of sight, which is originally difficult to understand only by appearance, is superimposed and displayed on the learner's visual field image, so it is a trick that is difficult to learn when the instructor directly teaches the technique. However, the learner can learn while checking.

In the learning support system of the first embodiment, an example in which one learner performs training using the learning support program stored in the control system 101 has been shown. It is possible to train at the same time or at different timings by using each of the eyeglass-type devices connected via the. Therefore, in the learning support system of the first embodiment, the advanced technology acquired by the instructor can be efficiently transmitted to many learners.

Further, in this embodiment, the skill is mastered in the two training modes of the beginner mode and the normal mode, but the training mode may be one mode or three or more modes. .. Further, the information storage unit 17 may store a mode effective for a learner's skill acquisition in addition to the above modes.

Further, the internal configuration of the control system 101 (the calculation unit 16 and the information storage unit 17) may be stored in a cloud connected to the glasses-type device 201 via the network 30.

<<Learning Support System of Second Embodiment>>
The differences between the learning support system of the second embodiment and the learning support system of the first embodiment will be described below. The learning support system of the second embodiment includes a glasses-type device 201 and a control system 101 connected to the glasses-type device 201 via a network, like the learning support system of the first embodiment. However, in the learning support system of the first embodiment, the display adjustment of the model moving image displayed using the model data of the instructor already stored in the information storage unit 17 of the control system 101 is performed. In the learning support system, the new model data is generated from the model video of the instructor that is not currently stored in the information storage unit 17, and the newly generated model data is used for display adjustment. In this respect, the learning support system of the second embodiment is different from the learning support system of the first embodiment.

As shown in FIG. 20, the learning support system of the second embodiment includes a glasses-type device 201 that can be worn by the learner 102 and a device that can be worn by the instructor 103 that is away from the learner (for example, glasses). Type device) 203, and a glasses-type device 201, and a control system 101 connected to the glasses-type device 203 via the network 30.

The glasses-type device 203 has the same configuration as the glasses-type device 201, and as shown in FIG. 21, the camera 15B for capturing the visual field image of the instructor 103, the wireless communication device 13, the CPU 12, the memory 18, and the learner's operation. A display unit 19B for displaying an image is provided. Further, the glasses-type device 203 may be provided with a device for recording the work sound, a speaker for reproducing the work sound of the instructor, and the like as necessary.

In the second embodiment, the control system 101 has the same configuration as that of the control system 101 in the first embodiment, and includes a calculation unit 16 that processes information received from the glasses-type device 201 and glasses-type device 203, and model data. It includes an information storage unit 17 that stores the information.

Further, in the second embodiment, as shown in FIG. 22, the calculation unit 16 includes a video capture unit 161 that captures the visual field image of the learner 102 and the visual field image of the instructor 103, the superimposed display unit 162, and the difference calculation unit 163. At least a difference determination unit 16A, a display adjustment unit 16C, and a model generation unit 164 that generates model data from the instructor's visual field image.

Hereinafter, with respect to the flow of the training operation of the learning support system of the second embodiment, differences from the flow of the training operation of the learning support system of the first embodiment will be described with reference to FIG. First, the learner performs work for training. The camera 15 captures the visual field image of the learner 102 (step S41). The video capturing unit 161 receives the visual field image captured by the camera 15, and outputs the visual field image of the learner to the display unit 19B of the glasses-type device 203 of the instructor 103 (step S41B1). The instructor sees the learner's training movement displayed on the display unit 19B, grasps the characteristics of the learner's work movement, and performs a model work in a style that matches the characteristics of the learner's work movement. .. The camera 15B captures a visual field image of the instructor 103 while the instructor 103 is performing a model work. (Step S41B2). The image capturing unit 161 receives the visual field image captured by the camera 15B, and the model generation unit 164 generates a model video from the visual field image of the instructor 103 and also generates model data from the example video of the instructor 103 (step S41B3). ). The model data newly generated from the model video of the instructor 103 is stored in the information storage unit 17, and is used when the display adjustment unit 16C adjusts the display content according to the work operation of the learner. In the second embodiment, the step S42 and subsequent steps (steps S42 to S45) in which the display adjusting unit 16C superimposes and displays the model moving image and the visual field image of the learner are the same as those in the first embodiment.

In the learning support system of the second embodiment, in addition to the effects obtained by the learning support system of the first embodiment, the instructor's technique not stored in the information storage unit can be utilized for the learner's learning support. The effect that can be obtained is obtained. Further, in the learning support system of the second embodiment, the learner can learn the technique of the instructor who is away from the learner.

In the learning support system of the second embodiment, the work video of the instructor not stored in the information storage unit is recorded to create a model video, and the created model video is used for the learner's learning support. However, the examples of allowing learners to receive training support from remote leaders are not limited to the above examples.

For example, it is possible to send each other's video and audio to the instructor's glasses-type device and the learners' glasses-type device. Specifically, on the glasses-type device on the side of the instructor, an image of the learner performing a training motion using the glasses-type device is displayed, and the instructor watches the state of the training motion of the learner, Advise the learner on the tips of the work, etc. by movements and voices. The instructor's action image data and voice data are sent to the learner's glasses-type device, and the instructor's action image is displayed on the learner's glasses-type device, and the instructor's voice is played. It will be done. In this way, the learner can learn the technique more efficiently by transmitting the advice from the instructor who is far from the learner to the learner who is training.

<<Learning Support System of Third Embodiment>>
Hereinafter, the differences between the learning support system of the third embodiment and the learning support system of the first embodiment will be described. In the learning support system of the third embodiment, the artificial intelligence (AI) that has learned the techniques of a plurality of instructors selects the model video that is the closest to the characteristics of the learner's work motion as the learner's work motion progresses. , The displayed model video changes every hour.

Like the learning support system of the first embodiment, the learning support system of the third embodiment includes a glasses-type device 201 and a control system 101 connected to the glasses-type device 201 via a network.

In the third embodiment, the control system 101 has the same configuration as the control system 101 in the first embodiment, and includes a calculation unit 16 and an information storage unit 17 as shown in FIG. 24. Further, the configuration of the calculation unit 16 in the third embodiment is the same as that of the calculation unit 16 in the first embodiment, and the video capture unit 161, the superimposed display unit 162, the difference calculation unit 163, and the difference determination unit 16A , A display adjusting unit 16C is provided at least.

The information storage unit 17 in the third embodiment is an artificial intelligence that changes the display of the model video by using the storage unit 171 including the learner's basic information, skill data, and the weakness list, and the technical features of a plurality of instructors. And a section 172. As a result of selecting the learned model 172A for selecting the combination data having the feature closest to the feature of the learner's work motion, and the model video to be displayed from the combination data selected by the learned model 172A, in the artificial intelligence unit 172. And a display unit 172B.

The learned model 172A is a model in which a large number of combination data of learner characteristics and instructor characteristics are learned in addition to the model video. The learned model 172A is composed of, for example, a neural network. As a combination of the learner's characteristics and the teacher's characteristics learned by the learned model 172A, there are six motion elements (position, speed, pressure, motion locus, gazing point, and sound) of work motion, basic information (for example, dominant hand). , The examination site where the ultrasonic examination is trained, the examination contents, and the patient's body shape) are included.

Hereinafter, the operation flow of the learning support system of the third embodiment will be described with reference to FIG. 25.

First, the camera 15 captures the visual field image of the learner 102. Next, the image capturing unit 161 captures the view image of the learner captured by the camera 15 via the network 30. (Step S41). Next, the trained model 172A extracts the characteristics of the learner's work movement from the learner's visual field image, and compares the characteristics of the learner's movement with a large number of combination data learned in the trained model 172A. As a result, the combination data having the feature closest to the learner's feature is selected (step S411). The result display unit 172B selects, from the combination data selected by the learning model 172A, a model moving image having characteristics closest to the learner's characteristics, and transfers the model moving image to the superimposition display unit 162. The superimposition display unit 162 superimposes and displays the received model video on the visual field image of the learner (step S42). When the model moving image closest to the characteristics of the learner's work motion is displayed, the flow of motion returns to step S41, and this flow is repeated every unit time.

As described above, in the learning support system of the third embodiment, the model video closest to the characteristics of the learner's work motion is changed and displayed, so that the model video most suitable for the learner's instruction is displayed every hour. To be done. This makes it possible to provide training content that matches the characteristics of the learner at any time during the training, and the learner can learn the technique more efficiently.

<<Modification of Third Embodiment>>
Another example of adjusting the model moving image using artificial intelligence (AI) that has learned the techniques of a plurality of instructors will be described with reference to FIG. In this modified example, the artificial intelligence unit changes the method of displaying the difference between the work elements of the learner and the instructor in which the difference is larger than the threshold value, depending on the magnitude of the difference between the respective operation elements.

The learning support system of the modification has the same configuration as the learning support system of the third embodiment, but the contents of the combination data learned by the learned model 172A are different. In the learning support system of the modified example, the learned model 172A includes a large number of combinations of the magnitude of the difference for each motion element and the optimal method for displaying the magnitude of the difference. For example, regarding the difference between the positions of the elbows of the learner and the instructor, when the difference is less than or equal to a predetermined value, it is displayed with the letters “lift the elbow more!” as shown in FIG. The combination of the size of the difference and the optimum display method is learned, such as showing with a sign like 19b and superimposing the skeleton diagram 19c as shown in Fig. 6 on the hand of the instructor of the model video when the difference is large. Learned by the model 172A.

Also, the learned model 172A includes model videos, learner characteristics and teacher characteristics. The characteristics of the learner and the instructor are the six motion elements (position, speed, pressure, motion trajectory, gazing point, and sound) of the work motion, and basic information (for example, dominant hand, examination for training ultrasonic examination). The site, examination contents, patient's body type, etc. are included.

The flow of operation of the learning support system of the modified example of the third embodiment will be described below.

In the flow of the learning support system of the modified example, steps S41 and S42 from when the camera 15 captures the visual field image of the learner 102 to when the model video is superimposed on the visual field image are the flow in the first embodiment ( 3). When the model video is superimposed on the learner's visual field image, the difference calculation unit 163 extracts each of the learner's six motion elements from the learner's visual field image, and reads out each of the instructor's six motion elements from the model 172A. .. Next, the difference calculation unit 163 calculates the difference between the action of the learner and the action of the instructor for each action element. (Step S43C).

Next, the difference determination unit 16A determines whether or not the difference calculated by the difference calculation unit 163 is greater than or equal to a threshold for each action element, and notifies the artificial intelligence unit 172 of the determination result (step S44C).

Next, the learned model 172A captures the visual field image of the learner, and extracts the motion element whose difference is equal to or more than the threshold and the magnitude of the difference. The trained model 172A compares the extracted motion elements and the magnitude of the difference with the combination data trained in the trained model 172A to obtain combination data including the optimum display method of the difference for each motion element. Select (step S45C1). The result display unit 172B determines the most effective method for training the learner as a method for displaying the difference between each motion element from the combination data selected by the learning model 172A, and adjusts the display using the method. To the display adjustment unit 16C (step S45C2). The display adjustment unit 16C adjusts the display method of the model moving image and displays it on the display unit 19 according to the instruction received from the result display unit 172B (step S45). When the model moving image processed in step S45 is displayed to the learner, the flow returns to step S41.

As described above, in the learning support system according to the modified example of the third embodiment, the difference between the learner's motion and the instructor's motion is displayed by an optimum display method for each motion element, so that the learner is displayed. You can easily understand how you should change your behavior by looking at the differences.

<<Learning Support System of Fourth Embodiment>>
Hereinafter, the learning support system of the fourth embodiment will be described by referring to differences from the learning support system of the first embodiment. As shown in FIG. 27, the learning support system according to the third embodiment includes an eyeglass-type device 201 and a control system 101 connected to the eyeglass-type device 201 via a network 30, and the network 30 further includes an ultrasonic wave. The imaging device 104 is connected.

The ultrasonic imaging device 104 may have the same configuration as the conventional ultrasonic imaging device. The captured image acquired by the ultrasonic image pickup device is displayed side by side on the display unit 19 of the glasses-type device 201, in parallel with the work video of the learner 102 and the model video, as shown in FIG. 28, for example. Preferably.

When using the ultrasonic imaging device, it is assumed that the user's viewpoint is not on the hand holding the probe, but on the screen on which the captured image is displayed. Since the user cannot see the motion at hand and the captured image at the same time, it is difficult for a learner unfamiliar with the technique to view the screen while operating the probe, which is one of the causes for making learning difficult.

In the fourth embodiment, the image captured by the ultrasonic image pickup device is displayed around the image at the learner's hand by the glasses-type device 201, so that the movement of the viewpoint of the learner 102 is minimized. It is possible to learn by intuitively grasping the movement of the hand and the change in the captured image due to the movement.

In the second to fourth embodiments, the sensing device 204 described in the first embodiment may be connected to the control system 101, and the learner 102 may use the sensing device 204 during training. Further, in the fourth embodiment, a sensor similar to the sensor included in the sensing device 204 is incorporated in the probe of the ultrasonic imaging apparatus 104 so that the learner 102 can learn the technique of handling the probe while using the actual probe. May be.

<< learning support device >>
In the above embodiment, an example in which the control system 101 is connected to the glasses-type device 201 via the network 30 has been described, but the configuration of the control system 101 is not limited to the above example. The control system 101 may be mounted in the memory 18 in the glasses-type device 201. Further, the control system 101 may be mounted in a memory that can be externally attached to the glasses-type device 201.

In the present embodiment, the glasses-type device 201 is used as the learning support device, but the learning support device only needs to include at least a camera and a display unit. For example, a helmet or hair band equipped with a camera and a user's It may be a display arranged nearby or a combination of portable displays.

Further, in the present embodiment, the case where the technique for learning by the learner is the probe operation of the ultrasonic imaging device has been exemplified, but the present invention can be applied to learning in all technical fields. For example, it may be used for learning surgical methods, rehabilitation, sports, how to move the body in traditional performing arts, nursing care methods, cooking methods, playing musical instruments, practicing how to write letters, etc. You may be asked. Further, the present invention may be used for technical learning in the industrial field such as wiring work and welding work.

12 ... CPU, 13 ... wireless communication device, 14 ... motion sensor, 15 ... camera, 16 ... calculation unit, 16A ... difference judgment unit, 16B ... level evaluation unit, 16C ... display adjustment unit, 16D ... statistical calculation unit, 17 ... Information storage unit, 18... Memory, 19... Display unit, 20... Eye sensor, 30... Network, 101... Control system, 201, 203... Glasses type device, 204... Sensing device

Claims

A display unit attached to the learner, an image capturing unit attached to the learner to capture the visual field image of the learner, and a model video which is a video of the work motion of the instructor serving as a model of the work motion of the learner is stored. A storage unit and a calculation unit,
The arithmetic unit displays the model moving image on the display unit by superimposing the model moving image on the visual field image captured by the image capturing unit, and moves the moving image in accordance with the feature of the work operation of the learner included in the visual field image. A learning support system characterized by changing the display content of the model video.
The arithmetic unit is
A difference calculation unit that calculates the difference between the work motion of the instructor included in the model video and the work motion of the learner included in the visual field image, and a display that changes the display content of the model video according to the difference. The learning support system according to claim 1, wherein the learning support system includes a coordinating unit.
The difference calculation unit calculates the difference for each of a plurality of predetermined operating elements.
The learning support system according to claim 2, wherein the display adjustment unit changes the display content of the model moving image in accordance with the difference for each operation element.
The difference calculation unit calculates the difference for each of a plurality of predetermined operating elements.
Among the plurality of motion elements, the display adjustment unit displays the display content of the model video so that the calculated motion difference exceeds a predetermined value, and that the difference is emphasized more than a motion element having a difference equal to or less than a predetermined value. The learning support system according to claim 2, wherein the learning support system is changed.
The difference calculation unit calculates the difference for each of a plurality of predetermined operating elements.
The display adjustment unit increases the transmittance of the model moving image displayed on the display unit for an operation element of which the difference is smaller than a predetermined value among the plurality of operation elements. The learning support system described in.
The difference calculation unit calculates the difference for each of a plurality of motion elements,
The display adjustment unit determines a learner's proficiency level from the difference calculated for each of the plurality of motion elements, and changes the content of the model video in accordance with the determined proficiency level. The learning support system described in 2.
The calculation unit further has a proficiency level determination unit that determines the initial proficiency level of the learner prior to the process of displaying the model video on the display unit,
The learning support system according to claim 1, wherein the display adjustment unit changes the content of the model moving image displayed on the display unit according to the initial proficiency level.
The learner's visual field image captured by the imaging unit is stored in the storage unit.
The learning support system according to claim 1, wherein the arithmetic unit reproduces a retrospective moving image in which a model moving image is superimposed on the learner's visual field image stored in the storage unit.
Further comprising a glasses-type device and a control system connected to the glasses-type device via a network,
The display unit and the imaging unit are provided on a glasses-type device worn by a learner.
The learning support system according to claim 1, wherein the calculation unit and the storage unit are provided in the control system.
Further comprising a sensing device connected to the control system via a network,
The learning support system according to claim 9, wherein the arithmetic unit dynamically changes the display content of the model moving image based on the information detected by the sensing device.
Further comprising a device connected to the control system via a network for capturing a visual field image of the instructor,
The learning support system according to claim 9, wherein the calculation unit generates the model moving image from a visual field image of the instructor captured by the device.
The storage unit stores an artificial intelligence unit that has learned a plurality of features of the learner's work action and the features of the instructor's work action,
The artificial intelligence unit selects, as needed, a model video of the work motion of the instructor having characteristics close to the characteristics of the work motion of the learner, and modifies the model video overlaid on the visual field video at any time. The learning support system according to Item 1.
An ultrasonic imaging device is further connected to the network,
The learning support system according to claim 9, wherein the arithmetic unit causes the display unit to display the ultrasonic image acquired by the ultrasonic imaging apparatus.
A display unit attached to the learner, an imaging unit attached to the learner to capture the learner's visual field image, and an imaging unit.
A storage unit that stores a model video that is a video of the work movement of the instructor who is a model of the learner's movement,
A calculation unit, and the calculation unit causes the display unit to superimpose the model moving image on the visual field image captured by the imaging unit and displays the work motion of a learner included in the visual field image. A learning support device characterized in that the display content of the model moving image is dynamically changed according to the above.
Computer,
Learning is performed by superimposing a model video, which is a video of the instructor's work motion, which is a model of the learner's operation, on the visual field image captured by the image capturing unit attached to the learner and capturing the visual field image of the learner. A program that is displayed on a display unit worn by a person and that functions as a calculation unit that dynamically changes the display content of the model moving image according to the characteristics of the learner's work motion included in the visual field image.