JP4134147B2 - Operation instruction device - Google Patents

Description

  The present invention relates to an operation instruction device that detects an operation of a user and instructs an operation to be performed next by the user.

  Interfaces for activating communication of care recipients with higher brain dysfunction and dementia and supporting family members who are caregivers have been studied. For example, Non-Patent Document 1 discloses information using a network. A therapy interface is disclosed. Such an information therapy interface provides information according to the condition and intention of the care recipient who is the user, and provides a place for communication between the care recipient and the caregiver via the network. It is a system that reduces the burden.

In the system as described above, it is necessary to extract the behavior pattern of the cared person who freely moves in the room to estimate the state and intention of the cared person, and it is necessary to detect the movement of the person, particularly the posture, with high accuracy. There is. As a method for detecting the posture of a person, there is a passive observation method in which an image including a person is captured and the posture of the person is detected by image processing. Since this passive observation method can use image processing that is a non-wearing and non-contact detection method, the burden on the care recipient can be reduced.
Shinji Tetsuya et al., “Information therapy interface using network”, IEICE Tech. 103-747, p. 31-p. 36

  However, in the passive observation method as described above, detection processing is unstable due to factors depending on the scene such as clothes of the cared person, changes in environmental conditions such as indoor lighting conditions, and occlusion between people. In some cases, the posture of the person cannot be accurately estimated. As a result, the movement of the care recipient cannot be accurately estimated, and an accurate instruction for supporting the movement cannot be given. In addition, when the posture of the cared person is detected using an image that can identify the cared person, the care receiver's privacy cannot be ensured.

  On the other hand, the active observation method in which a sensor or marker is attached to the cared person's body makes it possible to detect the posture with high accuracy. The burden on the person becomes excessive.

  The objective of this invention is providing the operation | movement instruction | indication apparatus which can give an exact instruction | indication to a user while being able to reduce a user's burden, ensuring a user's privacy.

The motion instruction apparatus according to the present invention includes a detection unit that detects a user's three-dimensional shape in a non-contact manner with respect to the user, and an estimation that estimates a user's motion based on the user's three-dimensional shape detected by the detection unit. And a presenting means for presenting to the user an instruction related to an action to be performed next based on the user's action estimated by the estimating means, and the detecting means emits a plurality of spot lights to the user. Projecting means for projecting in a predetermined pattern, photographing means for photographing a plurality of light spots projected on the surface of the user by the projecting means, and a plurality of light spots photographed by the photographing means as the three-dimensional shape of the user Position detecting means for detecting a three-dimensional position distribution on the surface of the user based on the position of the projection, the projection means into a spot light group that does not interfere with each other on the captured image. Splitting means for splitting, and splitting projection means for projecting spot light to the user for each spot light group divided by the splitting means, wherein the photographing means is provided for each spot light group by the split projection means. A plurality of light spots projected on the surface of the user are photographed, and the position detecting means detects a three-dimensional position distribution on the surface of the user based on the positions of the light spots photographed for each spot light group by the photographing means. Is.

In the motion instruction device according to the present invention, the user's three-dimensional shape is detected in a non-contact manner with respect to the user, the user's motion is estimated based on the detected user's three-dimensional shape, and the estimated user's motion since instructions regarding operation which a user should do next based on is presented to the user, without Ru with identifiable images like the user, also in a state of non-attachment and non-contact observation instrument for the user The user's motion can be estimated and an instruction suitable for the motion can be given, the user's burden can be reduced while ensuring the user's privacy, and an accurate instruction can be given to the user.
In addition, a plurality of spot lights are projected on the user in a predetermined pattern, a plurality of light spots projected on the user's surface are photographed, and the three-dimensional surface of the user is based on the positions of the photographed light spots. Since the position distribution is detected, it is possible to detect the three-dimensional position distribution on the surface of the user with high accuracy in a state where the observation apparatus is not attached to the user and in a non-contact state. Moreover, since the position of each light spot is actively observed by irradiating a plurality of spot lights, it is possible to stably estimate the user's operation without being excessively influenced by factors depending on the scene. it can.
Further, a plurality of spot lights are divided into spot light groups that do not interfere with each other on the captured image, and the spot lights are projected to the user for each of the divided spot light groups, and each spot light group is projected onto the user's surface. Since a plurality of light spots are photographed and the three-dimensional position distribution of the user's surface is detected based on the positions of the light spots photographed for each spot light group, the three-dimensional position distribution of the user's surface can be obtained at high speed. It can be detected efficiently.

  The estimation means is stored in a distribution storage means that stores in advance a reference three-dimensional position distribution of the user's surface for each user posture, a three-dimensional position distribution detected by the position detection means, and the distribution storage means. A posture estimation means for estimating the current posture of the user by comparing with a reference three-dimensional position distribution, and a motion memory that stores in advance posture information representing the posture of the user constituting the motion for each motion of the user It is preferable to include means for estimating the current motion of the user by comparing the posture of the user estimated by the posture estimation device and the posture information stored in the motion storage device.

  In this case, the current posture of the user is estimated by comparing with a reference three-dimensional position distribution stored in advance for each posture of the user, and compared with the posture information stored in advance for each operation of the user. Therefore, the user's motion can be estimated with high accuracy.

  The distribution storage means further stores in advance a reference three-dimensional position distribution at the time of use of the appliance for each appliance used by the user, and the posture estimation means further stores 3 of the appliances detected by the position detection means. The use state of the appliance is estimated by comparing the three-dimensional position distribution with the reference three-dimensional position distribution of the appliance stored in the distribution storage means, and the action storage means further relates to the action for each user action. Use state information indicating the use state of the appliance to be stored in advance, and the motion estimation means includes the posture of the user estimated by the posture estimation means and the use state of the appliance, and the posture information stored in the motion storage means, and It is preferable to compare the usage state information to estimate the current operation of the user.

  In this case, the usage state of the appliance is also estimated in comparison with the reference three-dimensional position distribution of the appliance stored in advance for each appliance used by the user, and the posture information and the usage status stored in advance for each user operation Since the user's current motion is estimated compared with the information, the user's motion can be estimated with higher accuracy.

  The position detection unit detects a user's observation position based on the detected three-dimensional position distribution, and the projection unit corrects the position of the spot light according to the user's observation position detected by the position detection unit. It is preferable.

  In this case, the observation position of the user is detected based on the three-dimensional position distribution, and the position of the spot light is corrected according to the detected observation position of the user, so that the three-dimensional position distribution can be detected with high accuracy. Since the user's motion is estimated by comparing the three-dimensional position distribution with the reference three-dimensional position distribution, the user's motion can be estimated with high accuracy even when the user moves.

  The photographing means includes first and second photographing means for photographing a plurality of light spots projected on the surface of the user by the projecting means from different photographing directions, and the position detecting means includes the first and first photographing means. The three-dimensional position distribution of different surfaces of the user is detected on the basis of the positions of the plurality of light spots photographed by the two photographing means, and the motion instruction device detects the three different surfaces of the user detected by the position detecting means. A holding means for holding a three-dimensional position distribution; a three-dimensional position distribution of different surfaces of the user detected by the position detecting means; and a three-dimensional position distribution of different surfaces of the user held by the holding means a predetermined time ago. A part detecting means for detecting a part where the surface of the user has changed and a part where the surface of the user is changed by attaching / detaching the clothes for each piece of clothing worn by the user And further comprising an identification means for identifying the type of clothing that has changed by comparing the part detected by the part detection means with the part stored in the clothes storage means. It is preferable that the estimation means estimates the user's change of clothes operation in consideration of the type of clothes identified by the identification means.

  In this case, the three-dimensional position distribution of different surfaces of the user is detected based on the positions of the plurality of light spots photographed by the first and second photographing means, and the detected three-dimensional position distribution of the different surfaces of the user and A part where a change has occurred is detected by comparing the three-dimensional position distributions of different surfaces of the user a predetermined time ago, and the detected part is changed by attaching and detaching the clothes recorded in advance for each piece of clothes worn by the user. Since the user's dressing motion is estimated in consideration of the identified clothing type, the user's dressing motion is And can be estimated with high accuracy.

  The projection unit projects a plurality of spot lights to the user in a predetermined pattern using infrared light, and the photographing unit photographs a plurality of infrared light spots projected on the surface of the user by the projection unit. It is preferable to do.

  In this case, since the three-dimensional position distribution on the surface of the user is detected using infrared light that is not visible to the user, it is possible to perform robust detection against environmental changes such as illumination conditions, and user privacy. Can be secured.

  According to the present invention, the user's operation is estimated without using an image or the like that can identify the user, and the observation device is not attached or in a non-contact state, and an instruction suitable for the operation is given. It is possible to reduce the burden on the user while ensuring the privacy of the user, and to give an accurate instruction to the user.

  Hereinafter, an operation instruction apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of an operation instruction apparatus according to an embodiment of the present invention.

  1 includes a projector 11, a projection pattern dividing unit 12, a projection pattern generating unit 13, two video cameras 21a and 21b, a three-dimensional position detecting unit 22, a three-dimensional position restoring unit 23, and an attitude estimating unit. 24, distribution database 25, previous position holding unit 26, change part detection unit 27, clothes change identification unit 28, clothes change part database 29, motion estimation unit 30, motion database 31, audio output and video display unit 32, and motion instruction information A database 33 is provided.

  The projector 11 is composed of, for example, a DLP (digital light processing) projector (resolution 800 × 600 pixels), an infrared filter is attached to the lens unit, and a user (for example, a cared person) using infrared light. In addition, a plurality of spot infrared lights are projected in a predetermined pattern onto an instrument used by the user (for example, a toilet bowl). The video cameras 21a and 21b are composed of, for example, a CCD camera, an infrared filter is attached to the lens unit, and a plurality of infrared light spots projected on the surface of the user and the surface of the instrument used by the user are photographed.

  FIG. 2 is a schematic diagram for explaining the positional relationship between the projector 11 and the video cameras 21a and 21b shown in FIG. As shown in FIG. 2, the projector 11 installed above the room, for example, in the middle or one side of the ceiling, projects a plurality of spot infrared lights in a predetermined pattern on the user and the appliance used by the user, The video camera 21a installed above the room, for example, on one side of the ceiling, has a plurality of infrared rays projected onto the surface of the user (specifically, the surface of the body or clothes) and the surface of the appliance used by the user. Take a light spot. The video camera 21b is attached to the opposite side of the video camera 21a and shoots a plurality of infrared light spots projected on the surface of the user and the equipment used by the user from a shooting direction different from the shooting direction of the video camera 21a. . In this way, a plurality of infrared light spots projected on the front surface (front surface) and the back surface (rear surface) of the user and the appliance used by the user can be photographed by the video cameras 21a and 21b.

  3 and 4 are diagrams showing an installation example of the projector 11 and the video camera 21a shown in FIG. 3 and 4, only the video camera 21a is illustrated for the sake of simplicity. In the example shown in FIG. 3, the projector 11 and the video camera 21a are installed in the toilet. In this case, the user's operation when using the toilet can be observed and an instruction regarding the operation of using the toilet can be taught to the user. In the example shown in FIG. 4, the projector 11 and the video camera 21 a are installed in a room where clothes dance is arranged. In this case, the user's change operation can be observed and an instruction regarding the change operation can be taught to the user. .

  In this embodiment, an example in which one projector 11 and two video cameras 21a and 21b are installed in one room will be described. However, the number of projectors and video cameras is particularly limited to this example. However, various modifications are possible. For example, when estimating a user's operation and identifying a change in clothes for each room such as a corridor, bedroom, living room, kitchen, washroom, and bathroom, at least one projector and at least one video for each room. A camera may be installed, and only a user's operation may be estimated using only one video camera. Furthermore, when a wider range of observation is required, a plurality of projectors and a plurality of video cameras may be installed. In particular, when detecting a change in clothes, it is desirable to combine a plurality of projectors and video cameras in order to obtain a three-dimensional position around the user.

  In order to measure a user's three-dimensional position distribution (three-dimensional shape), the projection pattern generation unit 13 is arranged with a whole infrared spot pattern composed of a plurality of spot infrared lights, for example, spot infrared lights arranged in a matrix. An infrared spot pattern is generated. The projection pattern dividing unit 12 divides the plurality of spot infrared lights into spot infrared light groups that do not interfere with each other on the captured image. Specifically, the projection pattern dividing unit 12 allows spot infrared light in each partial infrared spot pattern so that spot infrared light can be independently observed in each partial infrared spot pattern during observation by the video cameras 21a and 21b. The positional relationship of light is calculated to divide the entire infrared spot pattern into a plurality of partial infrared spot patterns.

  The projection pattern dividing unit 12 controls the projector 11 to project spot infrared light with the divided partial infrared spot pattern, and notifies the three-dimensional position detection unit 22 of the current partial infrared spot pattern. The projector 11 projects a plurality of spot infrared lights on the user with the divided partial infrared spot patterns. Therefore, confusion between a plurality of spot infrared lights can be avoided, and the process of detecting spot infrared lights can be simplified and speeded up.

  The three-dimensional position detection unit 22 refers to the current partial infrared spot pattern and calculates the three-dimensional position distribution of the surface of the user and the instrument from the positions of the spot infrared light on the two-dimensional images photographed by the video cameras 21a and 21b. To detect. The three-dimensional position restoration unit 23 collects the three-dimensional position distributions of the surface of the user and the instrument detected for each partial infrared spot pattern and separates the three-dimensional position distributions of the surface of the user and the instrument with respect to the entire infrared spot pattern, respectively. Restore. The three-dimensional position restoration unit 23 detects the user's observation position from the restored three-dimensional position distribution on the surface of the user and outputs the detected position to the projection pattern generation unit 13. Chase.

  The projection pattern generation unit 13 corrects the projection position of the whole infrared spot pattern according to the detected observation position, and the projection pattern division unit 12 divides the corrected whole infrared spot pattern into a plurality of partial infrared spot patterns. The projector 11 projects a plurality of spot infrared lights onto the user and the instrument with the corrected partial infrared spot pattern. At this time, the three-dimensional position detection unit 22 detects the three-dimensional position distribution of the surface of the user and the instrument from the position of the spot infrared light according to the corrected partial infrared spot pattern, and the three-dimensional position restoration unit 23 The three-dimensional position distributions of the surface of the user and the tool detected for each partial infrared spot pattern are collectively collected and restored, and the three-dimensional position distribution of the surface of the user and the tool with respect to the corrected whole infrared spot pattern is separated and restored to estimate the posture. To the unit 24.

  In the distribution database 25, a reference three-dimensional position distribution of the human surface is stored in advance for each human typical posture. For example, a posture in which the human is standing upright and facing forward, and a human is standing upright and facing left. Posture, human standing upright, human being seated and facing forward, human seated and facing left, human seated and facing right A reference three-dimensional position distribution representing a posture, a posture in which a human is changing clothes, a posture in which a human is holding out a hand, and the like is stored.

  The distribution database 25 stores in advance a reference three-dimensional position distribution of the surface of the instrument for each typical usage state of the instrument used by the user. Appliances include electric parts with moving parts, furniture, daily necessities, etc., for example, toilets, clothes dance, etc., toilet cover closed and open, clothes dance door closed and open. A reference three-dimensional position distribution representing a state or the like is stored. The reference three-dimensional position distribution of the user and the instrument stored in the distribution database 25 is not particularly limited to the above example, and a reference three-dimensional position distribution representing another posture and use state may be stored. .

  The posture estimation unit 24 compares the restored three-dimensional position distribution of the surface of the user and the device and the reference three-dimensional position distribution of the user and the device stored in the distribution database 25 to compare the current posture of the user and the device. The usage state is estimated, and the estimation result is output to the motion estimation unit 30. This posture estimation process may be executed using each image obtained from the two video cameras 21a and 21b, or may be executed for each video camera 21a and 21b or using only one video camera. Also good.

  On the other hand, when identifying a change in clothes of the user, the three-dimensional position detection unit 22 refers to the current partial infrared spot pattern, and the position of the spot infrared light on the two-dimensional image captured by the video cameras 21a and 21b. The three-dimensional position distribution of the user's surface (different surfaces) in each shooting direction is detected. The three-dimensional position restoration unit 23 collects the three-dimensional position distribution of the user's surface in each photographing direction detected for each partial infrared spot pattern and determines the three-dimensional position distribution of the user's surface in each photographing direction with respect to the entire infrared spot pattern. The data is restored and output to the previous position holding unit 26 and the change site detection unit 27.

  The previous position holding unit 26 holds a three-dimensional position distribution on the surface of the user in each shooting direction. The change site detection unit 27 reads the three-dimensional position distribution of the user's surface in each imaging direction before a predetermined time from the previous position holding unit 26 and outputs the surface of the user in each imaging direction output from the three-dimensional position restoration unit 23. A part where a change in the clothes of the user has occurred is detected as compared with the three-dimensional position distribution, and is output to the clothes change identification unit 28 along with the increase / decrease direction of the three-dimensional position of this part.

  In the clothing change part database 29, part information representing a part where a change occurs due to attachment / detachment of the clothes for each typical human clothes is stored in advance. The clothes change identification unit 28 compares the detected part with the part stored in the clothes change part database 29 to identify the type of clothes that have changed, and based on the increase / decrease direction of the three-dimensional position. If it has increased, it is determined that the identified clothes are being worn, and if it has decreased, it is determined that the identified clothes have been removed, and the identification result and the determination result are output to the motion estimation unit 30.

  In the motion database 31, for each motion that can be performed by the user in each room, posture information indicating the posture of the user constituting the motion and / or usage status information indicating the usage status of the appliance, and the type of clothing as necessary. Clothes information is stored in advance. In the operation database 31, an operation instruction flag is stored in advance for each operation for an operation that requires an instruction regarding an operation to be performed by the user next.

  For example, when the video cameras 21a, 21b, etc. are installed in the toilet, the posture in which the user stands upright and the front of the user standing in front of the toilet and the closed state of the toilet lid are associated with each other. The opening state of the toilet bowl is associated with the movement of the user opening the toilet lid, and the human being seated and facing forward with respect to the movement of the user exiting the toilet and standing up The postures standing upright and facing forward are associated with each other and stored in the operation database 31 as toilet operation information in the toilet.

  In addition, when video cameras 21a, 21b, etc. are installed in a room where clothes dance is arranged, a posture in which a human is standing upright and facing forward with respect to the movement of the user standing in front of clothes dance and The posture in which the closed state of the door is associated, the opening state of the clothes dance door is associated with the movement of the user opening the clothes dance, and the user is changing clothes for the movement of the user wearing the jacket In the room where the outerwear is associated and the posture that the person is changing and the trousers (or skirt) are associated with the movement of the user wearing the trousers (or skirt), and the clothes dance is arranged respectively. It is stored in the operation database 31 as changing operation information. Note that the posture information, use state information, and clothing information stored in the motion database 31 are not particularly limited to the above example, and information representing other postures, use states, and clothes may be stored.

  The motion estimation unit 30 compares the estimated posture of the user, the usage state of the appliance and the type of clothing with the posture information, usage state information, and clothing information stored in the motion database 31 to determine the current motion of the user. By estimating and determining whether or not an operation instruction flag is added to the estimated operation, it is determined whether or not an operation instruction is necessary for the next operation to be performed by the user. When it is determined that an operation instruction is necessary, the motion estimation unit 30 identifies the operation that requires the instruction and presents the operation instruction information associated with the operation to the user. 32.

  The operation instruction information database 33 stores operation instruction information for each operation that requires an instruction. The audio output and video display unit 32 includes a speaker, a display device, and control circuits thereof, and reads and reads out the operation instruction information associated with the operation specified by the operation estimation unit 30 from the operation instruction information database 33. The operation instruction information is output by voice and displayed by video (character information).

  For example, when video cameras 21a, 21b, etc. are installed in the toilet, when the user enters the toilet and stands in front of the toilet, the operation instruction information is associated with the operation in which the user stands in front of the toilet. When the user opens the toilet lid, the voice instruction “Open the lid” is displayed, and when the user opens the toilet lid, the operation instruction information associated with the operation of the user opening the toilet lid is “Sit down. When the user exits the toilet and stands up, the operation instruction information associated with the operation that the user finishes and exits the toilet is “flowing water”. Please give a voice instruction and video display.

  In addition, when video cameras 21a, 21b, etc. are installed in a room where clothes dance is arranged, if the user enters a room where clothes dance is arranged and stands before the clothes dance, the user will see before the clothes dance. The voice instruction and video display of “Please open clothes dance.”, Which is the operation instruction information associated with the standing action, is performed, and when the user opens the clothes dance, the user can open the clothes dance. When the user wears a jacket, a voice instruction and a video display indicating “Please wear a jacket”, which is the attached operation instruction information, are performed, and the user is associated with the operation of wearing the jacket. When a voice instruction and video display indicating “please wear trousers (or skirt)” as operation instruction information are performed and the user wears trousers (or skirt), the user wears trousers (or skirt). Wear and is an operation instruction information that is associated with the operation is "Please close the wardrobe." Voice instruction and video display and is carried out.

  The method of presenting the operation instruction information is not particularly limited to the above example, and may be presented using only one of audio and video. Also, the operation instruction information is not particularly limited to the above example, and other information regarding the operation to be performed next may be presented.

  In the present embodiment, the projector 11, the projection pattern dividing unit 12, the projection pattern generating unit 13, the video cameras 21a and 21b, the three-dimensional position detecting unit 22, and the three-dimensional position restoring unit 23 correspond to an example of a detection unit, and the posture The estimation unit 24, the distribution database 25, the motion estimation unit 30, and the motion database 31 correspond to an example of an estimation unit, and the audio output and video display unit 32 and the motion instruction information database 33 correspond to an example of a presentation unit.

  In addition, the projector 11, the projection pattern dividing unit 12, and the projection pattern generation unit 13 correspond to an example of a projection unit, and the video cameras 21a and 21b correspond to an example of an imaging unit and first and second imaging units, and are three-dimensional. The position detection unit 22 and the three-dimensional position restoration unit 23 correspond to an example of a position detection unit.

  In addition, the distribution database 25 corresponds to an example of a distribution storage unit, the posture estimation unit 24 corresponds to an example of a posture estimation unit, the motion database 31 corresponds to an example of a motion storage unit, and the motion estimation unit 30 includes a motion estimation unit. It corresponds to an example.

  The previous position holding unit 26 corresponds to an example of a holding unit, the change site detection unit 27 corresponds to an example of a site detection unit, the clothing change site database 29 corresponds to an example of a clothing storage unit, and a clothing change identification unit. 28 corresponds to an example of an identification unit, the projection pattern division unit 12 and the projection pattern generation unit 13 correspond to an example of a division unit, and the projector 11 corresponds to an example of a division projection unit.

  Next, each operation of the operation instruction device configured as described above will be described in detail. First, the infrared spot pattern dividing process will be described. In order to accurately estimate human motion, it is necessary to obtain a high-density map of the three-dimensional position on the surface of the human body. When performing high-density three-dimensional position detection using pattern projection, multiple spots are used. Confusion occurs in the three-dimensional reconstruction process due to the overlapping of the observation positions on the camera image of the pattern. For this reason, in this Embodiment, the overlap between the observation positions on the camera image of a several spot pattern is prevented by the division | segmentation process of the infrared spot pattern which divides | segments a whole infrared spot pattern into a partial infrared spot pattern. In this infrared spot pattern dividing process, the entire infrared spot pattern is divided into a plurality of partial infrared spot patterns based on the intersection determination in order to simplify the detection process.

  First, projection spots on the projector image plane are selected at regular intervals. The spot interval is appropriately determined according to the application. Next, an observable region (for example, a finite portion on the epipolar line) for each point on the camera image is calculated. At this time, it is known that the point (Xp, Yp) on the projector plane and the two-dimensional observation point (Xc, Yc) on the camera plane satisfy the following expressions.

Here, Xp and Yp are the positions of projection points on the projector image plane, and Xc and Yc indicate observation points of camera coordinates. Usually, the height of the target point projected in the scene is within a certain range (Z ₀ <Z <Z _top ), and the observation range can be limited to a part of the epipolar line. Here, the two-dimensional observation point for Z = Z _{0 is represented} by (S _X , S _Y ), and the two-dimensional observation point for Z = Z _top is represented by (L _X , L _Y ) (for example, in the example of FIG. Z ₀ = 0, Z _top = 200), and the observation point for the projection point between them is on a segment connecting two points (S _X , S _Y ) and (L _X , L _Y ).

Next, the intersection between each two segments is determined. Here, if the start point and the end point between two segments are (S _X1 , S _Y1 ) − (L _X1 , L _Y1 ), (S _X2 , S _Y2 ) − (L _X2 , L _Y2 ), respectively, ((S _{X1 -L X1) * (S Y2} -S Y1) + (S Y1 -L Y1) * (S X1 -S X2)) * ((S X1 -L X1) * (L Y2 -S Y1) + (S _{Y1 -L Y1) * (S X1} -L X2)) <0 and _{((S X2 -L X2) *} (S Y1 -S Y2) + (S Y2 -L Y2) * (S X2 -S X1)) If * ((S _X2 −L _X2 ) * (L _Y1 −S _Y2 ) + (S _Y2 −L _Y2 ) * (S _X2 −L _X1 )) <0, it is determined that the two segments intersect. Actually, it is necessary to perform the intersection determination in consideration of the area of each pixel, but the description is omitted here.

  There are N × N combinations for N segments, and the result is represented by the following matrix consisting of binary values (0: crossing, 1: no crossing).

When there are M video cameras, the above matrix is also M, and by calculating the logical product of all the matrices C _i (1 ≦ i ≦ M), a condition that satisfies an appropriate projection is calculated as follows. can do.

Next, the efficiency of the projection pattern will be described. If the number of three-dimensional points (spot infrared light) that needs to be observed is L, the combination of projection points i and j (1 <i, j <L) is L × L. Here, C _{i, j} = 1 if the projection points i, j are simultaneously projected and C _{i, j} = 0 if they do not interfere with each other, and C _{i, j} = 0 if the projection points i, j are projected simultaneously _{. A} matrix X composed of _j (1 <i, j <L) is prepared. The diagonal elements of the matrix X are all 1.

  The projection pattern generation unit 13 outputs the matrix X stored in advance to the projection pattern division unit 12 as an overall infrared spot pattern. The projection pattern dividing unit 12 determines the minimum number of partial infrared spot patterns and the partial infrared spot patterns at that time by applying the procedure P represented by the following equation (6) to the matrix X.

Specifically, A _i ∪A _j = A _i among all combinations of G × (G−1) for an arbitrary matrix A (G rows and H columns, each element A (g, h) of the matrix). The j-th row (plurality) is removed from the matrix A, and a matrix B (G ′ row H column, matrix element B (g ′, h)) is created. The total S of “1” in each column of the matrix B is calculated by the above formula (7), and the column l (l = 1 (1), l (2),..., L (p)) where S is the smallest is calculated. For each k-th row (k = k (1), k (2),..., K (q)) where B _kl = 1, from the elements excluding the k-th row in the m-th column satisfying B _km = 0 A matrix C is created. The projection pattern dividing unit 12 repeats the above processing, and selects a combination of rows A _{k (l)} , A _{k (2)} ,... That minimizes the number of steps P to create a partial infrared spot pattern.

  FIG. 5 is a diagram for explaining an example of a procedure P for creating a partial infrared spot pattern, and FIG. 6 shows a matrix C shown in FIG. 5B from a matrix A shown in FIG. It is a figure for demonstrating in detail the procedure P to produce | generate. From the matrix A shown in FIG. 5A, the row A4 is selected by the procedure P, and the matrix C shown in FIG. 5B is generated. Specifically, as shown in FIG. 6A, in the matrix A (matrix A shown in FIG. 5A), A1∪A5 = A5 and A2∪A4 = A4. A2 is deleted, and as shown in FIG. 6B, the matrix B is generated from the elements in the A3, A4, and A5 rows. The total S of “1” in each column (h = 1 to 5) of the matrix B is 1, 1, 3, 2, 2. Next, attention is paid to the column having the smallest S. For example, paying attention to the second column (h = 2) which is one of the columns in which S is the smallest, the row B2 of the matrix B in which the element of the second column is “1” (row A4 of the original matrix) select. Focusing on the selected row B2 and taking out a column whose element is “0”, a matrix shown in FIG. 6C is created. Next, when the row B2 is removed, a matrix C shown in (d) of FIG. 6 (matrix C shown in (b) of FIG. 5) is generated. With the above processing, the procedure P is executed once.

  Referring to FIG. 5 again, row A5 is selected by procedure P from matrix C shown in FIG. 5B, and matrix C shown in FIG. 5C is generated. In this case, the number of procedures P is two, and the route is A4 → A5. Similarly, a matrix C shown in (d) of FIG. 5 is generated from the matrix A shown in (a) of FIG. 5 by the procedure P (focusing on the first column (h = 1) of the matrix B, row B3 (Select original matrix row A5)). Further, the row A4 is selected from the matrix C shown in FIG. 5D by the procedure P, and the matrix C shown in FIG. 5E is generated. In this case, the number of times of the procedure P is 2, and the route is A5 → A4. In this example, since the minimum number of times of procedure P is two, the partial infrared spot pattern is created using route A4 → A5 or route A5 → A4.

  By the above infrared spot pattern division processing, a plurality of spot infrared lights are divided into partial infrared spot patterns that do not interfere with each other on the captured image, and the spot infrared lights are directed to the user and the instrument by the divided partial infrared spot patterns. Therefore, the detection process of the three-dimensional position distribution on the surface of the user and the instrument can be simplified, and robust three-dimensional measurement can be performed.

  FIG. 7 is a diagram illustrating an example of an edge detection filter used when detecting the contour of a user, and FIG. 8 is a diagram illustrating an example of a detection result of an infrared light spot. The video cameras 21a and 21b sequentially photograph a plurality of infrared light spots projected on the surface of the user for each partial infrared spot pattern in which spot infrared light does not interfere with each other on the photographed image, and the three-dimensional position detection unit 22 The infrared light spot is detected by applying a 3 × 3 simple edge filter shown in FIG. 7B to the target pixel AP shown in FIG. The restoration unit 23 totally restores the three-dimensional position distribution on the surface of the user. As a result, the infrared light spot shown in FIG. 8 could be detected.

  In this way, a plurality of infrared light spots projected on the surface of the user and the instrument are sequentially photographed for each partial infrared spot pattern in which spot infrared light does not interfere with each other on the photographed image, and photographed for each partial infrared spot pattern. Since the three-dimensional position distribution of the surface of the user and the instrument is sequentially detected based on the position of the infrared light spot, and finally the entire three-dimensional position distribution of the surface of the user and the instrument can be restored. In addition, the three-dimensional position distribution on the surface of the instrument can be detected quickly and efficiently.

  Next, a description will be given of correction processing of the observed position of the detected three-dimensional position distribution on the surface of the user. When an infrared spot pattern is always projected at a certain position, a shift occurs in the three-dimensional position of the user to be observed, that is, the observation position as the user moves, and the lower the spatial resolution of the projected infrared light, Generally, the deviation becomes large. Comparison with reference coordinate values (a reference three-dimensional position distribution of the user's surface for each posture stored in the distribution database 25) relating to a specific shape acquired in advance by this deviation, or between coordinate values obtained at different times The comparison makes it difficult to evaluate the shape change, and the observation position correction process described below is necessary.

  In the present embodiment, observation is performed by irradiating an infrared spot pattern having a high-resolution part for detecting the above-described deviation, and a position suitable for comparison with the reference three-dimensional position distribution according to the deviation detection result An infrared spot pattern for observing the entire observation region centered on the corrected observation position is projected.

  FIG. 9 is a schematic diagram for explaining a coordinate system at the time of observation of the motion instruction apparatus shown in FIG. As shown in FIG. 9, the position on the floor located immediately below the video camera 21a (or 21b) as the observation device is the origin of the world coordinate system, and the horizontal line connecting the origin and the previous observation position of the user is the Y axis, An orthogonal axis that forms a right-handed system together with the Z-axis, the Y-axis, and the Z-axis is defined as the X-axis.

  FIG. 10 is a schematic diagram illustrating an example of an infrared spot pattern, and black circles in the figure indicate the infrared spot pattern. As shown in FIG. 10A, the infrared spot pattern has a spatial resolution in a characteristic part of the human outline, preferably in the horizontal area AR based on the vicinity of the shoulder and the vertical area BR above the center. Has a high part. For example, an infrared spot pattern in which the spatial resolution of the high resolution portion is 2 cm and the spatial resolution of the other portions is 10 cm can be used. At this time, what is obtained from the image captured by the video camera 21a (or 21b) is the three-dimensional position of each observation point in the space, and the change point (x1, x2) of the depth Y in each of the areas AR, BR. , Z1) The boundary position between the person region and the background region can be determined with reference to (the shoulder points of the human body and the top point of the head).

  Here, as shown in FIG. 10B, the three-dimensional position restoration unit 23 determines the midpoint ((x1 + x2) / 2) of x1 and x2 from the restored three-dimensional position distribution as the center line of the human body and z1. It is determined as the height of the head top point of the human body, and an overall infrared spot pattern for observing the entire observation region is generated with reference to the human head position ((x1 + x2) / 2, z1) as the observation position. The projection pattern generation unit 13 is controlled, and a partial infrared spot pattern corresponding to the observation position is projected from the projector 11.

  In this way, the user's three-dimensional position is detected from the three-dimensional position distribution on the user's surface, and infrared light for observing the entire observation region centered on the observation position corrected according to the detected user's three-dimensional position. Since the spot pattern is projected, the three-dimensional position distribution can be detected with high accuracy even if the user moves. Since the user's posture is estimated by comparing the three-dimensional position distribution with the reference three-dimensional position distribution, the user's posture can always be estimated with high accuracy.

  Since the human body has a vertically long shape, the human head position cannot be observed temporarily, and z1 detection may become unstable. However, the horizontal area AR near the shoulder is As long as the center line of the human body can be detected, z1 can be detected again by enlarging the vertical region BR above the center in the vertical direction. For example, as shown in FIG. 9, z1 is detected again by setting the height of the horizontal region AR near the shoulder in the next observation to z1-α (where α> 0, for example, α = 30 cm). be able to.

  Next, a user motion tracking process using the infrared spot pattern corrected as described above will be described. The motion tracking process measures the background including the user, projects the infrared spot pattern to the pre-process for acquiring the user's first observation position, and corrects the observation position to be suitable for the acquired three-dimensional position of the user. A tracking process that measures the three-dimensional position distribution of the user's surface using the infrared spot pattern determined, determines the position of the next user in comparison with the previously detected user position, and repeats these processes. Is done.

  FIG. 11 is a schematic diagram for explaining preprocessing by the operation instruction apparatus shown in FIG. As preprocessing, an infrared spot pattern IP indicated by white circles in FIG. 11 is projected to the user, and the three-dimensional position restoration unit 23 observes and observes the restored three-dimensional position distribution, that is, the three-dimensional shape of the background including the user. By observing each point in the region continuously and randomly, the observation position of the user is intensively detected in the region where the depth changes. In this observation, for example, a block of 25 × 25 pixels can be used as the size of the target block AB. After detecting the user's observation position, the three-dimensional position restoration unit 23 repeatedly measures the user's three-dimensional position distribution, observes the user in a block near the center of the observation area, and displays the vicinity area (horizontal direction) of this position. Intensively search for (placed blocks).

  FIG. 12 is a schematic diagram for explaining the tracking process by the operation instruction apparatus shown in FIG. The three-dimensional position restoration unit 23 outputs the observation position of the user detected by the preprocessing to the projection pattern generation unit 13, and the projection pattern generation unit 13 determines the projection position of the infrared spot pattern according to the detected observation position. Then, the projector 11 irradiates the user with a plurality of spot infrared lights with a new infrared spot pattern.

  At this time, the three-dimensional position restoration unit 23 measures the three-dimensional distribution of the user's surface using the new infrared spot pattern, and the three-dimensional position of each block in the search area SA and the previously detected user's observation position. The point closest to the user's observation position PP detected last time is selected as the next user's observation position NP. By repeating the above tracking process, the user can be tracked accurately.

  Next, the operation instruction process using the infrared spot pattern will be described. FIG. 13 is a flowchart for explaining the operation instruction process by the operation instruction device shown in FIG.

  First, in step S11, the surface of the user and the instrument is irradiated with the infrared spot pattern (three-dimensional dot sequence pattern), and the three-dimensional position detection unit 22 is on the two-dimensional image photographed by the video cameras 21a and 21b. The three-dimensional position distribution of the surface of the user and the instrument is detected from the position of the spot infrared light, and the three-dimensional position restoration unit 23 summarizes the three-dimensional position distribution of the surface of the user and the instrument detected for each partial infrared spot pattern. The three-dimensional position distribution of the surface of the user and the tool with respect to the entire infrared spot pattern is separated and restored, and the three-dimensional position distribution of the surface of the user and the tool is output to the posture estimation unit 24 to Is output to the previous position holding unit 26 and the change site detection unit 27.

  Next, in step S <b> 12, the posture estimation unit 24 compares the three-dimensional position distribution of the user's surface output from the three-dimensional position restoration unit 23 and the reference three-dimensional position distribution of each posture stored in the distribution database 25. The distance is calculated, and the posture with the smallest distance difference is estimated as the current user's posture. Similarly, the posture estimation unit 24 is a distance between the three-dimensional position distribution of the surface of the instrument output from the three-dimensional position restoration unit 23 and the reference three-dimensional position distribution of each use state of the instrument stored in the distribution database 25. And the use state with the smallest distance difference is estimated as the current use state of the appliance.

  Here, Pi is the i-th three-dimensional position of the model P (reference three-dimensional position distribution), and Qj is the j-th point of the current three-dimensional pattern (three-dimensional position distribution output from the three-dimensional position restoring unit 23). Then, the distance between the three-dimensional positions P and Q is expressed by the following formula (10), and in the present embodiment, the above distance is calculated using this formula.

  Next, in step S <b> 13, the clothing change identification unit 28 or the like detects a user's clothing change as described below, and outputs the type of clothing to the motion estimation unit 30. Note that step S13 may be omitted in the motion estimation process in a place where there is no possibility of performing a change-over operation.

  When identifying clothes change, for example, the video camera 21a captures the user from one side (for example, the front side), and the video camera 21b captures the user from the other side (for example, the back side). The three-dimensional position detection unit 22 and the three-dimensional position restoration unit 23 detect and restore the three-dimensional position distribution on the surface of the user in each photographing direction, and output the restored three-dimensional position distribution to the previous position holding unit 26 and the change site detection unit 27. The previous position holding unit 26 holds the three-dimensional position distribution of the user's surface in each shooting direction, and the change site detection unit 27 holds the previous position of the three-dimensional position distribution of the user's surface in each shooting direction a predetermined time ago. The portion where the change in the user's clothes has occurred and the direction of the increase / decrease are identified as compared with the three-dimensional position distribution on the surface of the user in each current photographing direction output from the three-dimensional position restoration unit 23. To the unit 28.

  FIG. 14 is a diagram showing an example of a part stored in the clothing change part database 29 shown in FIG. The first part JA shown in (a) of FIG. 14 is a part that covers the upper half of the human body and is used to identify the outerwear, and the second part PA shown in (b) is a part that covers the lower half of the human body. The third part SA shown in (c) is a part covering the upper half of the lower body of the human body, and is used to specify the skirt. In this way, the part corresponding to each piece of clothing is stored in advance in the clothes change part database 29 in association with the clothes.

  The clothing change identification unit 28 compares the part detected by the change part detection unit 27 with each part stored in the clothes change part database 29, and changes the clothes associated with the part having the largest number of overlapping parts. It is determined as the clothes that produced. In addition, the clothing change identification unit 28 wears the identified clothing because the detected part is inflated when the three-dimensional position increases or decreases based on the increase / decrease direction of the three-dimensional position detected by the change part detection unit 27. If it is decreased, the detected part is reduced, so that it is recognized that the identified clothes are taken off. For example, it is possible to identify the attachment / detachment of clothes according to the increase / decrease of the movement amount in the normal direction of the surface of the object, that is, the increase / decrease of the thickness of the object.

  As described above, in this process, the three-dimensional position distribution of the front and rear surfaces of the user is detected and detected based on the positions of the infrared light spots captured by at least two video cameras 21a and 21b having different shooting directions. By comparing the three-dimensional position distribution of the front and back surfaces of the user with the three-dimensional position distribution of the relevant part before a predetermined time, a region where a change has occurred is detected and recorded for each detected region and each piece of clothing worn by the user. Compared with the parts that change due to the attachment and detachment of the clothes, it is possible to identify the type of clothes that have changed, so that the user wears or removes clothes such as jackets, pants, and skirts. The change of clothes operation can be detected with high accuracy.

  Next, in step S14, the motion estimation unit 30 compares the estimated posture of the user, the usage state of the appliance, and the type of clothing with the posture information, usage status information, and clothing information stored in the motion database 31. To estimate the user's current behavior.

  Next, in step S15, the motion estimation unit 30 determines whether or not an operation instruction is necessary for the operation to be performed by the user after the estimated operation, and when it is determined that no operation instruction is necessary. If the process after step S11 is repeated and it is determined that an operation instruction is necessary, the process proceeds to step 16.

  If it is determined that an operation instruction is necessary, the motion estimation unit 30 identifies the operation that requires the instruction and presents the operation instruction information associated with the operation to the user as an audio output and video display unit. 32, the audio output and video display unit 32 reads the operation instruction information associated with the operation specified by the operation estimation unit 30 from the operation instruction information database 33, and reproduces the read operation instruction information by sound. At the same time, the image is displayed, and then the processing after step S11 is repeated.

  With the above processing, in the present embodiment, the user's motion is estimated by image processing using the captured infrared light spot, and therefore the user is not wearing and not in contact with the observation equipment. Can be estimated and the burden on the user can be reduced. In addition, since the infrared light spot is actively observed by irradiating a plurality of spot infrared lights, the user's operation can be stably performed without being affected by scene-dependent factors such as lighting conditions. Can be estimated. Further, the user's posture and the usage state of the device are estimated by comparing with the reference three-dimensional position distribution stored in advance for each user's posture and the usage state of the device, and the user's operation is estimated based on these. Therefore, the user's operation can be estimated with high accuracy.

  Next, the detection accuracy of the operation instruction device shown in FIG. 1 will be described. FIG. 15 is a diagram illustrating a measurement result of detection accuracy of the operation instruction device illustrated in FIG. 1. In this measurement, the three-dimensional shape of the actual object was measured using the motion instruction device shown in FIG. 1, and the average value (Ave.) and standard deviation (SD) of the error from the actual shape were measured. . From FIG. 15, it was found that the motion instruction apparatus shown in FIG. 1 can accurately measure the three-dimensional position with an accuracy of 10 cm or less in the X direction, the Y direction, and the Z direction.

  Next, an actual human three-dimensional shape was reconstructed using the motion instruction apparatus shown in FIG. FIG. 16 is a diagram illustrating an example of reconstruction of a three-dimensional shape by the operation instruction apparatus illustrated in FIG. As a result of measuring the three-dimensional shape of the person shown in FIG. 16A and reconstructing the three-dimensional shape from the measurement result, the three-dimensional position distribution shown in FIG. It was found that the dimensional shape can be accurately reconstructed.

  Next, a moving person was tracked using the operation instruction device shown in FIG. FIG. 17 is a diagram showing a human trajectory tracked by using the motion instruction device shown in FIG. From the human trajectory shown in FIG. 17, it was found that the human position XY on the floor can be accurately tracked.

  Next, the human posture was estimated using the motion instruction apparatus shown in FIG. 18 is a diagram illustrating a result of detecting a three-dimensional position distribution for eight different types of human poses using the motion instruction device illustrated in FIG. 1, and FIG. 19 is a diagram illustrating the motion instruction device illustrated in FIG. It is a figure which shows the result of having estimated eight types of different postures of humans.

  18A shows a posture STF in which a human stands upright and faces forward, FIG. 18B shows a posture STL in which a human stands upright and faces left, and FIG. 18C shows a posture STF. Shows a posture STR that stands upright and faces right, (d) shows a posture SIF in which a human is seated and faces forward, and (e) shows a posture in which a human is seated and faces left. (F) shows a posture SIR in which a human is seated and faces right, (g) shows a posture DR in which a human is dressed, and (h) shows a human being Indicates postures RE in which a hand is put out, each upper row is an example of an image showing each posture of a human, and the lower row is an example of a three-dimensional position distribution detected for each posture. Moreover, the result shown in FIG. 19 is a result of estimating each posture with respect to 48 samples, and indicates a recognition ratio (%) for each posture. 18 and 19 that each posture can be estimated with high accuracy.

  Finally, the human posture was continuously observed using the motion instruction device shown in FIG. FIG. 20 is a diagram illustrating a result of continuously estimating a human posture using the motion instruction device illustrated in FIG. 1. In FIG. 20, the horizontal axis indicates time (frame), and the vertical axis indicates the estimated posture (solid line) and the actual posture (broken line) in each frame. From FIG. 20, it was found that each posture can be estimated with high accuracy continuously even when a human is continuously acting.

  In this way, in this embodiment, since the movement of the care recipient who is the user can be tracked and the movement can be recognized, the posture of the user in daily life (changing clothes, using the toilet, bathing, washing the face, etc.) Can be estimated to estimate the user's movement, and an appropriate instruction can be given to the user based on the estimated user's movement.

  For example, people with brain disorders such as encephalitis, head wounds, subarachnoid hemorrhage, dementia, cerebellar vascular disorders often have memory impairment, and people with this or other severe brain disorders It is difficult to send, and constant care and attention to these people is a family burden. By using this operation instruction device for such a memory handicapped person, it is possible to automatically detect the intention of the memory handicapped person and support daily life. In addition, when network-type interaction treatment is performed using this operation instruction device, communication with the caregiver is facilitated and the family can obtain appropriate information. Before the family experiences, it is possible to provide comfortable information to the family and to release the stress of the family and reduce the burden.

  In this embodiment, the user's movement is estimated using the user's posture as well as the usage state of the appliance. However, the present invention is not particularly limited to this example, and the user's movement is estimated using only the user's posture. Various modifications such as these are possible. Further, as the user's three-dimensional shape, the three-dimensional position distribution on the user's surface is detected using the projector 11 and the video cameras 21a, 21b, etc., but the present invention is not particularly limited to this example, and the user is detected using another detection device. The three-dimensional shape may be detected. Moreover, although the use state of the appliance was estimated using the projector 11 and the video cameras 21a and 21b, etc., it is not particularly limited to this example, and various changes such as detection of opening / closing of the toilet lid using a sensor or the like are possible. Is possible. Moreover, although infrared light was used as light to project, it is not specifically limited to this example, Light other than natural light and light having other wavelengths may be used.

It is a block diagram which shows the structure of the operation | movement instruction | indication apparatus by one embodiment of this invention. It is a schematic diagram for demonstrating the positional relationship of the projector and video camera which are shown in FIG. It is a figure which shows the example of installation of the projector and video camera which are shown in FIG. It is a figure which shows the other example of installation of the projector and video camera which are shown in FIG. It is a figure for demonstrating an example of the procedure P which produces a partial infrared spot pattern. It is a figure for demonstrating in detail the procedure P which produces | generates the matrix C shown to (b) of FIG. 5 from the matrix A shown to (a) of FIG. It is a figure which shows an example of the edge detection filter used when detecting a user's outline. It is a figure which shows an example of the detection result of an infrared-light spot. It is a schematic diagram for demonstrating the coordinate system at the time of observation of the operation | movement instruction | indication apparatus shown in FIG. It is a schematic diagram which shows an example of an infrared spot pattern. It is a schematic diagram for demonstrating the pre-processing by the operation | movement instruction | indication apparatus shown in FIG. It is a schematic diagram for demonstrating the tracking process by the operation instruction apparatus shown in FIG. It is a flowchart for demonstrating the operation instruction | indication process by the operation instruction | indication apparatus shown in FIG. It is a figure which shows an example of the site | part memorize | stored in the clothing change site | part database shown in FIG. It is a figure which shows the measurement result of the detection accuracy of the operation instruction | indication apparatus shown in FIG. It is a figure which shows the reconstruction example of the three-dimensional shape by the operation | movement instruction | indication apparatus shown in FIG. It is a figure which shows the human locus | trajectory tracked using the operation | movement instruction | indication apparatus shown in FIG. It is a figure which shows the result of having detected the three-dimensional position distribution with respect to eight different postures of a human using the operation | movement instruction | indication apparatus shown in FIG. It is a figure which shows the result of having estimated eight types of different postures of the human using the motion instruction device shown in FIG. It is a figure which shows the result of having estimated the human posture continuously using the operation instruction device shown in FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 Projector 12 Projection pattern division | segmentation part 13 Projection pattern production | generation part 21a, 21b Video camera 22 3D position detection part 23 3D position restoration part 24 Posture estimation part 25 Distribution database 26 Previous position holding part 27 Change part detection part 28 Clothes change identification Unit 29 Clothing change part database 30 Motion estimation unit 31 Motion database 32 Audio output and video display unit 33 Motion instruction information database

Claims (7)

Detecting means for detecting the three-dimensional shape of the user without contact with the user;
Estimating means for estimating the user's movement based on the three-dimensional shape of the user detected by the detecting means;
Presenting means for presenting to the user an instruction regarding the next action to be performed by the user based on the user's action estimated by the estimating means ;
The detection means includes
Projecting means for projecting a plurality of spot lights in a predetermined pattern to the user;
Photographing means for photographing a plurality of light spots projected on the surface of the user by the projection means;
Position detection means for detecting a three-dimensional position distribution on the surface of the user based on the positions of a plurality of light spots photographed by the photographing means as the three-dimensional shape of the user;
The projection means includes
A dividing unit that divides a plurality of spot lights into spot light groups that do not interfere with each other on a captured image;
Divided projection means for projecting spot light to the user for each spot light group divided by the dividing means,
The photographing means photographs a plurality of light spots projected on the surface of the user for each spot light group by the divided projecting means,
The position detection means detects a three-dimensional position distribution of a user's surface based on the position of a light spot photographed for each spot light group by the photographing means . The estimation means includes
Distribution storage means for storing in advance a reference three-dimensional position distribution of the user's surface for each user posture;
Posture estimation means for comparing the three-dimensional position distribution detected by the position detection means and the reference three-dimensional position distribution stored in the distribution storage means to estimate the current posture of the user;
Action storage means for storing in advance posture information representing the posture of the user constituting the action for each user action;
Claim 1, characterized in that it comprises a motion estimation means for estimating a current operation of the user by comparing the orientation information stored in the posture and the operation storage unit of the user estimated by the position estimation means The operation instruction device described. The distribution storage means further stores in advance a reference three-dimensional position distribution at the time of use of the appliance for each appliance used by the user,
The posture estimation means further estimates the use state of the appliance by comparing the three-dimensional position distribution of the appliance detected by the position detection means with the reference three-dimensional position distribution of the appliance stored in the distribution storage means. And
The operation storage means further stores in advance usage state information representing the usage state of the appliance associated with the operation for each user operation,
The motion estimation unit estimates the current motion of the user by comparing the posture of the user and the usage state of the appliance estimated by the posture estimation unit with the posture information and the usage state information stored in the motion storage unit. The operation instruction device according to claim 2, wherein: The position detection means detects a user's observation position based on the detected three-dimensional position distribution,
It said projection means, the operation instruction apparatus according to any one of claims 2-3, characterized in that to correct the position of the spot light in accordance with the observation position of the user detected by the position detecting means. The photographing means includes first and second photographing means for photographing a plurality of light spots projected on the surface of the user by the projecting means from different photographing directions,
The position detecting means detects a three-dimensional position distribution of different surfaces of the user based on the positions of a plurality of light spots imaged by the first and second imaging means;
Holding means for holding a three-dimensional position distribution of different surfaces of the user detected by the position detecting means;
The three-dimensional position distribution of the different surface of the user detected by the position detecting means is compared with the three-dimensional position distribution of the different surface of the user held by the holding means before a predetermined time, and the user's surface changes. Site detection means for detecting the site that has occurred,
Clothing storage means for storing in advance a site where a change occurs on the surface of the user due to attachment / detachment of the clothing for each clothing worn by the user;
An identification means for comparing the part detected by the part detection means and the part stored in the clothes storage means to identify the type of clothes that have changed,
The motion instruction device according to any one of claims 2 to 4 , wherein the motion estimation unit estimates a user's dressing motion in consideration of the type of clothes identified by the identification unit. The projection means projects a plurality of spot lights in a predetermined pattern to the user using infrared light,
The imaging means, the operation instruction apparatus according to any one of claims 1 to 5, characterized in that to shoot a plurality of infrared light spot projected on the surface of the user by the projection means. It said presenting means any of claims 1-6, characterized in that direct the operation which a user should do next based on the operation of the user detected by the detecting means to the user by the display of the audio output or video The operation instruction device according to any one of the above.

Images