JP2017059088A - Calibration information creation apparatus, calibration information creation method, calibration information creation program, attitude detection apparatus, attitude detection method, attitude detection program, and information processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology for simple and quick calibration between a hand, as a real object, and a virtual video.SOLUTION: A calibration information creation apparatus includes: a feature point detection unit which detects one or a plurality of feature points from a video obtained by imaging a finger after a stimulus based on actual attitude data of the finger is presented to the finger; and a calibration information creation unit which creates calibration information for calibrating a virtual video, on the basis of the one or a plurality of feature points.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a calibration information creation apparatus, a calibration information creation method, a calibration information creation program, an attitude detection apparatus, an attitude detection method, an attitude detection program, and an information processing system.

  In recent years, a mixed reality system that provides a synthesized video obtained by synthesizing a real space video and a virtual space video to a user's field of view has been disclosed (for example, see Patent Document 1). For example, the virtual space image is generated according to the viewpoint position (or line-of-sight direction) of the user. According to the mixed reality system, it is possible to provide the user with the sensation that a virtual image actually exists in the real space, and to provide the user with observations that are more realistic than the virtual reality system. Is possible.

  Mixed reality systems are applied in various fields. As an example, in a design / manufacturing field, a virtual image displayed in a virtual space exists in a real space as a method for verifying the operability, maintainability, and assemblability of an object designed by 3D CAD (computer aided design). There is a method of operating with a hand as a real object. With this method, it is possible to allow the user to operate and verify the object while giving the user a feeling as if he / she is directly touching the object being prototyped with his / her hand.

  In order to realize such a method, it is necessary to calibrate the parameters of the virtual video (for example, the position, orientation, magnification factor, etc. of the virtual video) so as to match the hand as a real object. As described above, in order to calibrate the parameters of the virtual video so as to match the hand as the real object, it is necessary to execute the calibration to acquire the calibration degree of the parameter of the virtual video with respect to the real object as calibration information in advance. is there.

JP 2002-159019 A

  However, in order to accurately perform calibration between the hand as the real object and the virtual image, the size of the entire hand as the real object, the width / height of the palm as the real object, the hand as the real object It is necessary to consider many factors such as finger length. Further, in the calibration between a virtual image and an object having a complicated and flexible shape such as a hand as a real object, it is necessary to consider the deformation of the hand shape.

  In general, an apparatus that can specify an arbitrary point in space (for example, a mouse, a three-dimensional pointing device, etc.) is used to specify all feature points for calibrating the magnification. Therefore, a method for calibrating the magnification is often used. However, with such a method, calibration takes a long time, and the convenience of the system interface may be impaired.

  Accordingly, the present invention has been made in view of the above problems, and an object of the present invention is to enable simple and quick calibration between a hand as a real object and a virtual image. To provide technology.

  In order to solve the above problem, according to an aspect of the present invention, one or a plurality of feature points from an imaged image in which the finger is imaged after a stimulus corresponding to actual finger posture data is presented to the finger. There is provided a calibration information creation device comprising a feature point detection unit for detecting a calibration point and a calibration information creation unit for creating calibration information for calibrating a virtual image based on the one or more feature points.

  The calibration information creation device may include a display conversion unit that obtains a presentation video by converting the virtual video based on the calibration information.

  The calibration information creation apparatus may include a video generation unit that generates the virtual video.

  The calibration information creating apparatus may include a video display unit that displays the presentation video.

  The feature point detection unit may detect the one or more feature points when an absolute difference between the actual posture data and the ideal posture data of the finger falls below a threshold value.

  According to another aspect of the present invention, one or a plurality of feature points are detected from a captured image in which the finger is imaged after a stimulus corresponding to the actual posture data of the finger is presented to the finger. And creating calibration information for calibrating a virtual image based on the one or more feature points.

  According to another aspect of the present invention, the computer detects one or a plurality of feature points from a captured image obtained by capturing the finger after a stimulus corresponding to the actual posture data of the finger is presented to the finger. Calibration information creation for functioning as a calibration information creation device comprising: a feature point detection unit that performs calibration information creation unit that creates calibration information for calibrating a virtual image based on the one or more feature points A program is provided.

  Further, according to another aspect of the present invention, based on a finger posture detection unit that detects actual posture data of a finger, the actual posture data, and the ideal posture data of the finger determined in advance, There is provided a posture detection device including a stimulus information control unit that controls a stimulus presented to a finger.

  The stimulus information control unit may control the stimulus based on a difference between the actual posture data and the ideal posture data.

  The stimulus information control unit may control the position of the stimulus presented to the finger based on the sign of the difference.

  The stimulus information control unit may change the position of the stimulus presented to the finger as time passes.

  The stimulus information control unit may control the magnitude of the stimulus presented to the finger based on the absolute value of the difference.

  The posture detection apparatus may include a stimulus information presentation unit that presents a stimulus to the finger.

  The finger posture detection unit may detect the actual posture data by contact or non-contact, and the stimulus information control unit may control the stimulus presented by contact or non-contact.

  One or a plurality of feature points are detected from a captured image obtained by capturing the finger after the stimulus is presented to the finger, and calibration information for calibrating a virtual image based on the one or a plurality of feature points is provided. It may be created.

  According to another aspect of the present invention, the actual posture data of the finger is detected and presented to the finger based on the actual posture data and the predetermined ideal posture data of the finger. Controlling a stimulus to be provided.

  According to another aspect of the present invention, the computer is based on a finger posture detection unit that detects actual finger posture data, the actual posture data, and the predetermined ideal finger posture data. There is provided a posture detection program for functioning as a posture detection device including a stimulus information control unit that controls a stimulus presented to the finger.

  Further, according to another aspect of the present invention, based on a finger posture detection unit that detects actual posture data of a finger, the actual posture data, and the ideal posture data of the finger determined in advance, A posture detection device comprising: a stimulus information control unit that controls a stimulus presented to the finger; and one or more feature points are detected from a captured image obtained by imaging the finger after the stimulus is presented to the finger A calibration information creation device, comprising: a feature point detection unit that performs calibration information creation unit that creates calibration information for calibrating a virtual image based on the one or more feature points. A system is provided.

  As described above, according to the present invention, it is possible to easily and quickly perform calibration between a hand as a real object and a virtual image.

It is a figure for demonstrating the outline | summary of the 1st Embodiment of this invention. It is a block diagram showing an example of composition of an information processing system concerning the embodiment. It is a figure for demonstrating the example which detects the attitude | position of a user's real index finger. It is a figure which shows the state in which the contact stimulus information presentation part is attached to two places of a finger | toe. It is a flowchart which shows the flow of operation | movement of the information processing system which concerns on the embodiment. It is a figure for demonstrating the ideal attitude | position data of each finger | toe. It is a figure for demonstrating the ideal attitude | position data of each finger | toe. It is a figure for demonstrating the ideal attitude | position data of each finger | toe. It is a figure for demonstrating the ideal attitude | position data of each finger | toe. It is a figure for demonstrating the ideal attitude | position data of each finger | toe. It is a block diagram which shows the structural example of the information processing system which concerns on the 2nd Embodiment of this invention. It is a figure showing an example of composition of an information processing system concerning the embodiment. It is a figure for demonstrating the outline of the technique which detects the real attitude | position of a finger | toe by non-contact. It is a figure for demonstrating the outline of the technique which detects the real attitude | position of a finger | toe by non-contact. It is a figure for demonstrating the outline of the technique which detects the real attitude | position of a finger | toe by non-contact. It is a flowchart which shows the flow of operation | movement of the information processing system which concerns on the embodiment. It is a figure for demonstrating the production | generation of a tactile sense presentation pattern. It is a figure for demonstrating the example which changes the position of the stimulus shown to a finger with progress of time.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

  In the present specification and drawings, a plurality of constituent elements having substantially the same functional configuration are distinguished by attaching different numerals to the same reference numerals. Further, similar constituent elements of different embodiments are distinguished by attaching different alphabets after the same reference numerals. However, when there is no need to particularly distinguish each of a plurality of constituent elements having substantially the same functional configuration, only the same reference numerals are given.

[1-1. Overview of First Embodiment]
First, an outline of the first embodiment of the present invention will be described. FIG. 1 is a diagram for explaining the outline of the first embodiment of the present invention. In the first embodiment, as shown in FIG. 1, there is a calibration information creation device 100. FIG. 1 shows an example in which the calibration information creation device 100 is a glass-type wearable device having a video acquisition unit 110 and a video display unit 120 and is worn by a user U. The form of 100 is not limited to such a form.

  For example, the proofreading information creation apparatus 100 is not a glass-type wearable device, but may be another mobile terminal (for example, a smartphone, a tablet terminal, a mobile phone, etc.), or a PC (Personal Computer). . FIG. 1 shows an example in which the video acquisition unit 110 and the video display unit 120 exist inside the calibration information creation device 100. The video acquisition unit 110 and the video display unit 120 are not included in the calibration information creation device. 100 may exist outside.

  As shown in FIG. 1, when the image acquisition unit 110 captures the user's hand Ha and acquires the captured image of the hand Ha, the image display unit 120 superimposes the virtual image Vi on the captured image. In the first embodiment, an example in which the virtual video Vi is superimposed on the captured video will be mainly described. However, when the video display unit 120 is a see-through wearable device, the real space is compared. Thus, the virtual video Vi may be superimposed. That is, the video display unit 120 superimposes the virtual video Vi on the field of view of the user U. 1 shows an example in which the virtual video Vi is an image of a ball, the virtual video Vi may be an image of an object other than the ball.

  Here, in order for the video display unit 120 to superimpose the virtual video Vi on the captured video, parameters of the virtual video Vi (for example, the position, orientation, and magnification of the virtual video Vi so as to match the hand Ha as a real object). Etc.) must be calibrated. In order to calibrate the parameters of the virtual video Vi so as to match the hand Ha as the real object, calibration is executed in advance to acquire the calibration degree of the parameters of the virtual video Vi for the hand Ha as the real object as calibration information. It is necessary to

  However, in order to accurately perform calibration between the hand Ha as the real object and the virtual image Vi, the overall size of the hand Ha as the real object and the width and height of the flatness of the hand Ha as the real object It is necessary to consider many factors such as the length of the finger of the hand Ha as a real object. Furthermore, in the calibration between the virtual image Vi and an object having a complicated and flexible shape such as the hand Ha as a real object, it is necessary to consider deformation of the hand shape.

  In general, an apparatus that can specify an arbitrary point in space (for example, a mouse, a three-dimensional pointing device, etc.) is used to specify all feature points for calibrating the magnification. Therefore, a method for calibrating the magnification is often used. However, with such a method, calibration takes a long time, and the convenience of the system interface may be impaired.

  In the first embodiment, the number of users may be one, but it is also assumed that the system is sequentially used by a plurality of users. In such a case, since it is necessary to newly execute calibration every time the user who uses the system changes, it is desired that the calibration be performed particularly quickly. Therefore, the present specification mainly proposes a technique that can easily and quickly perform calibration between the hand Ha as a real object and the virtual image Vi.

  As such a technique, in the first embodiment, an example will be described in which the user U wears a contact stimulus information presentation unit 230 that presents a stimulus to the user U's hand by contact. In the first embodiment, an example will be described in which the contact stimulus information presentation unit 230 is a vibrator configured by an eccentric motor or the like, and the stimulus presented by contact to the user U's hand is a stimulus by vibration. . However, the type of stimulus presented to the user U's hand is not particularly limited. For example, the stimulus presented to the user U's hand may be an electrical stimulus or the like.

  FIG. 1 shows an example in which the contact stimulus information presentation unit 230 is attached to a glove and the user U wears the glove. However, the contact stimulus information presentation unit 230 may be attached to a substance other than gloves (for example, a ring), and the user U may wear the substance. Alternatively, the contact stimulus information presentation unit 230 may be directly attached to the user U's hand. In such a case, the contact stimulus information presentation unit 230 may be attached to the user U's hand, or may be wound around the user U's hand with a tape or the like.

  The outline of the first embodiment of the present invention has been described above.

[1-2. System configuration example]
Next, a configuration example of the information processing system according to the first embodiment of the present invention will be described. FIG. 2 is a block diagram illustrating a configuration example of the information processing system according to the first embodiment of the present invention. As shown in FIG. 2, the information processing system 10A according to the first embodiment of the present invention includes a calibration information creation device 100 and an attitude detection device 200A.

  As shown in FIG. 2, the calibration information creation apparatus 100 includes a video acquisition unit 110, a video display unit 120, a video generation unit 130, a display conversion unit 140, a feature point detection unit 150, and calibration information creation. Unit 160 and a calibration information storage unit 170. As illustrated in FIG. 2, the posture detection device 200 </ b> A includes a contact finger posture detection unit 210, a stimulus information control unit 220, and a contact stimulus information presentation unit 230.

  The contact finger posture detection unit 210 is an example of a finger posture detection unit that detects the actual posture of the finger, and detects the actual posture of the finger by contact. The contact finger posture detection unit 210 includes a predetermined sensor that can detect the actual posture of the finger by contact. Here, an example will be described in which the contact finger posture detection unit 210 detects the actual posture of the finger based on the joint bending angle detected by the bending sensor attached to the finger. However, the contact finger posture detection unit 210 may detect the actual posture of the finger based on the acceleration detected by the acceleration sensor attached to the finger.

  FIG. 3 is a diagram for explaining an example of detecting the actual posture of the index finger of the user. FIG. 3 shows an example in which the contact finger posture detection unit 210 detects the bending angle at the root joint of the user's index finger for the sake of simplicity. However, the kind of finger whose bending angle is detected is not limited to the index finger, and the joint whose bending angle is detected is not limited to the root joint. Further, FIG. 3 shows an example in which the bending angle of one joint in one finger is detected, but the number of fingers whose bending angle is detected may be plural, or the bending angle is detected 1. The number of joints per finger may be plural.

  Referring to FIG. 3, a hand Ha as a real object is shown. Further, the root joint of the index finger is set as the origin, the direction toward the tip of the thumb is set as the X axis, the direction toward the palm side is set as the Y axis, and the direction toward the tip of the index finger is set as the Z axis, (dx, dy, dz) The contact finger posture detection unit 210 detects the actual posture data of the index finger based on the three-dimensional vector normalized to length 1. Note that actual posture data of fingers other than the index finger can be detected in the same manner.

  As an example, in the example shown in FIG. 3, it is assumed that the finger is rotated by −45 degrees in the YZ plane while maintaining the hand shape from the state where the finger extends in the Z-axis direction. In such a case, a three-dimensional vector (dx, dy, dz) indicating the actual posture of the finger is expressed as the following formula (Formula 1).

・・・（数式１）

... (Formula 1)

  The stimulus information control unit 220 controls the contact stimulus information presentation unit 230 so that a stimulus corresponding to the actual posture data of the finger is presented to the finger. More specifically, the stimulus information control unit 220 controls the stimulus presented to the finger based on the actual finger posture data and predetermined ideal finger posture data. The ideal posture data can also be expressed by a three-dimensional vector in the same way as actual posture data. Further, for example, when the ideal posture data of the finger is held by the calibration information creation unit 160, it may be acquired from the calibration information creation unit 160. The stimulus is presented to the finger by the contact stimulus information presentation unit 230 according to the control by the stimulus information control unit 220.

  FIG. 4 is a diagram illustrating a state in which the contact stimulus information presentation unit 230 is attached to two positions of the finger. Referring to FIG. 4, a contact stimulus information presentation unit 230-1 is attached to the back side of the hand of the finger, and a contact stimulus information presentation unit 230-2 is attached to the palm side of the finger. In addition, although the example in which the contact stimulus information presentation part 230 is attached to two places of a finger | toe is shown in FIG. 4, the number and position in which the contact stimulus information presentation part 230 is attached are not limited. The finger to which the contact stimulus information presentation unit 230 is attached may be the same finger as the contact finger posture detection unit 210.

  The image acquisition unit 110 includes an imaging device, and acquires a captured image of a finger by the imaging device. When the stimulus to the finger is presented by the contact stimulus information presentation unit 230, the user changes the posture of the finger according to the stimulus. After the finger posture changes, it is expected that the actual posture of the finger approaches the ideal posture of the finger. Therefore, the feature point detection unit 150 detects one or a plurality of feature points from the captured video in which the finger is captured by the video acquisition unit 110 after a stimulus according to the finger's actual posture data is presented to the finger.

  The one or more feature points detected by the feature point detection unit 150 may be any point, but one or more feature points (for example, the base of the finger, the fingertip) that can be easily used to create calibration information. Etc.). Various methods can be employed as the feature point detection method. For example, the feature point detection unit 150 may detect the feature points using a technique such as fingertip detection by corner detection.

  Alternatively, the feature point detection unit 150 uses local feature quantities such as SIFT (Scale-Invariant Feature Transform), SURF (Speed-Upd Robust Feature), HOG (Histograms of Oriented Gradients) and the like. A point corresponding to a fingertip portion or a finger base portion having a feature amount registered in advance by learning may be detected, or another feature amount may be used.

  In addition, as described above, after the finger posture is changed, it is expected that the actual posture of the finger is close to the ideal posture of the finger. It may be more reliably grasped that the finger is approaching the ideal posture. Therefore, it is preferable that the feature point detection unit 150 detects one or a plurality of feature points when the absolute difference value between the actual posture data of the finger and the ideal posture data of the finger falls below a threshold value.

  The calibration information creation unit 160 creates calibration information for calibrating the virtual image based on one or more feature points detected by the feature point detection unit 150. The calibration information indicates the degree of calibration of virtual video parameters (for example, the position, posture, magnification factor, etc.) of the virtual video for the hand as a real object. According to such a configuration, calibration between a hand as a real object and a virtual image can be performed easily and quickly. The calibration information created by the calibration information creation unit 160 is stored in the calibration information storage unit 170.

  The video generation unit 130 generates a virtual video. Here, the video generation unit 130 may generate a virtual video by acquiring a virtual video stored in advance, or may generate a virtual video by computer graphics or the like. The display conversion unit 140 obtains a presentation video by converting the virtual video based on the calibration information stored in the calibration information storage unit 170. The video display unit 120 displays a presentation video.

  The configuration example of the information processing system 10A according to the first embodiment of the present invention has been described above.

[1-3. Example of operation]
Then, the example of the flow of operation | movement of 10 A of information processing systems which concern on the 1st Embodiment of this invention is demonstrated. FIG. 5 is a flowchart showing an operation flow of the information processing system 10A according to the first embodiment of the present invention. The flowchart shown in FIG. 5 merely shows an example of the operation flow of the information processing system 10A. Therefore, the operation of the information processing system 10A is not limited to the example shown in FIG. Here, an example will be described in which ideal posture data for each of five fingers is registered in advance before the calibration process is started.

  The ideal posture data for each of the five fingers can be expressed by the three-dimensional vector (dx, dy, dz) as described above, but may be any specific posture data. For example, the ideal posture data may be data indicating a posture in which a predetermined feature point of each finger is hidden. The ideal posture data will be specifically described below. 6A to 6E are diagrams for explaining ideal posture data of each finger. As shown in FIGS. 6A to 6E, feature points Fp exist at the tips and roots of the five fingers.

  In the example shown in FIG. 6A, the two feature points Fp of the thumb are hidden by bending only the thumb. The thumb posture at this time may be registered as ideal posture data of the thumb. Similarly, in the example shown in FIG. 6B, only the index finger is bent, so that the two characteristic points Fp of the index finger are concealed. The posture of the index finger at this time may be registered as ideal posture data of the index finger. Similarly, in the example shown in FIG. 6C, only the middle finger is bent, so that the two feature points Fp of the middle finger are concealed. At this time, the posture of the middle finger may be registered as ideal posture data of the middle finger.

  Similarly, in the example shown in FIG. 6D, only the ring finger is bent, so that the two feature points Fp of the ring finger are concealed. The posture of the ring finger at this time may be registered as ideal posture data of the ring finger. Similarly, in the example shown in FIG. 6E, only the little finger is bent, so that the two feature points Fp of the little finger are concealed. The posture of the little finger at this time may be registered as ideal posture data of the little finger. In this way, the ideal posture data of each of the five fingers is registered as five sets of three-dimensional vectors (dx, dy, dz).

  As described above, when the ideal posture data is data indicating a posture in which the two feature points Fp of each finger are concealed, the data is detected from the captured image when the actual posture approaches the ideal posture. Two feature points Fp of the finger that is bent are excluded from the feature points. Therefore, the correspondence between the finger to be bent and the detected feature point becomes clear. Therefore, it can be said that it is preferable to use such ideal posture data in order to create calibration information.

  Here, for convenience of explanation, it is assumed that each finger is bent at the root joint, but the number and positions of the joints bent for each finger are not particularly limited. For example, the position of the joint that is bent for each finger may be the position of a joint other than the root joint. Also, for example, there may be a plurality of joints that can be bent for each finger. In such a case, five sets of three-dimensional vectors (dx, dy, dz) need only be registered in a number corresponding to the number of joints.

  For example, when the calibration process is started by a predetermined start operation by the user (step S501), the calibration information creating unit 160 (when the ideal posture data registered in advance does not remain ( In step S503, “No”), the operation is shifted to step S525. On the other hand, when the ideal posture data registered in advance still remains (“Yes” in step S502), the calibration information creation unit 160 selects one ideal posture data (step S505).

  Subsequently, ideal posture data is transmitted from the calibration information creation unit 160 to the stimulus information control unit 220 (step S507). When the ideal posture data is received by the stimulus information control unit 220, the contact finger posture detection unit 210 acquires the actual posture data of the finger by touching (step S509). Subsequently, the stimulus information control unit 220 calculates a difference between the ideal posture data and the actual posture data (difference between posture data) (step S511). Here, an example of a method for calculating a difference between posture data will be described.

  Here, as an example, a case where a stimulus is presented to the index finger will be described as described with reference to FIG. 4, but a stimulus to another finger may be presented in the same manner as a stimulus to the index finger. For example, when the ideal posture data of the finger is expressed by a three-dimensional vector (rx, ry, rz), the ideal posture data R1 of the finger based on the Z-axis direction in the ZY plane is expressed by the following (Equation 2). It is expressed as

・・・（数式２）

... (Formula 2)

  For example, when the actual posture data of the finger is expressed by a three-dimensional vector (0, dy, dz), the actual posture data R2 of the finger based on the Z-axis direction in the ZY plane is expressed by the following (formula It is expressed as 3).

・・・（数式３）

... (Formula 3)

  Accordingly, the difference dR between the ideal posture data R1 of the finger and the actual posture data R2 of the finger is expressed as the following (Formula 4).

・・・（数式４）

... (Formula 4)

  Subsequently, the stimulus information control unit 220 controls whether to present a stimulus to the finger based on the relationship between the absolute value of the difference dR and the threshold value. Specifically, if the absolute value of the difference dR is equal to or greater than the threshold (“≧” in step S513), the stimulus information control unit 220 determines that the actual posture needs to be close to the ideal posture, and the stimulus The finger part to be presented with information is determined (step S514). Although the finger part for which the stimulus information is to be presented may be determined in any way, for example, it may be determined based on the sign of the difference dR.

  For example, when the difference dR is positive, in order to bring the actual posture closer to the ideal posture, the actual posture of the finger in the positive direction on the ZY plane (when the Z axis is the horizontal axis and the Y axis is the vertical axis) Must be rotated (counterclockwise). Therefore, when the difference dR is positive, the stimulus information control unit 220 is a contact that exists in the negative direction with reference to the Z axis on the ZY plane in order to urge the actual posture of the finger to rotate in the positive direction. Stimulation information presentation unit 230-1 may be determined as a finger part to be presented with stimulation information.

  On the other hand, when the difference dR is negative, in order to bring the actual posture closer to the ideal posture, the actual posture of the finger in the negative direction on the ZY plane (when the Z axis is the horizontal axis and the Y axis is the vertical axis) Must be rotated (clockwise). Therefore, when the difference dR is negative, the stimulus information control unit 220 is a contact that exists in the positive direction with reference to the Z axis on the ZY plane in order to urge the actual posture of the finger to rotate in the negative direction. The stimulation information presentation unit 230-2 may be determined as a finger part for which stimulation information is to be presented.

  Further, the magnitude of the stimulus presented to the finger may be constant, but may be changed according to the difference dR in order to make the user know how much the difference dR is. That is, the stimulus information control unit 220 may control the magnitude of the stimulus presented to the finger based on the absolute value of the difference dR. The stimulus information control unit 220 can intuitively make the user know how much the difference dR is when the stimulus presented to the finger is increased as the absolute value of the difference dR is larger.

  Note that, as described in the second embodiment, the stimulus information control unit 220 may change the position of the stimulus presented to the finger as time passes. For example, the stimulus information control unit 220 may change the position of the stimulus from the far side to the near side with reference to the ideal posture of the finger. In this way, by changing the position of the stimulus presented to the finger with the passage of time, it becomes possible for the user to grasp the ideal posture of the finger.

  Subsequently, the contact stimulus information presentation unit 230 determined as the finger part to be presented with the stimulus information generates a vibration pattern (step S515), and presents the stimulus based on the generated vibration pattern on the finger. When the stimulus is presented to the finger, the user changes the posture of the finger according to the vibration pattern (step S517). It is expected that the actual posture of the finger approaches the ideal posture by changing the posture of the finger. The timing to stop the stimulation is not limited, but when the absolute value of the difference dR becomes less than the threshold, the stimulation may be stopped immediately, or the stimulation is stopped after the operation for the next ideal posture data starts. May be.

  On the other hand, if the absolute value of the difference dR is less than the threshold value (“<” in step S513), the stimulus information control unit 220 determines that the actual posture has sufficiently approached the ideal posture, and the video acquisition unit 110 The captured image of the user's hand is acquired (step S519). Subsequently, the feature point detection unit 150 extracts one or more feature points from the captured image of the user's hand (step S521), and stores the extracted one or more feature points in association with ideal posture data of the finger. (Step S523). For feature point extraction, a feature point group previously associated with ideal posture data is referred to. Thereafter, the operation proceeds to step S503.

  When the operation is shifted to step S503 and the ideal posture data registered in advance remains (step S503: Yes), the operation of step S505 has already been described. Executed. In addition, when there is no pre-registered ideal posture data (“No” in step S503), the calibration information creation unit 160 calculates calibration information (for example, hand size, The position of the finger joint, etc.) is created (step S525), and the calibration process is terminated (step S527).

  After the calibration process is completed, the video generation unit 130 generates a virtual video, and the display conversion unit 140 presents the virtual video based on the calibration information created by the calibration information creation unit 160. Get a picture. The video display unit 120 displays the presentation video obtained by the display conversion unit 140. As a method for converting a virtual video based on the calibration information, a technique that has already been disclosed can be used (for example, Japanese Patent Application Laid-Open No. 2014-106642).

  The example of the operation flow of the information processing system 10A according to the first embodiment of the present invention has been described above.

[1-4. Summary of First Embodiment]
According to the first embodiment of the present invention, the feature point detection unit 150 includes one or a plurality of captured images obtained by capturing a finger after a stimulus by contact according to the actual posture data of the finger is presented to the finger. Detect feature points. When a stimulus to the finger is presented, the user changes the posture of the finger according to the stimulus. After the finger posture changes, it is expected that the actual posture of the finger approaches the ideal posture of the finger.

  Then, the calibration information creation unit 160 creates calibration information for calibrating the virtual image based on one or a plurality of feature points. Therefore, according to the first embodiment of the present invention, the calibration information is created based on the feature points detected from the image captured after the actual posture of the finger approaches the ideal posture of the finger. Calibration between a hand as a real object and a virtual image can be performed easily and quickly.

[2-1. Outline of Second Embodiment]
In the first embodiment of the present invention, an example has been described in which the posture of a finger is detected by contact and a stimulus is presented to the finger by contact. In the second embodiment of the present invention, an example will be described in which the posture of a finger is detected by non-contact and a stimulus is presented to the finger by non-contact. Hereinafter, the configuration different from the configuration according to the first embodiment of the present invention will be mainly described, and the configuration similar to the configuration according to the first embodiment of the present invention will be described according to the first embodiment of the present invention. The same reference numerals as those used in the configuration are attached and the description thereof is omitted.

[2-2. System configuration example]
Subsequently, a configuration example of an information processing system according to the second embodiment of the present invention will be described. FIG. 7 is a block diagram showing a configuration example of an information processing system according to the second embodiment of the present invention. FIG. 8 is a diagram illustrating a configuration example of an information processing system according to the second embodiment of the present invention. As shown in FIGS. 7 and 8, an information processing system 10B according to the second embodiment of the present invention includes a calibration information creation device 100 and an attitude detection device 200B.

  Further, as shown in FIG. 7, the calibration information creating apparatus 100 has the same configuration as that of the first embodiment of the present invention. On the other hand, as shown in FIG. 7, the posture detection device 200 </ b> B includes a stimulus information control unit 220 as in the first embodiment of the present invention, but as shown in FIGS. 7 and 8, A non-contact finger posture detection unit 240 is provided instead of the contact finger posture detection unit 210 according to the first embodiment, and a non-contact stimulus information presentation unit is used instead of the contact stimulus information presentation unit 230 according to the first embodiment of the present invention. 250.

  The non-contact finger posture detection unit 240 is an example of a finger posture detection unit that detects the actual posture of the finger, and detects the actual posture of the finger by non-contact. The non-contact finger posture detection unit 240 includes a predetermined sensor that can detect the actual posture of the finger by non-contact. As a method in which the non-contact finger posture detection unit 240 detects the actual posture of the finger by non-contact, a technique that has already been disclosed can be used (for example, US Pat. No. 8,638,892 B2). Hereinafter, an outline of such a technique will be described.

  9A to 9C are diagrams for explaining an outline of a technique for detecting an actual posture of a finger in a non-contact manner. As shown in FIG. 9A, the camera 241 and the camera 242 image the user's hand Ha. Here, the three-dimensional positions in the spaces of the camera 241 and the camera 242 are known, and there are overlaps in the imaging areas of the camera 241 and the camera 242, respectively. Here, the left hand in which the little finger and the ring finger are bent is imaged by the camera 241 and the camera 242.

  Imaging with the camera 241 and the camera 242 is performed with respect to each direction about a straight line connecting the camera 241 and the camera 242 as an axis. For example, a cross section Se as shown by a dashed ellipse is imaged. A broken line area on the right side of the cross section Se is an area of the skin surface of the hand that is simultaneously imaged by both the camera 241 and the camera 242. Based on the position, the three-dimensional position of the region is calculated.

  As shown in FIG. 9B, the non-contact finger posture detection unit 240 approximates each cross-section imaged in this way to an ellipse including the areas that are not actually imaged by the camera 241 and the camera 242. Among them, the ellipse Es that best fits the set of calculated three-dimensional positions is calculated. The non-contact finger posture detection unit 240 performs the process of calculating the ellipse Es in this manner on each cross section, and estimates the combination of the process results as a three-dimensional structure of the user's hand as shown in FIG. 9B. .

  Subsequently, the non-contact finger posture detecting unit 240 can estimate a feature point by collating the three-dimensional structure of the user's hand with a hand-shaped model prepared in advance. FIG. 9C shows feature points (joints) Fp of the hand, and an axis Bn connecting the feature points (joints) Fp. The axis Bn indicates the posture of the finger of the hand. That is, the non-contact finger posture detection unit 240 can acquire the finger posture based on the positional relationship between the plurality of feature points (joints) Fp. In addition, TOF (Time-of-Flight) may be used for calculation of the three-dimensional position of the skin surface area of the hand, or a distance image camera using a laser may be used.

  The non-contact stimulus information presentation unit 250 presents a stimulus to the finger by non-contact. The non-contact stimulus information presentation unit 250 is typically described in literature (star, “airborne ultrasonic tactile display equipped with a hand tracking function”, IEICE Technical Report, HIP2012-80 (2013-3)). As shown in the figure, an element that emits ultrasonic waves (hereinafter also referred to as “speaker”) is constituted by an ultrasonic array in which the elements are arranged in a three-dimensional space. Then, the non-contact stimulus information presentation unit 250 transmits the vibration sensed by the haptic nerve to a predetermined position of the finger so that the vibration stimulus is presented to the predetermined part of the finger existing in the three-dimensional space.

  In the example shown in FIG. 8, a plurality of arrows directed to fingers are a plurality of vibration energies output from each ultrasonic element, and the non-contact stimulus information presentation unit 250 intersects outputs from a plurality of speakers. In part, it is possible to generate an amplitude stimulus on the finger, as indicated by the wavy line in FIG. Here, it is described in the literature (Watanabe et al., “All-around tactile sensation using a plurality of ultrasonic transducer arrays”, Robotics and Mechatronics Lectures Collection 2011, “2A2-006”, 2011-05-26). As described above, ultrasonic arrays may be arranged in a plurality of directions so that stimulation can be transmitted from a plurality of directions.

[2-3. Example of operation]
Then, the example of the flow of operation | movement of the information processing system 10B which concerns on the 2nd Embodiment of this invention is demonstrated. FIG. 10 is a flowchart showing an operation flow of the information processing system 10B according to the second exemplary embodiment of the present invention. The flowchart shown in FIG. 10 merely shows an example of the operation flow of the information processing system 10B. Therefore, the operation of the information processing system 10B is not limited to the example illustrated in FIG. Here, the difference from the operation of the information processing system 10A according to the first embodiment of the present invention shown in FIG. 5 will be mainly described.

  In the second embodiment of the present invention, when the ideal posture data is received by the stimulus information control unit 220, the non-contact finger posture detection unit 240 acquires the actual posture data of the finger in a non-contact manner (step S709). Also in the second embodiment of the present invention, if the absolute value of the difference is equal to or greater than the threshold value (“≧” in step S513), the stimulus information control unit 220 determines the finger part to which the stimulus information is to be presented ( Step S514).

  Subsequently, the non-contact stimulus information presentation unit 250 determined as the finger part to be presented with the stimulus information generates a tactile sense presentation pattern (step S715). FIG. 11 is a diagram for explaining generation of a tactile sense presentation pattern. As shown in FIG. 11, a case is assumed where the stimulus information presenting target finger part is determined to be part Re1 and part Re2 in the user's hand Ha by the stimulus information control unit 220. In such a case, the non-contact stimulus information presentation unit 250 corresponds to the part Re1 and the part Re2 in the user's hand Ha among the plurality of feature points (feature point group) Fp detected by the non-contact finger posture detection unit 240. Based on the three-dimensional positions of Re1 and Re2, the output distribution from the ultrasonic array is determined as a tactile sense presentation pattern.

  Then, the non-contact stimulus information presentation unit 250 presents the stimulus based on the generated tactile sense presentation pattern on the finger. When the stimulus is presented to the finger, the user changes the finger posture according to the tactile sense presentation pattern (step S717). It is expected that the actual posture of the finger approaches the ideal posture by changing the posture of the finger. Note that, similarly to the first embodiment, the stimulus information control unit 220 may control the size of the stimulus presented to the finger based on the absolute value of the difference between the posture data. When the stimulus information control unit 220 increases the stimulus presented to the finger as the absolute value of the difference increases, the stimulus information control unit 220 can intuitively make the user know how much the difference is.

  Further, the stimulus information control unit 220 may change the position of the stimulus presented to the finger as time passes. By changing the position of the stimulus presented to the finger with the passage of time, it becomes possible for the user to grasp the ideal posture of the finger. FIG. 12 is a diagram for explaining an example in which the position of the stimulus presented to the finger is changed with time. Here, it is assumed that a stimulus is presented to the thumb.

  Referring to FIG. 12, the actual posture of the finger is opened outward as compared to the ideal posture Rp of the finger. At this time, the stimulus information control unit 220 may gradually change the position of the presented stimulus to the inside as time elapses (as time elapses in the order of time t0, time t1, and time t2) ( The stimulation may be changed in the order of stimulation St0, stimulation St1, and stimulation St2.) In addition, when the actual posture of the finger is opened inward compared to the ideal posture Rp of the finger, the stimulus information control unit 220 gradually changes the position of the presented stimulus to the outside as time passes. It is good to let

[2-4. Summary of Second Embodiment]
According to the second embodiment of the present invention, as in the first embodiment of the present invention, calibration between a hand as a real object and a virtual image can be performed easily and quickly. Further, according to the second embodiment of the present invention, the non-contact finger posture detection unit 240 detects the actual posture of the finger without contact, and the non-contact stimulus information presentation unit 250 presents a stimulus to the finger without contact. To do. Therefore, according to the second embodiment of the present invention, it is possible to eliminate the troublesomeness of the user wearing the device.

[3. Modified example]
The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

  For example, in the above description, the calibration information creation apparatus 100 has been described. However, some or all of the functions of the calibration information creation apparatus 100 may be configured by hardware or realized by a program (calibration information creation program). May be. For example, some or all of the functions of the calibration information creating apparatus 100 may be realized by executing a program that is read from the ROM by the CPU and expanded in the RAM. A computer-readable recording medium that records the program can also be provided.

  Moreover, although the attitude | position detection apparatus 200 was demonstrated above, a part or all of the function which the attitude | position detection apparatus 200 has may be comprised by the hardware, and may be implement | achieved by the program (attitude detection program). . For example, some or all of the functions of the posture detection device 200 may be realized by a program read from the ROM by the CPU and loaded in the RAM. A computer-readable recording medium that records the program can also be provided.

10A, 10B Information processing system 100 Calibration information creation device 110 Video acquisition unit 120 Video display unit 130 Video generation unit 140 Display conversion unit 150 Feature point detection unit 160 Calibration information creation unit 170 Calibration information storage unit 200 (200A, 200B) Attitude detection Device 210 Contact finger posture detection unit 220 Stimulus information control unit 230 Contact stimulus information presentation unit 240 Non-contact finger posture detection unit 241 Camera 242 Camera 250 Non-contact stimulus information presentation unit

Claims (18)

A feature point detection unit that detects one or a plurality of feature points from a captured image in which the finger is imaged after a stimulus according to actual finger posture data is presented to the finger;
A calibration information creation unit that creates calibration information for calibrating a virtual image based on the one or more feature points;
A calibration information creation device comprising: The calibration information creating device is
A display conversion unit that obtains a presentation video by converting the virtual video based on the calibration information;
The calibration information creation device according to claim 1. The calibration information creating device is
A video generation unit for generating the virtual video;
The calibration information creation device according to claim 2. The calibration information creating device is
A video display unit for displaying the presented video;
The calibration information creation device according to claim 3. The feature point detection unit detects the one or more feature points when a difference absolute value between the actual posture data and the ideal posture data of the finger falls below a threshold;
The calibration information creation device according to claim 1. Detecting one or a plurality of feature points from an imaged image obtained by imaging the finger after a stimulus corresponding to the actual posture data of the finger is presented to the finger;
Creating calibration information for calibrating a virtual image based on the one or more feature points;
Calibration information creation method including Computer
A feature point detection unit that detects one or a plurality of feature points from a captured image in which the finger is imaged after a stimulus according to actual finger posture data is presented to the finger;
A calibration information creation unit that creates calibration information for calibrating a virtual image based on the one or more feature points;
A calibration information creation program for causing a calibration information creation device to function. A finger posture detector that detects actual posture data of the finger;
A stimulus information control unit that controls a stimulus presented to the finger based on the actual posture data and predetermined ideal posture data of the finger;
An attitude detection device comprising: The stimulus information control unit controls the stimulus based on a difference between the actual posture data and the ideal posture data;
The posture detection apparatus according to claim 8. The stimulus information control unit controls the position of the stimulus presented to the finger based on the sign of the difference.
The posture detection apparatus according to claim 9. The stimulus information control unit changes the position of the stimulus presented to the finger with time.
The posture detection apparatus according to claim 8. The stimulus information control unit controls the magnitude of the stimulus presented to the finger based on the absolute value of the difference.
The posture detection apparatus according to claim 9. The posture detection device is
A stimulus information presentation unit for presenting a stimulus to the finger;
The posture detection apparatus according to claim 8. The finger posture detection unit detects the actual posture data by contact or non-contact,
The stimulus information control unit controls the stimulus presented by contact or non-contact;
The posture detection apparatus according to claim 8. One or a plurality of feature points are detected from a captured image obtained by capturing the finger after the stimulus is presented to the finger, and calibration information for calibrating a virtual image based on the one or a plurality of feature points is provided. Created,
The posture detection apparatus according to claim 8. Detecting actual posture data of the finger;
Controlling a stimulus presented to the finger based on the actual posture data and predetermined ideal posture data of the finger;
A posture detection method including: Computer
A finger posture detector that detects actual posture data of the finger;
A stimulus information control unit that controls a stimulus presented to the finger based on the actual posture data and predetermined ideal posture data of the finger;
A posture detection program for causing a device to function as a posture detection device. A finger posture detector that detects actual posture data of the finger;
A stimulus information control unit that controls a stimulus presented to the finger based on the actual posture data and predetermined ideal posture data of the finger;
An attitude detection device comprising:
A feature point detection unit for detecting one or more feature points from an imaged image obtained by imaging the finger after the stimulus is presented to the finger;
A calibration information creation unit that creates calibration information for calibrating a virtual image based on the one or more feature points;
A calibration information creation device comprising:
An information processing system.

Images