JP2013164413A - Virtual point determination device and method, and device used to determine virtual point - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a virtual point determination device capable of reducing persons required for determination work of a virtual point for setting a virtual marker.SOLUTION: Such a pointer 1 which includes three markers M, M, Mand a tip part 20 for pointing a virtual point and can specify the position of the tip part 20 by a marker position on the pointer is used. Distances among the respective markers of the pointer 1 are known. The pointer 1 in which one marker Mamong the three markers is not recognized by cameras is photographed by a plurality of cameras. When a virtual point position is pointed by the tip part 20 of the pointer, the pointer 1 is detected by changing a state in which the one marker Mamong the three markers is not recognized by the cameras to a state in which the three markers are recognized by the cameras. The coordinates of the tip part 20 are acquired as virtual point coordinates from marker coordinates during pointer detection with the pointer detection as a determination input.

Description

The present invention relates to a virtual point determination apparatus and method, and a device used for virtual point determination.

Analyzing the three-dimensional movement of a person is useful in the medical field, the care and welfare field, the sports field, robotics, computer graphics, and the like. A motion capture system is known as means for measuring a three-dimensional motion of a person or an object. In an optical motion capture system, subjects having markers attached to a plurality of predetermined points are photographed by a plurality of cameras, each marker is identified in the camera image, and time-series data of the three-dimensional coordinates of each marker is acquired. Is done. A three-dimensional motion analysis apparatus using optical motion capture is described in Patent Document 1, for example.

However, since the three-dimensional motion measurement measures the moving person or object in the first place, the marker photographed by the camera at a certain time is hidden at another certain time t, and the position information of the marker cannot be acquired. There is a case. Although it may be considered to acquire the position information of each marker at each time by increasing the number of cameras, there is a problem that increasing the number of cameras increases the cost of the entire system. It is also conceivable to estimate and interpolate the hidden marker position information in the frame _t from the marker position information in the frame _t-1 and the frame _{t + 1} before and after the time t when the marker position information could not be acquired. Computation processing is complicated, and it is difficult to interpolate the position information of markers that are continuously hidden over a plurality of frames.

A virtual marker may be used as means for solving such a problem. The virtual marker is not a marker (actual marker) actually attached to a subject or the like, but a virtual marker whose position is specified by a relative position with respect to a plurality of actual markers. More specifically, the “virtual marker” is at a certain distance from three or more actual markers, and when there are three markers, one of the front and back surfaces of the surface created by the three markers. A point is set, a coordinate value is calculated from the coordinate value of the actual marker and the setting, and the calculated coordinate value is handled as a marker. By using the virtual marker, it is possible to avoid the phenomenon that the marker is hidden and data cannot be acquired. It is also possible to acquire time-series data of coordinate values of places that cannot be seen from the installed camera.

When setting a virtual marker in a three-dimensional motion measurement system, it is performed by setting a virtual point with a device called a pointer. In the field of human body motion analysis and the like, a pointer that points to a predetermined part of the human body is often used (Non-Patent Document 1). In one aspect, the pointer includes three markers, and the position of the tip can be specified by the positions of the three markers. In the conventional virtual marker setting work, two or more operators, that is, an operator pointing a virtual point with a pointer and a PC operator are required. Since the point used as a virtual marker often uses a characteristic position (marking point) of the bone near the body surface, the “operator pointing to the virtual marker” finds this position by palpation and Work to point the pointer to the place. The “operator pointing to the virtual marker” is the subject himself / herself or another person (such as a doctor) to whom the real marker is attached.

A pointer (three markers) pointing to a virtual marker and a subject (at least a real marker that identifies the virtual marker) to which a real marker is attached are photographed by a plurality of cameras, and image data is transmitted to a computer. The computer obtains the coordinates of the pointer tip by calculating the coordinates of the three markers on the pointer based on the input image data. The “PC operator” operates the input means of the computer based on the report from the operator that the “operator pointing to the virtual marker” has found the mark and pointed to the place with the pointer. The virtual point is determined and input.

In a three-dimensional motion measurement system, a subject that is photographed by a camera and a computer that takes in image data and performs various calculations are physically separated. In addition, the position indicated by the skin may be shifted when the pointer is pressed against the reference point, and it may be difficult to find the reference point, so the operator pointing to the virtual marker does another work. It is difficult. Therefore, conventionally, it has not been possible to perform a virtual point determination operation for setting a virtual marker alone.

JP 2004-344418 A

Journal of Biomechanics 36 (2003) 1159-1168 “Repeatability of gaitdata using a functional hip joint center and a mean helical knee axis”, Thor F. Besier, Daina L. Sturnieks, Jacque A. Alderson, David G. Lloyd

The present invention provides a virtual point determination apparatus and method that can reduce the number of personnel required for the virtual point determination work for setting a virtual marker and that can also be performed alone. It is intended to do.

The first technical means adopted by the present invention is:
Multiple cameras,
A device comprising: at least three markers; and one or more indication units indicating a virtual point, the position of the virtual point being identifiable by a marker position on the device;
Means for storing information capable of specifying a distance between each marker of the device;
Device detection means for detecting the device using a plurality of camera images photographed by the plurality of cameras and a distance between the markers of the device;
Virtual point coordinate acquisition means for acquiring the coordinates of the virtual point from the marker coordinates using the relative position of each marker on the device and the virtual point;
With
The device detecting means detects the device when the at least three markers are recognized by the camera from a state where at least one of the at least three markers is not recognized by the camera;
The virtual point coordinate acquisition means uses the detection of the device as a decision input, and acquires the coordinates of the virtual point from the marker coordinates at the time of device detection.
A virtual point determination device.
In one aspect, the said instruction | indication part is an edge part of a device.
Typically, the virtual point is set on the device, and the instruction unit (end or the like) is the position of the virtual point, but is not limited to the one set on the device. For example, the virtual point may be set on an extension line indicated by the device.
In one aspect, the instruction unit is an end of a device, and the position of the virtual point is determined by the position of the end. In particular,
Multiple cameras,
A device comprising at least three markers and an end (indicator) pointing to a virtual point, the position of the end being identifiable by the marker position on the device;
Means for storing information capable of specifying a distance between each marker of the device;
Device detection means for detecting the device using a plurality of camera images photographed by the plurality of cameras and a distance between the markers of the device;
End coordinate acquisition means for acquiring the coordinates of the end from the marker coordinates using the relative position of each marker and the end on the device;
With
The device detecting means detects the device when the at least three markers are recognized by the camera from a state where at least one of the at least three markers is not recognized by the camera;
The end coordinate acquisition means uses the detection of the device as a determination input, and acquires the coordinates of the end as virtual point coordinates from the marker coordinates at the time of device detection.
A virtual point determination device.

In one aspect, at least one of the at least three markers of the device is a hidden marker having a first state that is not recognized by the camera and a second state that is recognized by the camera;
The device includes switching means for switching the hidden marker from the first state to the second state when the device indicates a desired virtual point position.
For example, if the device comprises three markers, only one marker may be a hidden marker, two markers may be hidden markers, and all three markers may be hidden markers.

In one aspect, the switching means includes a switch provided on the device,
The switch can be manually operated by an operator holding the device and pointing to a desired virtual point,
The acquisition of the virtual point coordinates is executed by a remote determination input by operating a switch on the device.
In one aspect, the switch is a push switch provided at the end, and an operator uses the device to operate on the object or within the object (the object is the body of the subject, the object held by the subject, the subject The push switch hits the object and is inputted.
Note that a plurality of parts (instruction parts) pointing to the virtual points may be provided on the device. For example, the device is formed in a part other than the first instruction part formed at the end part and the end part of the device. A second instruction unit. On the device, at least one marker is provided on the first side across the second indicator, and the remaining markers are provided on the second side.

In one aspect, the hidden marker is changed from the first state of being turned off to the second state of being turned on by the switching means.
For example, the hidden marker is formed of an LED, and when a switch on the device as a switching unit is pressed, a current is supplied to the LED and the LED is turned on.

In one aspect, the hidden marker includes a movable body movable between a first position for hiding the hidden marker and a second position for exposing the hidden marker, and the movable body is moved by the switching means. By moving from the first position where the hidden marker is hidden to the second position where the hidden marker is exposed, the second state is changed from the first state.
For example, the movable body is configured to slide or rotate from the first position to the second position in conjunction with the pushing operation of the switch on the device as the switching means. It is configured to return from the position to the first position.

  In one aspect, when the virtual point coordinates are acquired using detection of the device as a decision input, a virtual marker is set by the virtual point coordinates and the coordinates of three or more real markers on the subject.

The second technical means adopted by the present invention is:
Multiple cameras,
A device comprising: at least three markers; and one or more indication units indicating a virtual point, the position of the virtual point being identifiable by a marker position on the device;
Device detection means for detecting the device using a plurality of camera images photographed by the plurality of cameras and a known distance between each marker;
Virtual point coordinate acquisition means for acquiring the coordinates of the virtual point from the marker coordinates using the relative position of each marker on the device and the virtual point;
Use
Photographing the device with the plurality of cameras, wherein at least one of the at least three markers is not recognized by the camera;
When at least one of the at least three markers is not recognized by the camera when the desired virtual point position is pointed by the pointing unit of the device, the at least three markers are recognized by the camera. Detecting the device; and
Using the detection of the device as a decision input, obtaining the coordinates of the virtual point from the marker coordinates at the time of device detection;
A virtual point determination method comprising:
In one aspect, the said instruction | indication part is an edge part of a device.
Typically, the virtual point is set on the device, and the instruction unit (end or the like) is the position of the virtual point, but is not limited to the one set on the device. For example, the virtual point may be set on an extension line indicated by the device.
In one aspect, the instruction unit is an end of a device, and the position of the virtual point is determined by the position of the end. In particular,
Multiple cameras,
A device comprising at least three markers and an end (indicator) pointing to a virtual point, the position of the end being identifiable by the marker position on the device;
Device detection means for detecting the device using a plurality of camera images photographed by the plurality of cameras and a known distance between each marker;
End coordinate acquisition means for acquiring the coordinates of the end from the marker coordinates using the relative position of each marker and the end on the device;
Use
Photographing the device with the plurality of cameras, wherein at least one of the at least three markers is not recognized by the camera;
When at least one of the at least three markers is not recognized by the camera when the desired virtual point position is pointed by the end of the device, the at least three markers are recognized by the camera. Detecting the device; and
With the detection of the device as a decision input, obtaining the coordinates of the end from the marker coordinates at the time of device detection as virtual point coordinates;
A virtual point determination method comprising:

In one aspect, at least one of the at least three markers of the device is switchable from a first state not recognized by the camera to a second state recognized by the camera;
When the desired virtual point position is pointed by the device, the device is detected by switching the hidden marker from the first state to the second state.

In one aspect, the switching of the hidden marker from the first state to the second state is performed by a switch provided on the device,
The switch can be manually operated by an operator holding the device and pointing to a desired virtual point,
The acquisition of the virtual point coordinates is executed by a remote determination input by operating a switch on the device.
In one aspect, the switch is a push switch provided at the end, and an operator uses the device to operate on the object or within the object (the object is the body of the subject, the object held by the subject, the subject The push switch hits the object and is inputted.
Note that a plurality of parts (instruction parts) pointing to the virtual points may be provided on the device. For example, the device is formed in a part other than the first instruction part formed at the end part and the end part of the device. A second instruction unit. On the device, at least one marker is provided on the first side across the second indicator, and the remaining markers are provided on the second side.

  In one aspect, the device is detected when the hidden marker is changed from the first state of being turned off to the second state of being turned on.

  In one aspect, the movable body in the first position that hides the hidden marker moves to the second position that exposes the hidden marker, so that the hidden marker is in the second state from the first state. Thus, the device is detected.

In one aspect, the plurality of cameras, together with the device, photograph at least a part of a subject to which a real marker is attached, so that the camera image has three or more real markers for specifying a virtual point. Is included,
When the virtual point coordinates are acquired using the detection of the device as a decision input, a virtual marker is set by the virtual point coordinates and the coordinates of the three or more real markers.

The third technical means adopted by the present invention is:
A device that includes at least three markers and an instruction unit (for example, an end) that indicates a virtual point, and is used for determining a virtual point,
The distance between each marker is known, and the position of the virtual point can be specified by the marker position on the device,
One of the at least three markers is a hidden marker having a first state not recognized by the camera and a second state recognized by the camera;
The device is a device including switching means for switching the hidden marker from the first state to the second state.
In one aspect, the switching means includes a switch provided on the device.
In one aspect, the switch is a push switch provided at the end, and an operator uses the device to operate on the object or within the object (the object is the body of the subject, the object held by the subject, the subject The push switch hits the object and is inputted.
Note that a plurality of parts (instruction parts) pointing to the virtual points may be provided on the device. For example, the device is formed in a part other than the first instruction part formed at the end part and the end part of the device. A second instruction unit. On the device, at least one marker is provided on the first side across the second indicator, and the remaining markers are provided on the second side.

  In one aspect, the hidden marker is changed from the first state of being turned off to the second state of being turned on by the switching means.

  In one aspect, the hidden marker includes a movable body movable between a first position for hiding the hidden marker and a second position for exposing the hidden marker, and the movable body is moved by the switching means. By moving from the first position where the hidden marker is hidden to the second position where the hidden marker is exposed, the second state is changed from the first state.

According to the present invention, a virtual marker can be set remotely in a three-dimensional motion analysis system. More specifically, in the present invention, when a device that has not been detected is detected, the detection of the device is used as a decision input, and the virtual point coordinates are obtained from the marker coordinates at the time of device detection. In addition, since it is not necessary for the operator on the PC side to make a decision input using the input means, and the operator on the PC side is not required, it is possible to reduce the number of personnel necessary for the determination of the virtual point for setting the virtual marker. It is also possible to perform the virtual point determination work alone.

It is a figure which shows embodiment of the device used for a virtual point determination. FIG. 6 is a partial view showing another embodiment of a device. It is a figure which shows the distance between the markers on a device. It is a figure which shows the positional relationship of the marker on a device, and a front-end | tip part. It is the schematic which shows the setting operation | work of virtual marker _MH . It is a figure which shows the real marker attached to the test subject used for a virtual marker setting. FIG. 6 illustrates another embodiment of a device. It is a figure explaining the effect | action of the device shown in FIG. It is a figure which shows the aspect from which the setting position of a virtual point differs in relation to a device.

An embodiment of the present invention will be described. The virtual point determination system according to the present embodiment can be composed of a device called a pointer and an optical motion capture system. In general, an optical motion capture system includes a plurality of optical markers (for example, infrared reflection markers) attached to a plurality of predetermined parts of a subject and a plurality of cameras that simultaneously photograph a subject wearing the optical markers from a plurality of angles. And the 2D position of the marker in the image information of the marker obtained by each camera is reconstructed to calculate the 3D position of the marker, and the body is determined from the 3D position of the optical marker and the 3D model of the body. And a display unit for displaying the result of the processing unit (exercise data acquired as time-series data of the posture of the subject).

Various calculations in the motion capture system may be performed by a computer. Specifically, an image input unit that captures image information acquired by the camera, a storage unit that stores the captured image information and information calculated by the image processing unit, the image information, measurement results, and analysis results are displayed. A display unit, an image processing unit for performing image processing on the image information, and a calculation unit for calculating time-series data of the three-dimensional position of the marker are provided. A processing unit that performs data processing using various measured data and a display unit that displays input camera images and processing results are a general-purpose computer (an input device for inputting data, for outputting processed data). Output device, display device for displaying data, arithmetic device mainly composed of CPU, storage device such as ROM, RAM, hard disk, bus connecting them, stored in the storage device so that the computer can execute predetermined processing A predetermined program, etc.).

As shown in FIG. 1, the pointer 1 according to the embodiment has a T shape from a first shaft portion 2 and a second shaft portion 3. The first shaft portion 2 has a longer dimension than the second shaft portion 3, and the second shaft portion 3 is formed at the base end portion of the first shaft portion 2 with respect to the first shaft portion 2. It extends so as to be orthogonal to the length direction. The first shaft portion 2 is a gripping portion, and the distal end portion 20 of the first shaft portion 2 is an end portion (indicating portion) that points to a desired virtual point. One end of the second shaft portion 3 first marker M ₁ is the second marker M ₂ is provided at the other end. A third marker M ₃ is disposed on the first shaft portion 2 so as to be located on the distal end side.

The first marker M ₁ and the second marker M ₂ are infrared reflective markers. The third marker M ₃ are a marker consisting of infrared LED, which functions as a marker by lights. Pointer on (in the illustrated embodiment on the first shaft 2, the part close to the third marker M ₃₎ in Yes the switch 4 is provided, by pressing the switch 4, incorporated in the pointer 1 the power (shown not) current to the infrared LED lights are supplied from the third marker M ₃ is a first state of oFF (second state lighting from a state) that is not recognized by the camera (state recognized by the camera) Become.

A means for detecting a pointer using a plurality of camera images photographed by a plurality of cameras and stored distances between markers will be described. The distance between each marker of the pointer is known (including a case where it can be calculated from a known value), and is stored in a storage unit (typically, a storage unit of a computer constituting the motion capture system). In the present embodiment, the distance _{D 2} between 3 to shown, first distance _{D 1} of the between markers _{M 1} and the second marker _{M 2,} first marker _{M 1} and the third marker _{M 3,} second distance _{D 3} between the markers _{M 2} and the third marker _{M 3} is used. When the shape of the marker is spherical or circular, in one embodiment, the distance between the markers is the distance between the centers of the markers.

In a specific aspect, as shown in FIG. 4, the distance L <b> 1 between the third marker M <b> ₃ and the distal end portion 20 of the first shaft portion 2, the base of the third marker M <b> ₃ and the first shaft portion 2. end distance L2, the 2 markers M ₂ and the base end portion of the first shaft portion 2 between the (first marker M ₁ to be located on a line connecting the second marker M ₂ center) (first the distance between the position on the longitudinal center line of the shaft portion 2) L3, the first marker M ₁ and the first base end portion of the shaft portion 2 (the first shank 2 lengthwise The distance L4 from the center line is stored in the storage unit. The inter-marker distance D ₁ between the first marker M ₁ and the first marker M ₂ is the distance L 3 and the distance L 4, and the inter-marker distance D ₃ between the first marker M ₁ and the first marker M ₂ is the distance L 2 and the distance L 4. Accordingly, the inter-marker distance D ₃ between the first marker M ₂ and the first marker M ₃ is calculated in real time from the distance L 2 and the distance L 3. Alternatively, the distance _{D 1} of the between the first marker _{M 1} and the second marker _{M 2,} the distance _{D 2} between the first marker _{M 1} and the third marker _{M 3,} the second marker _{M 2} and the third marker _{M 3} it may be stored in the direct storage unit distance D ₃ between. The information stored in advance in the storage unit in order to specify the distance between the markers on the device is a matter that can be appropriately set by those skilled in the art according to the shape of the pointer, the marker mounting position on the pointer, and the like.

The three markers M ₁ , M ₂ , and M ₃ on the pointer 1 are recognizable by the camera, and the three dimensions of the three markers in the acquired image are obtained from a plurality of camera images obtained by simultaneously photographing the pointers by the plurality of cameras. Coordinates M ₁ , M ₂ , M ₃ can be obtained, and using the coordinates of each marker, three distances d ₁ , d ₂ , d ₃ between each marker can be calculated.

By comparing the calculated three distance sets d ₁ , d ₂ , d ₃ with the stored distance sets D ₁ , D ₂ , D ₃ , the distance sets D ₁ , D ₂ , When a distance set set d ₁ , d ₂ , d ₃ matching D ₃ is found, it can be considered that the pointer 1 has been detected. The distance sets D ₁ , D ₂ , and D ₃ are provided to the subject in order to prevent erroneous detection of the pointer in consideration of the case where the actual marker provided to the subject is also included in the camera image. It is desirable to set a distance set that cannot be obtained by a combination of actual markers.

A means for acquiring the coordinates of the tip from the marker coordinates using the relative position between each marker on the pointer and the tip of the first shaft portion will be described. On the pointer 1, the relative positional relationship between the distal end portion 20 of the first shaft portion 2 and the first marker M ₁ , second marker M ₂ , and third marker M ₃ is known, and the first marker M ₁ , information that can specify the position of the distal end portion 20 of the first shaft portion 2 using the positions of the second marker M ₂ and the third marker M ₃ (for example, L1 to L4 in FIG. 4) is stored in the storage unit. It is remembered. Based on the coordinates of the three markers in the acquired image when distance set (d ₁ , d ₂ , d ₃ ) = distance set (D ₁ , D ₂ , D ₃ ), The coordinates of the distal end portion 20 (in one aspect, located on the center line in the length direction of the first shaft portion 2) can be acquired.

In the present embodiment, the third marker M ₃ among the _three markers M ₁ , M ₂ , and M ₃ on the pointer 1 is turned off before the desired virtual point is indicated by the tip 20 of the pointer 1. (State not recognized by the camera), when the tip 20 of the pointer 1 points to a desired virtual point, the switch 4 is pressed with the tip 20 pointing to the virtual point, and the third marker M ₃ is as lighting state (a state recognized by the camera), the distance set between the three markers recognized by the camera (d _{1, d} 2, _{d 3),} the stored three distances set between markers (D ₁ , D ₂ , D ₃ ) to detect the pointer 1 in the image taken by the camera.

Pointer non-detection state pointer detection state from the (third marker M ₃ is a state that is not recognized by a turned off camera) (third marker M ₃ is turned on in all three markers M _1, M _2, M _{As a} decision input, the tip 20 of the first shaft portion 2 of the pointer 1 is determined from the coordinates of the markers M ₁ , M ₂ , and M ₃ on the pointer 1 when the pointer is detected. Are obtained as virtual point coordinates. The storage unit of the computer causes the computer to execute such processing (that is, using the detection of the device as a decision input and acquiring the coordinates of the pointer end from the marker coordinates at the time of device detection as the virtual point coordinates). The program is stored. In the illustrated embodiment, one hidden marker (third marker M ₃ ) is used, but two or more hidden markers may be provided. For example, a mechanism may be adopted in which all three markers on the pointer are LEDs and all LEDs are lit when a button is pressed.

The switch 4 on the pointer 1 can be manually operated by an operator who holds the first shaft portion 2 of the pointer 1 and points to a desired virtual point, and the acquisition of the virtual point coordinates is determined by the switch operation of the pointer 1. Executed by input. That is, by performing remote determination input by an operator pointing to a virtual point, it is not necessary to perform determination input work on the computer side. In the illustrated embodiment, the switch 4 on the first shaft portion 2 is provided. However, it is understood that the position of the switch can be appropriately changed by those skilled in the art. Also, the overall shape of the pointer, the number of markers (only three or more), the attachment site, and the position of the end pointing to the virtual point are not limited to the illustrated form.

Another aspect of the pointer provided with the hidden marker will be described with reference to FIG. The third marker M ₃ to be hidden marker is an infrared reflective marker, the first shaft portion 2, the exposing the first position and the third marker M ₃ to conceal the third marker M ₃ 2 A movable body 5 that is movable between positions (typically movable in the axial direction of the first shaft portion 2) is provided. On the first shaft portion 2, the movable body 5 at the first position (the upper diagram in FIG. 2) is moved to the second position (the lower diagram in FIG. 2), and the third marker M ₃ is moved to the first position. Switching means for switching from the state to the second state is provided.

In one aspect, the switching means is the slide knob 6 on the first shaft portion 2, and the slide of the slide knob 6 and the movement of the movable body 5 are linked to each other, and is movable in conjunction with the movement of the slide knob 6. body 5 switches the third marker M ₃ is moved from the first position to the second position from the first state to the second state. When the force acting on the slide knob 6 is released, the movable body 5 is configured to return from the second position to the first position by a spring (not shown) provided in the first shaft portion 2. The movable body 5 may be returned from the second position to the first position by manually moving the slide knob 6 without using a return means such as a spring. In place of the slide knob, the push operation of the switch and the movement of the movable body are linked, and the movable body moves from the first position to the second position in conjunction with the push operation of the switch, and the pressing force on the switch is increased. When released, the movable body may be configured to return from the second position to the first position by a spring (not shown) provided on the first shaft portion.

In the present embodiment, in a stage before pointing a desired virtual point at the tip 20 of the pointer 1, the third state of the marker M ₃ is concealed by the movable body 5 (the camera of the three markers on the pointer 1 When the tip 20 of the pointer 1 points to the desired virtual point, the movable body 5 is moved from the first position by the switch operation with the tip 20 pointing to the virtual point. 2 and moving the third marker M ₃ into an exposed state (a state recognized by the camera), a distance set (d ₁ , d ₂ , d ₃ ) between the three markers recognized by the camera and The pointer 1 in the image photographed by the camera is detected using the stored distance sets (D ₁ , D ₂ , D ₃ ) between the three markers.

Pointer non-detection state pointer detection state from the (third marker M ₃ is concealed in it state that is not recognized by the camera) (third marker M ₃ is an exposed state, all three markers M _1, M _2, M _{As a} decision input, the tip 20 of the first shaft portion 2 of the pointer 1 is determined from the coordinates of the markers M ₁ , M ₂ , and M ₃ on the pointer 1 when the pointer is detected. Are obtained as virtual point coordinates. In the illustrated embodiment, one hidden marker (third marker M ₃ ) is used, but two or more hidden markers may be provided. In this case, it is desirable to move a plurality of movable bodies by one switch operation.

Above two embodiments, the (specifically by giving certain feature to the structure of the pointer 1, the third switching the lighting of the marker M _3, third marker M ₃ of the physically using mechanical mechanism Hide), in the first phase, by a plurality of cameras, the three only the third marker M ₃ of the markers taking the pointer 1 in a state that is not recognized by the camera (the first marker M _1, the second markers M ₂ is recognized by the camera), when pointing to a desired virtual point position at the tip 20 of the pointer 1, since the third marker M ₃ is to be recognized from a state that is not recognized by the camera, It has been explained that all three markers M ₁ , M ₂ and M ₃ are recognized by the camera and the pointer 1 is detected.

In the present invention, the means for hiding the third marker M ₃ physically may be one performed manually without depending on the mechanism of the pointer. The operator hides at least one of the three markers on the pointer with a finger (or a hand or other object), so that the marker is in a first state that is not recognized by the camera. By using the distance between the three markers recognized by the camera by exposing the marker hidden by the operator's finger or the like with the tip pointing to the virtual point when the virtual point is pointed to May be detected. Even in this case, using the coordinate of the marker on the pointer at the time of detecting the pointer as the decision input that the switching from the pointer non-detection state to the pointer detection state is performed, the coordinates of the tip of the first axis portion of the pointer are virtually The points acquired as point coordinates are the same, and a program for causing the computer to execute in this way is stored in the storage unit of the computer.

When the virtual point coordinates are acquired by the pointer 1, the virtual marker _MH is set by the virtual point coordinate value and the coordinate values of the three real markers m ₁ , m ₂ , and m ₃ on the subject.

Acquisition of the coordinate values of the actual markers m ₁ , m ₂ , and m ₃ will be described. First, identification of a marker (actual marker) on the body will be described. A marker captured by two or more cameras can be calculated as a three-dimensional coordinate value by using camera parameters. However, it is necessary to associate (marker identification) which marker corresponds to the calculated three-dimensional coordinate value. Normally, a camera image may include a plurality of real markers provided to the subject in addition to the markers M ₁ , M ₂ , M ₃ on the pointer 1 and real markers m ₁ , m ₂ , m ₃ .

In one aspect, a three-point marker fixed on a rigid body is used when determining a virtual point using a pointer. Since these markers are fixed on the rigid body, the distance between the markers does not change. Using this property, it is determined as follows.
(1) The distance between markers attached on a rigid body on the PC is set in advance. Each distance is stored in the storage unit of the computer.
(2) Calculate the three-dimensional coordinate value of the marker.
(3) Select any three markers and calculate the distance of the calculated three-dimensional coordinate value.
(4) If the length is the same as the setting in (1), the marker is a rigid body. If the length is different, return to (3).

A marker fixture for realizing a three-point marker fixed on a rigid body is shown in FIG. As shown in FIG. 6, three markers are fixed on the marker fixture, and by setting the distance between the markers on the PC, the marker on the marker fixture in the camera image is identified. . One skilled in the art will appreciate that more than four markers fixed on a rigid body can be used by using similar algorithms. It should be noted that the method for identifying a marker on the body described here is merely one embodiment, and it will be understood by those skilled in the art that various methods can be used for identifying a marker on the body.

Next, setting of the virtual marker will be described. In setting the virtual marker, the marker coordinate values p ₁ ^{t = 0} , p ₂ ^{t = 2} , p ₃ ^{t = 0} and the virtual point coordinate value p ^{t = 0} when the virtual point is determined by the pointer To decide. These values are used when calculating virtual points. The operation of obtaining these values is the setting of the virtual marker. Here, the subscript t represents time, and t = 0 represents a value at the time of setting a virtual marker.

The calculation of virtual points during actual measurement will be described. A coordinate system made up of three markers m ₁ , m ₂ and m ₃ on the body is called a somite coordinate system. One of the markers on the body is used as the origin as the origin of the body segment coordinate system. In addition, each axis of the unit vector of the body segment coordinate system shall be determined as follows:

First axis: unit direction vector (= v ₁ ^t ) from marker m ₁ to marker m ₂ .
Second axis: a unit normal vector (= v ₂ ^t ) of a plane formed by _three markers m ₁ , m ₂ , and m ₃ on the body.
Third axis: a unit vector (= v ₃ ^t ) orthogonal to the first and second axes.
At this time, the transformation matrix from the world coordinate system to the body segment coordinate system at time t can be written as follows.

When the coordinate value of the virtual point at the time of setting the virtual marker is converted into the body segment coordinate system, it can be written as follows.
When this virtual point is converted to the world coordinate system at time t, the virtual point coordinate value at time t is
It can be expressed as. The virtual point coordinate value is calculated using this equation.

The overall flow of determining a virtual point remotely using the pointer 1 will be described. First, a subject wearing a real marker (including m ₁ , m ₂ , m ₃ ) and a pointer 1 (the third marker M ₃ is in the first state) are photographed by a plurality of cameras, Import camera image data into the computer.

The operator selects a desired virtual point position and places the tip 20 of the pointer 1 on the virtual point position. Here, the virtual point position is on the subject's body, on an object (tool such as a cane) worn by the subject or on an object held by the subject (tool such as a cane), or on the subject's environment (eg, staircase). Position. That is, the position where the virtual point can be set is a position “on the object related to measurement (including on the body)”. In Figure 5, shows a state in which the operator against the tip 20 to a predetermined position on the body by grabbing the pointer 1, the detection (determination) Input Thus the pointer 1 corresponding to the lighting of the marker M ₃ in this state The virtual marker _MH is set at the tip 20. Each camera image includes actual markers m ₁ , m ₂ and m ₃ attached to the body and markers M ₁ , M ₂ and M ₃ on the pointer 1.

After calculating the three-dimensional coordinates of each marker detected in the camera image, any three markers are selected, and the distance between each marker is set on the PC (stored in the storage unit of the PC) It is determined whether or not the distance between the markers on the pointer 1 (specified by the information) is the same. That is, the distance between all markers for which three-dimensional coordinates are calculated based on a plurality of camera images is calculated in real time, the calculated distance between markers, the distance between markers on the pointer set on the PC, Are compared, and if these lengths match, it is assumed that the pointer has been detected.

A third marker M ₃ on the pointer 1 when in the first state, the detection of the pointer is not performed. When pointed desired virtual point position by the pointer 1 of the tip portion 20, the third marker M _3, switching from the first state to the second state. When the third marker M ₃ is a second state, detects the pointer 1 by using a plurality of camera images shot by a plurality of cameras, each inter-marker distance on the stored pointer 1, the .

Using the detection of the pointer as a decision input, the coordinate value of the tip of the point is acquired as the virtual point coordinate value from the marker coordinates on the pointer when the pointer is detected. In parallel with the detection of the pointer, a search for three or more real markers on the subject specifying the virtual marker is executed to obtain the coordinate values of the three real markers on the subject.

Finally, the usage mode of the virtual marker will be described based on two examples. When a marker is affixed to the greater trochanter of a hemiplegic patient, the marker data may be hardly acquired by the arm. In such a case, by setting the virtual marker to the greater trochanter point, it can function as a greater trochanter marker. Specifically, by fixing three markers to the pelvis and determining the greater trochanter point based on these marker coordinates, it is possible to eliminate the problem of the marker becoming invisible by the arm.

Further, when acquiring wrinkle marker data, it is necessary to have a plurality of cameras in the rear. On the other hand, setting a virtual marker to wrinkle may cause it to function as a wrinkle marker. Specifically, by attaching three markers in front of the foot, and setting a virtual marker for the heel from there, it is possible to calculate a three-dimensional coordinate value from the virtual marker for the heel without a camera behind .

Another embodiment will be described with reference to FIG. The pointer 1 'has a T shape from the first portion 2' and the second portion 3 '. In the illustrated form, the first portion 2 'and the second portion 3' are plate-like longitudinal members. The first part 2 'has a longer dimension than the second part 3', and the second part 3 'is located at the proximal end of the first part 2' with respect to the first part 2 '. It extends so as to be orthogonal to the length direction. The first portion 2 ′ is a gripping portion, and the tip portion 20 ′ of the first portion 2 ′ is an instruction portion that points to a desired virtual point.

One end of the second portion 3 'first marker M ₁ is the second marker M ₂ is provided at the other end. The first part 2 ', the third marker M ₃ is disposed is positioned on the distal end side. The first marker M ₁ , the second marker M ₂ , and the third marker M ₃ are markers composed of infrared LEDs, and function as markers when turned on by energization.

Two button switches 4A and 4B are provided on the pointer 1 '. The button switch 4A is provided at the tip portion 20 'of the first portion 2', and the first portion 2 'is provided at the surface portion of the intermediate portion in the length direction of the first portion 2'. In the device 1 ′, the first marker M ₁ , the second marker M ₂ , and the third marker M ₃ are arranged so as to be spatially located on both sides with the button switch 4B interposed therebetween. More specifically, the first side with respect to the first part 2 '(the distal end side of the first portion 2') is a third marker M ₃ is disposed, the first portion 2 ' On the other hand, the first marker M ₁ and the second marker M ₂ are arranged on the second side (both ends of the second portion 3 ′ on the base end side of the first portion 2 ′). The button switches 4A and 4B are both provided in an instruction unit indicating a virtual point and function as an instruction unit. When the button switch 4A or 4B is pressed, a current is supplied to each infrared LED from a power source (not shown) built in the pointer 1 'to light up, and each marker is turned off in a first state (recognized by the camera). The second state of lighting (a state recognized by the camera). As will be described later, when the button switch 4A is pressed, the values used for calculating the virtual point coordinates are different from when the button switch 4B is pressed.

[Significance of button switch 4A]
In the virtual point input operation using the pointer 1 as shown in FIG. 1, the operator needs two actions of pointing the virtual point with the tip 20 of the pointer 1 → pressing the switch 4. If the switch 4 is pressed while the virtual point is pointed at 20, the tip 20 of the pointer 1 may be displaced at the moment the switch 4 is pressed. Also, pressing the button is troublesome.
In order to solve this problem, a button switch 4A is attached to the tip of the pointer 1 'so that the LED is turned on when the button switch 4A is pressed.

The button switch 4A provided at the distal end portion 20 'of the pointer 1' is pressed and input by a force acting in the length direction of the first portion 2 'when the distal end portion 20' is pressed against a predetermined part of the body. Thus, each LED is turned on. By attaching the button switch 4A to the tip of the pointer 1 ', the virtual point determiner points to the virtual point, and at the same time, the button switch 4A is pressed, and the virtual point instruction and the button switch 4A are pressed in one action. In other words, the trouble of pressing the switch can be omitted.

The operator holding the device 1 ′ points to a desired virtual point at the tip portion 20 ′ of the first portion 2 ′, and at the same time, the button switch 4A is pressed, and three first markers M ₁ , which are composed of infrared LEDs, The second marker M ₂ and the third marker M ₃ are turned on to recognize the device 1 ′, and the coordinates of the first marker M ₁ , the second marker M ₂ , and the third marker M ₃ are acquired. On the pointer 1 ', the relative position between the tip 20' of the first portion 2 '(the position of the pressed button switch 4A) and the first marker M1, the second marker M2, and the third marker M3 relationship is known, the first marker M _1, the second marker M _2, identifiable information of the position of the distal end portion 20 'of the first portion 2' by using the position of the third marker M ₃ is the storage unit Is remembered. For example, the information corresponding to L1 to L4 in FIG. 4 is stored in the storage unit, and when the distance set (d1, d2, d3) = distance set (D1, D2, D3), Based on the coordinates of the marker, the coordinates of the tip 20 of the first portion 2 ′ (in one aspect, located on the center line in the length direction of the first portion 2) can be acquired. For other details, the description of the above-described embodiment can be used.

[Significance of button switch 4B]
If you try to determine the virtual point of the heel when the camera is only in front, you can see three points on the rigid body attached to the foot (not shown, real marker according to FIG. 6) and three points on the pointer at the same time It is necessary to adjust the direction of the foot. In the case of a subject with a disability, moving the direction of the body requires labor.
In order to solve this problem, the button switch 4B is attached using the vicinity of the center of the first portion 2 'of the pointer 1' as an instruction unit, so that the direction of the foot is not changed even when the camera is only in front. It was possible to determine the virtual point of the cocoon. When a virtual point is pointed using the tip of the device as an instruction unit, the position of the marker close to the tip is hidden by the foot as shown in the left figure of FIG. On the other hand, in the case where the middle part of the device is used as an instruction unit indicating a virtual point, all the markers can be seen from the front as shown in the right diagram of FIG. 8 (the length of the first portion 2 ′ of the pointer 1 ′). The size is larger than the width of the foot, and the markers are located on both sides of the foot). Such a problem is not limited to the case where virtual points are set on the heel or the foot, but is not limited to the case where virtual points are set on the heel or the foot.

As in the case of using the button switch 4A, the virtual point can be determined by one action by pressing the button switch 4B against the desired part, but the length of the first portion 2 ′ of the pointer 1 ′ is near the center in the length direction. It is understood by those skilled in the art that the value of L1 in FIG. 4 may be negative (this negative value is referred to as L1 ′) when the virtual point is determined using as an instruction unit. In the storage unit of the computer, L1 ′ is stored in addition to L1.

As described above, in the pointer 1 ′ shown in FIG. 7, there are two indication portions pointing to the virtual points, specifically, the tip of the first portion 2 ′ and the middle of the first portion 2 ′ in the length direction. The button switch 4A is provided in the former, and the button switch 4B is provided in the latter. The operator can select one of the button switches 4A and 4B according to the part where the virtual point is set, and can instruct the virtual point. When the button switch 4A is selected, the setting value L1 in the computer is used, and when the button switch 4B is selected, the setting value L1 ′ in the computer is used. In a typical example, a virtual point is usually determined by instructing and pressing a desired part with the button switch 4A, and the button switch 4B is used when determining the position of a bag or the like. The present invention includes an aspect in which three or more instruction units are provided and an aspect in which only the button switch 4B is provided.

FIG. 9 shows different aspects of the virtual point setting position in relation to the device. The left figure shows a case where a virtual point is set at the tip of the device, a virtual point is set near the center of the device, and the right figure shows a case where a virtual point is set on the extension of the device axis. . In the case of the left figure, the virtual point coordinates can be calculated from the coordinates of each marker by setting the distance L1 between the tip and the marker (see FIG. 4). In the case of the central view, the virtual point coordinates can be calculated from the coordinates of each marker by setting the distance L1 ′ between the point near the center of the device and the marker (see FIG. 7). In the case of the right figure, a virtual point is set at a predetermined position on the extension line of the axis of the device (in FIG. 1, the first shaft portion 2), and a distance L1 ″ between the virtual point and the marker is set. By doing so, the virtual point coordinates can be calculated from the coordinates of each marker. That is, by setting the value of L1 in FIG. 4 to be larger than the distance between the marker and the tip, a virtual point can be provided, for example, at a position of ◯ mm from the tip of the pointer. By doing so, the virtual point inside the body of the subject can be pointed by the tip of the device, and the virtual point can be set in the body.

1 Pointer (device)
M ₁ 1st marker M ₂ 2nd marker M ₃ 3rd marker (hidden marker)
4 Switch 4A Button switch (indicator)
4B button switch (indicator)
20 End point indicating virtual point (indicator)

Claims (25)

Multiple cameras,
A device comprising: at least three markers; and one or more indication units indicating a virtual point, the position of the virtual point being identifiable by a marker position on the device;
Means for storing information capable of specifying a distance between each marker of the device;
Device detection means for detecting the device using a plurality of camera images photographed by the plurality of cameras and a distance between the markers of the device;
Virtual point coordinate acquisition means for acquiring the coordinates of the virtual point from the marker coordinates using the relative position of each marker on the device and the virtual point;
With
The device detecting means detects the device when the at least three markers are recognized by the camera from a state where at least one of the at least three markers is not recognized by the camera;
The virtual point coordinate acquisition means uses the detection of the device as a decision input, and acquires the coordinates of the virtual point from the marker coordinates at the time of device detection.
Virtual point determination device. At least one of the at least three markers of the device is a hidden marker having a first state not recognized by the camera and a second state recognized by the camera;
The device includes switching means for switching the hidden marker from the first state to the second state when the device indicates a desired virtual point position.
The virtual point determination apparatus according to claim 1. The switching means includes a switch provided on the device,
The switch can be manually operated by an operator holding the device and pointing to a desired virtual point,
Acquisition of virtual point coordinates is executed by remote determination input by switch operation on the device,
The virtual point determination apparatus according to claim 2.   The switch is a push switch provided in the instruction unit, and when the operator points to a virtual point on the object or in the object using the device, the push switch is pushed against the object. The virtual point determination device according to claim 3, wherein the virtual point determination device is input.   The virtual point determination device according to any one of claims 2 to 4, wherein the hidden marker is changed from a first light-off state to a second light-on state by the switching unit.   The hidden marker includes a movable body that is movable between a first position that hides the hidden marker and a second position that exposes the hidden marker, and the movable body conceals the hidden marker by the switching means. The virtual point determination device according to any one of claims 2 and 3, wherein the second state is changed from the first state by moving from the first position to a second position where the hidden marker is exposed. When the virtual point coordinates are acquired with the detection of the device as a decision input, a virtual marker is set by the virtual point coordinates and the coordinates of three or more real markers on the subject.
The virtual point determination apparatus of any one of Claims 1-6.   The virtual point determination apparatus according to claim 1, wherein the one or more instruction units include an instruction unit formed at an end of the device.   The one or more instruction units include an instruction unit formed at a site other than the end of the device, and on the device, at least one marker is provided on the first side across the instruction unit. The virtual point determination device according to any one of claims 1 to 8, wherein a remaining marker is provided on the second side. Multiple cameras,
A device comprising: at least three markers; and one or more indication units indicating a virtual point, the position of the virtual point being identifiable by a marker position on the device;
Device detection means for detecting the device using a plurality of camera images photographed by the plurality of cameras and a known distance between each marker;
Virtual point coordinate acquisition means for acquiring the coordinates of the virtual point from the marker coordinates using the relative position of each marker on the device and the virtual point;
Use
Photographing the device with the plurality of cameras, wherein at least one of the at least three markers is not recognized by the camera;
When at least one of the at least three markers is not recognized by the camera when the desired virtual point position is pointed by the instruction unit of the device, the at least three markers are recognized by the camera. Detecting the device; and
Using the detection of the device as a decision input, obtaining the coordinates of the virtual point from the marker coordinates at the time of device detection;
A virtual point determination method comprising: At least one of the at least three markers of the device is switchable from a first state not recognized by the camera to a second state recognized by the camera;
Detecting the device by switching the hidden marker from the first state to the second state when pointing to a desired virtual point position by the device;
The virtual point determination method according to claim 10. The switching of the hidden marker from the first state to the second state is performed by a switch provided on the device,
The switch can be manually operated by an operator holding the device and pointing to a desired virtual point,
Acquisition of virtual point coordinates is executed by remote determination input by switch operation on the device,
The virtual point determination method according to claim 11.   The switch is a push switch provided in the instruction unit, and when the operator points to a virtual point on the object or in the object using the device, the push switch is pushed against the object. The virtual point determination method according to claim 12, wherein the virtual point determination method is input.   The virtual point determination method according to any one of claims 11 to 13, wherein the device is detected when the hidden marker is changed from a first state of being turned off to a second state of being turned on.   When the movable body in the first position for concealing the hidden marker moves to the second position for exposing the hidden marker, the hidden marker is changed from the first state to the second state, whereby the device The virtual point determination method according to claim 11, wherein: is detected. The plurality of cameras, together with the device, photograph at least a part of a subject on which a real marker is attached, and the camera image includes three or more real markers for specifying a virtual point. ,
When the virtual point coordinates are acquired with the detection of the device as a decision input, a virtual marker is set by the virtual point coordinates and the coordinates of the three or more real markers.
The virtual point determination method according to claim 10.   The virtual point determination method according to claim 10, wherein the one or more instruction units include an instruction unit formed at an end of the device.   The one or more instruction units include an instruction unit formed at a site other than the end of the device, and on the device, at least one marker is provided on the first side across the instruction unit. The virtual point determination method according to claim 10, wherein there is a remaining marker on the second side. A device comprising at least three markers and one or more instruction units indicating a virtual point, and used for determining a virtual point,
The distance between each marker is known, and the position of the virtual point can be specified by the marker position on the device,
One of the at least three markers is a hidden marker having a first state not recognized by the camera and a second state recognized by the camera;
The device comprises switching means for switching the hidden marker from the first state to the second state. The switching means includes a switch provided on the device,
The device of claim 13.   The device according to claim 20, wherein the switch is a push switch provided in the instruction unit.   The device according to any one of claims 19 to 21, wherein the hidden marker is changed from a first light-off state to a second light-on state by the switching unit.   The hidden marker includes a movable body that is movable between a first position that hides the hidden marker and a second position that exposes the hidden marker, and the movable body conceals the hidden marker by the switching means. The device according to any one of claims 19 and 20, wherein the second state is changed from the first state by being moved from the first position to a second position at which the hidden marker is exposed.   The device according to any one of claims 19 to 23, wherein the one or more instruction units include an instruction unit formed at an end of the device.   The one or more instruction units include an instruction unit formed at a site other than the end of the device, and on the device, at least one marker is provided on the first side across the instruction unit. 24. A device according to any one of claims 19 to 23, wherein there is a remaining marker on the second side.