JP2021096607A - Motion capture system, motion capturing method and program - Google Patents

Abstract

To facilitate easy and precise measurement of a position of a measuring point of a subject person in a given space and of a history thereof.SOLUTION: A motion capture system 10 includes: a standard marker unit 12 that is placed in an imaging space SP; a reference marker 13 that is placed on a subject person 11; a camera 14 that obtains an image including the standard marker unit 12 and the reference marker 13; and an image processing device 15. The image processing device 15 obtains, on the basis of the picked-up image of the standard marker unit 12, standard coordinate values representing a position of the camera 14 in a standard coordinate that is set on the basis of the standard marker unit 12, obtains reference coordinate values representing a relative position of the reference marker 13 relative to imaging means on the basis of the picked-up image of the reference marker 13, and obtains a position of the reference marker 13 in the standard coordinate on the basis of the standard coordinate values and the reference coordinate values.SELECTED DRAWING: Figure 1

Description

The present invention relates to a motion capture system, a motion capture method, and a program.

Motion capture technology is widely used. Motion capture technology is a technology that captures the three-dimensional movement of people and objects and digitizes the movement.

Initially, motion capture technology was used, for example, to capture the movement of a person, digitize it, and move the CG character realistically based on the data. However, these days, it is also used for improving sports performance, reducing the burden of rehabilitation after surgery on joints, and analyzing human behavior.

For example, in Patent Document 1, in order to improve the performance of a golfer, spherical markers are attached to a large number of observation points of the golfer, the golfer swings in this state, and the movement is photographed by a plurality of cameras. The system to analyze is disclosed.

Japanese Unexamined Patent Publication No. 2019-07818 JP 2015-194732

In the system disclosed in Patent Document 1, it is necessary to install a large number of spherical markers at the measurement points of the subject, measurement in a dedicated studio is required, and it takes time to align between a plurality of cameras. It is expensive, the accuracy of the measurement data is low, and it is expensive.

In view of such a problem, a card-shaped high-precision position marker as disclosed in Patent Document 2 has also been developed. This high-precision position marker can estimate the position and orientation of the marker with high accuracy by performing image analysis of the pattern. As a result, the load of image processing is small and processing can be performed in real time.

When using precision position markers, it is relatively easy to identify the relative positions of the markers and their changes. However, it is still difficult to specify the position and its history in a specific space, that is, the movement of each measurement point in the specific space. In particular, it is not easy to capture motion based on a plurality of images captured by a plurality of imaging devices.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a system, a method, and a program capable of easily and accurately measuring the position of a measurement point of a subject in a certain space. To do.

In order to achieve the above object, the motion capture system according to the first aspect is
It is an image including a reference marker arranged in the imaging space and a reference marker arranged in the subject to be motion-captured, and is set based on the reference marker based on the image of the reference marker in the image. A reference coordinate value specifying means for obtaining a reference coordinate value indicating the position of the imaging means in the reference coordinate space to be obtained, and
A relative reference coordinate value specifying means for obtaining a relative reference coordinate value indicating a relative position of the reference marker with respect to the imaging means in the reference coordinate space based on the image of the reference marker in the image.
A position specifying means for obtaining the position of the reference marker in the reference coordinate space based on the reference coordinate value of the imaging means and the relative reference coordinate value of the reference marker.
To be equipped.

A reference posture specifying means for obtaining a reference posture indicating the posture of the imaging means in the reference coordinate space based on the image of the reference marker in the image.
A relative reference posture specifying means for obtaining a relative reference posture indicating the posture of the reference marker with respect to the imaging means in the reference coordinate space based on the image of the reference marker in the image.
A posture specifying means for obtaining the posture of the reference marker in the reference coordinate space based on the reference posture of the imaging means and the relative reference posture of the reference marker.
May be further provided.

In addition, the motion capture system according to the second viewpoint is
It is an image including a reference marker arranged in the imaging space and a reference marker arranged in the subject to be motion-captured, and is set based on the reference marker based on the image of the reference marker in the image. A reference coordinate value specifying means for obtaining a reference coordinate value indicating the position of the imaging means in the reference coordinate space to be obtained, and
Based on the image of the reference marker in the image, the reference coordinate value specifying means for obtaining the reference coordinate value indicating the position of the imaging means in the reference coordinate space set based on the reference marker, and
A position specifying means for obtaining the position of the reference marker in the reference coordinate space based on the reference coordinate value and the reference coordinate value of the imaging means.
To be equipped.

A reference posture specifying means for obtaining a reference posture indicating the posture of the imaging means in the reference coordinate space based on the image of the reference marker in the image.
A reference posture specifying means for obtaining a reference posture indicating the posture of the imaging means in the reference coordinate space based on the image of the reference marker in the image.
A posture specifying means for obtaining the posture of the reference marker in the reference coordinate space based on the reference posture and the reference posture of the imaging means.
May be further provided.

The reference marker includes, for example, a polyhedron and a plurality of markers arranged on a plurality of surfaces of the polyhedron.

It is desirable that the plurality of markers constituting the reference marker define a plurality of coordinate spaces having substantially the same origin.

The plurality of markers constituting the reference marker have, for example, different identification units, and can be identified by an image captured by the imaging means.

It is desirable that a plurality of the reference markers are arranged in the imaging space.

Further, a means for obtaining a history of the position of the reference marker at different timings in the reference coordinate space may be provided.

For example, the position and orientation of the imaging means are fixed, the reference coordinate value specifying means obtains the reference coordinate value at least once, and the position specifying means is relative to the same reference coordinate value. The position of the reference marker in the reference coordinate space is obtained based on the reference coordinate value or the reference coordinate value.

For example, the position of the imaging means is variable, and the reference coordinate value specifying means and the relative reference coordinate value specifying means or the reference coordinate value specifying means each time the reference coordinate value and the relative reference coordinate value or the relative reference coordinate value or the said. Find the reference coordinate value.

In addition, the motion capture method according to the third viewpoint is
An image obtained by imaging the reference marker and the reference marker placed on the subject to be motion-captured is acquired.
Based on the image of the reference marker, the reference coordinate value indicating the imaging position in the reference coordinate space set based on the reference marker is obtained.
Based on the image of the reference marker, the relative reference coordinate value indicating the position of the reference marker relative to the imaging position is obtained.
Based on the reference coordinate value and the relative reference coordinate value, the coordinates of the reference marker in the reference coordinate space are obtained.

In addition, the motion capture method according to the fourth viewpoint is
An image obtained by imaging the reference marker and the reference marker placed on the subject to be motion-captured is acquired.
Based on the image of the reference marker, the reference coordinate value of the imaging position in the reference coordinate space set based on the reference marker is obtained.
Based on the image of the reference marker, the reference coordinate value indicating the imaging position in the reference coordinate space set based on the reference marker is obtained.
Based on the reference coordinate value and the reference coordinate value, the coordinates in the reference coordinate space of the reference marker are obtained.

In addition, the program related to the fifth viewpoint is
On the computer
A process of acquiring an image obtained by imaging a reference marker and a reference marker placed on a subject to be motion-captured.
A process of obtaining the reference coordinate value of the imaging position in the reference coordinate space set based on the reference marker based on the image of the reference marker.
A process of obtaining a relative reference coordinate value indicating the relative position of the reference marker with respect to the imaging position based on the image of the reference marker.
The process of finding the coordinates in the reference coordinate space of the reference marker based on the reference coordinate value and the relative reference coordinate value,
To execute.

In addition, the program related to the sixth viewpoint is
On the computer
A process of acquiring an image obtained by imaging a reference marker and a reference marker placed on a subject to be motion-captured.
A process of obtaining the reference coordinate value of the imaging position in the reference coordinate space set based on the reference marker based on the image of the reference marker.
A process of obtaining a reference coordinate value indicating the coordinates of an imaging position in the reference coordinate space set based on the reference marker based on the image of the reference marker.
The process of finding the coordinates in the reference coordinate space of the reference marker based on the reference coordinate value and the reference coordinate value,
To execute.

According to the present invention, it is possible to easily and accurately obtain the coordinate values at the coordinates set in the reference space of each measurement point of the subject.

It is a figure which shows the structure of the motion capture system which concerns on embodiment of this invention. (A) is a diagram showing a configuration example of the reference marker unit shown in FIG. 1, and (B) and (C) are diagrams for explaining an arrangement example thereof. It is a figure explaining the physical quantity that can be measured by a marker. (A) and (B) are diagrams for explaining the coordinates and the origin defined by each marker of the reference marker unit shown in FIG. It is a figure for demonstrating the reference marker shown in FIG. It is a block diagram of the image processing apparatus shown in FIG. It is a flowchart of the 1st example of the image analysis processing (motion capture processing) executed by the image processing apparatus shown in FIG. It is a flowchart of the 2nd example of the image analysis processing (motion capture processing) executed by the image processing apparatus shown in FIG. It is a figure which shows an example of the history data of the position and the posture of the reference marker which is output by executing the image analysis process shown in FIG. 7. It is a figure which shows the modification which includes a plurality of cameras. (A) to (D) are diagrams showing a modification of the reference marker unit shown in FIG. 1. (A) and (B) are diagrams showing a modification in which a plurality of reference marker units are arranged.

Hereinafter, the motion capture system, the method, and the program according to the embodiment of the present invention will be described with reference to FIGS. 1 to 8.

The motion capture system 10 according to the present embodiment is a system for capturing the movement (position history) of a plurality of measurement points (observation points) of the subject, and as shown in FIG. 1, the subject 11 The reference marker unit 12 installed in the imaging space SP to which the subject moves, a plurality of reference markers 13 attached to the subject 11, and the subject 11 in a state where the reference marker unit 12 and the reference marker 13 are attached. A camera 14 for capturing images and an image processing device 15 are provided.

The imaging space SP is a space that can be imaged by the camera 14, and the orthogonal coordinates XYZ are set. Note that FIG. 1 shows a part of the imaging space SP, and the imaging space SP is not limited to the rectangular space. The orthogonal coordinates XYZ are examples of reference coordinates which are global coordinates set in the imaging space SP, and in this embodiment, they are set by the reference marker unit 12.

The subject 11 is a person to be extracted from the movement, and can move (walk) in the imaging space SP. The subject 11 is an example of a subject to be motion-captured, and although a human example is shown, it may be an animal, an inanimate object such as a machine or a robot, or a stationary object.

The reference marker unit 12 is placed in a stationary state at a position where the camera 14 can always take an image without obstructing the walking of the subject 11 in the imaging space SP and not being shaded by the movement of the subject 11. Has been done. The reference marker unit 12 serves as a reference for the Cartesian coordinates XYZ set in the imaging space SP and the origin OP thereof. The configuration of the reference marker unit 12 will be described later with reference to FIGS. 2 to 4 (B).

The reference marker 13 is attached to each measurement point (observation point) of the subject 11. Each reference marker 13 generates a coordinate space based on the reference marker 13. The configuration of the reference marker 13 will be described later with reference to FIG.

The camera 14 is an example of an arbitrary imaging means, and is a moving image camera having an imaging space SP as a shooting area. The image output by the camera 14 at each timing includes the image of the reference marker unit 12 and the image of any reference marker 13. The position of the image sensor of the camera 14 is an example of the image pickup position.

The image processing device 15 is composed of a computer, and includes a CPU 151, a program memory 152, a RAM 153, a storage device 154, a display device 155, an input device 156, a communication device 157, and the like, as illustrated in FIG. The image processing device 15 processes the image (image data) supplied from the camera 14, and the position and orientation of each reference marker 13 mounted on the subject 11 within the XYZ coordinates set by the reference marker unit 12. Ask for.

In FIG. 6, the analysis program is stored in the program memory 152. The image data acquired by the camera 14 is stored in the storage device 154. The CPU 151 realizes a necessary function by executing an analysis program using the RAM 153 as a work area.

Next, the configuration of the reference marker unit 12 will be described with reference to FIGS. 2 to 4 (B).
As shown in FIG. 2A, the reference marker unit 12 is composed of a support 24 which is a cube on six sides and markers 20 arranged on two adjacent faces of the support 24. Each marker 20 is composed of a central recognition code unit (identification unit) 21, a circular detection reference unit 23 at four corners, and a VMP (Variable Moire Pattern) unit 22 between the detection reference units 23. The recognition code units 21 on each surface have different patterns, and the recognition code unit 21 can determine which surface is in the reference marker unit 12. In addition, since the recognition code unit 21 is attached, erroneous recognition is reduced. Further, since the VMP unit 22 is provided, not only the position of the marker 20 but also the angle (posture) can be measured with high accuracy, and the position and the posture can be measured by one camera 14.

As in the present embodiment, a reference marker (reference marker) is arranged in a specific space, the reference marker is photographed from the imaging means, and the image is captured in the reference coordinate space set based on the reference marker. In order to obtain the reference coordinates of the means, the angle measurement accuracy is very important. For example, when a viewing angle error of 1 ° occurs in angle detection, the error in the position at a distance of 3 m is 5 cm or more (tan 1 ° × 3 m). Then, it is not suitable for the field of measuring the position and the posture with high accuracy. Therefore, it is preferable that the reference marker unit 12 is equipped with a marker 20 having a VMP unit 22 capable of measuring the position and orientation with high accuracy. Further, by using the reference marker unit 12 as the reference marker, the visibility of the marker 20 is improved, so that there are no restrictions on the arrangement and the number of imaging means, and the measurement becomes easier.

As shown in FIG. 3, each marker 20 can measure the position (coordinates U, V, W) and the posture (angle Roll, Pitch, Yaw) of 6 degrees of freedom by one marker 20 and one camera 14. Is. In FIG. 3, the origin of the coordinates is set at the center of the marker 20 for easy understanding. However, as described above, the origin of the coordinates is common to the plurality of markers 20, and preferably the support 24. It is installed at the top of.

Each marker 20 has unique coordinate axes U, V, W and defines unique coordinates. For example, as shown in FIGS. 4A and 4B, different coordinates are defined for the marker 20A on the front surface and the marker 20B on the upper surface of the reference marker unit 12. However, the origins of both coordinates are set to match. The positions of the origin of the coordinates set by the marker 20A on the front surface and the origin of the coordinates set by the marker 20B on the upper surface may be inside, on the surface, or outside of the support 24. Further, in the case of a surface, it may be on the side of the support 24 or may be an apex, but it is preferably an apex.

As illustrated in FIG. 1, the reference marker unit 12 is arranged so that its origin OP coincides with the reference of the position in the imaging space SP. 2 (B) and 2 (C) show an installation example of the reference marker unit 12.

As illustrated in FIGS. 2B and 2C, the reference marker unit 12 is installed so that one of the vertices V coincides with the corner corner of the installation location such as a space, an area, or a room. To. Specifically, in the case of FIGS. 2B and 2C, the reference marker unit 12 is installed at the corner so that the apex V1 coincides with the corner of the installation location. In this case, the common origin of the two markers 20 of the reference marker unit 12 is preferably the apex V1 that coincides with the corner corner of the installation location. In this way, when measuring the position of the reference marker unit 12, even if the image taken by the camera 14 includes the images of the markers 20 on both surfaces, the same measurement origin is taken so that each marker 20 can be moved. It is possible to prevent the origin of the coordinates to be set from being set with respect to the corner corner of the installation location.

The origin of the reference marker unit 12 is not limited to the vertex V1, as long as the origins of the coordinates set by both markers 20 are the same, and any of V2, V3, V4 and the like may be the origin.

In the reference marker unit 12, the number of faces on which the markers 20 for setting coordinates having the same origin are arranged may be two or more, and all three faces, four faces, five faces or more, and all faces of the polyhedron (for example,). In the case of the support 24, it may be 6 faces). In the case of the cube support 24, at least one surface is the installation surface, so the maximum number of surfaces on which the marker 20 can be detected is five. However, by arranging the marker 20 on the entire surface of the support 24, it is possible to easily install the marker 20 on an object without confirming whether or not the surface is the surface on which the marker 20 is arranged.

The reference marker 13 shown in FIG. 5 has a card shape and is installed at each measurement point of the subject 11, and includes a marker 20 equivalent to that arranged on each surface of the reference marker unit 12. .. As described above, the marker 20 is composed of the central recognition code unit 21, the circular detection reference unit 23 at the four corners, and the VMP unit 22 between the detection reference units 23. The recognition code unit 21 of each reference marker 13 has a pattern different from each other. By associating the recognition code unit 21 of the reference marker 13 with the mounting position, the mounting position can be specified from the read recognition code unit 21. As described above, the position (coordinates U, V, W) and the posture (angle Roll, Pitch, Yaw) of 6 degrees of freedom can be image-measured by one marker 20 and one camera 14.

Next, a method of capturing the movement of each measurement point of the subject 11 will be described using the motion capture system 10 according to the above embodiment.
(Advance preparation)
First, the reference marker unit 12 is prepared.
Here, as shown in FIG. 2A, a hexahedron 24 having markers 20 arranged on two or more faces is prepared. The reference marker unit 12 is set so that the origins of the UVW coordinates set by the plurality of markers 20 coincide with each other.

Next, the reference marker unit 12 is installed so that the origin is located at the origin of the XYZ coordinates of the imaging space SP. Next, the camera 14 is installed, and its position and viewing direction are adjusted. The camera 14 is adjusted so that at least two markers 20 of the installed reference marker unit 12 can take a picture.

In parallel, a reference marker 13 is attached to each measurement point of the subject 11. The recognition code unit 21 of the reference marker 13 and the attachment position to the subject 11 are associated with each other and registered in the storage device 154 of the image processing device 15.

(Shooting process)
When the above processing is completed, the imaging process is started.
First, the camera 14 is activated, and the subject 11 wearing the reference marker 13 moves in the imaging space SP. The camera 14 captures an image at an angle of view including the subject 11 and the reference marker unit 12. Shooting may be continuous or intermittent. The camera 14 supplies the captured image data to the image processing device 15, and the image processing device 15 receives the captured image data via the communication device 157 and sequentially stores the captured image data in the storage device 154.

(Image analysis processing)
The CPU 151 of the image processing device 15 accesses the image data stored in the storage device 154 online or after the imaging is completed, and executes the image analysis process.

The operation of the first example of this image analysis process will be described with reference to FIG. 7A.
First, the CPU 151 extracts a frame image at an arbitrary timing T from the storage device 154 (step S1).

Next, the CPU 151 identifies the image of the reference marker unit 12 in the extracted frame image, and identifies the image of the marker (hereinafter, reference marker) 20 that can be confirmed most clearly. Next, the position (coordinates U, V, W) and posture (angles Roll, Pitch, Yaw) of the camera 14 in the reference coordinate space determined by the reference marker 20 are obtained, and this is set as the reference set in the imaging space SP. Let the coordinate values (X, Y, Z) and angles (Roll, Pitch, Yaw) in the coordinate space be (step S2). At this stage, the CPU 151 identifies the reference coordinate value specifying means and the reference posture for obtaining the reference coordinate value and the reference angle indicating the position and orientation of the imaging means in the reference coordinate space set based on the image of the reference marker 20. It is functioning as a means.

As described above, in this example, the reference coordinate space (U, V, W) set by the reference marker 20 is set as it is as the reference coordinate space (X, Y, Z) which is the reference of the imaging space SP. Hereinafter, the coordinate space set by the reference marker 20 is referred to as a reference coordinate space, the coordinate values in the reference coordinate space are referred to as reference coordinate values, and the angle is referred to as a reference angle.

Next, the CPU 151 takes out one image of the reference marker 13 in the frame image (step S3), and positions (coordinates U, V, W) and attitude (angle) of the camera 14 in the reference coordinate space determined by the reference marker 13. Roll, Pitch, Yaw) is obtained and stored (step S4). Hereinafter, the coordinate space determined by the reference marker 13 is referred to as a reference coordinate space, the coordinate values in the reference coordinate space are referred to as reference coordinate values, and the angle is referred to as a reference angle. At this stage, the CPU 151 obtains a reference coordinate value and a reference posture that indicate the position and orientation of the imaging means in the reference coordinate space set based on the image of the reference marker 13, and the reference coordinate value specifying means and the reference posture specifying means. Is functioning as.

The reference coordinate value (X, Y, Z) of the camera 14 obtained in step S2 = the reference coordinate value (U, V, W) of the camera 14 obtained in step S4. Based on this, the CPU 151 obtains the reference coordinate value and the reference angle of the reference marker 13, that is, the reference coordinate value in the reference coordinate space of the origin of the reference coordinate space and the reference angle of the reference marker 13 (step S5). Therefore, at this stage, the CPU 151 functions as a position specifying means and a posture specifying means for obtaining a reference coordinate value and a reference angle indicating the position and the posture of the reference marker 13 in the reference coordinate space.

Next, the CPU 151 determines whether or not the processing for all the reference markers 13 has been completed (step S6). If the process is not completed (step S6: No), the reference marker 13 is updated to an unprocessed one (step S7), the process returns to step S4, and the same process is repeated.
If the process is completed (step S6: Yes), the current process is terminated.

The CPU 151 displays the obtained data on the display device 155 as necessary, and outputs the measurement point position history information to another device via the communication device 157.

In this way, the reference coordinate value and the reference angle of each reference marker 13 at the timing T can be obtained.

Next, a second example of the image analysis process will be described with reference to FIG. 7B. This example is an example in which the reference coordinate value and the reference angle of each reference marker 13 can be acquired more accurately and in the form of history as compared with the first example shown in FIG. 7A.

First, the CPU 151 sets the initial value TD at the timing T (step S11), and extracts the frame image at the timing T (step S12). The image at the time when the time TD has elapsed can be specified by, for example, the frame number. You may take a clock in the image.

Next, the CPU 151 extracts the clearest marker image in the extracted frame image. Here, it is assumed that the image is a marker with ID = 01, that is, the marker 20A. The CPU 151 obtains the position (coordinates U, V, W) and posture (angle Roll, Pitch, Yaw) of the camera 14 from the image (step S13). In this example, the UVW coordinates of the marker 20 selected in step S13 are set as they are as the XYZ reference coordinates of the imaging space SP. Therefore, the position (coordinates U, V, W) and posture (angles Roll, Pitch, Yaw) of the camera 14 obtained based on the marker 20A are the positions (coordinates X, Y, Z) based on the XYZ coordinates of the imaging space SP. And the posture (angle Roll, Pitch, Yaw) are set as they are. (Step S13). Hereinafter, the coordinate space set by the marker 20A, which is a reference marker (reference marker), will be referred to as a reference coordinate space, the coordinate values will be referred to as reference coordinate values, and the angle will be referred to as a reference angle.

Next, the CPU 151 reads an image of another marker 20 in the image of the reference marker unit 12 in the extracted frame image. Here, it is assumed that the marker with ID = 02, that is, the image of the marker 20B is read. The CPU 151 obtains the position (coordinates U, V, W) and posture (angle Roll, Pitch, Yaw) of the camera 14 from the image (step S14). Subsequently, the CPU 151 uses the reference coordinates of the marker 20B based on the relationship between the UVW coordinates formed by the reference marker 20A and the UVW coordinates formed by the marker 20B (the origin is the same and is rotated by 90 ° or a multiple thereof). The position (coordinates U, V, W) and attitude (angles Roll, Pitch, Yaw) of the camera 14 obtained based on the above are set to the reference coordinate values (X, Y, Z) of the reference coordinates set based on the reference marker 20A. ) And the reference angle (Roll, Pitch, Yaw) (step S14).

Next, the CPU 151 determines whether or not an image of another marker 20 is present in the image of the reference marker unit 12 in the extracted frame image (step S15), and if it is present (step S15: Yes). ), Returning to step S14, the same process is repeated.

On the other hand, when the CPU 151 determines that the image of the other marker 20 does not exist in the image of the reference marker unit 12 in the extracted frame image (step S15: No), the process proceeds to step S16.

In step S16, the CPU 151 reduces the error by averaging the plurality of reference coordinate values (X, Y, Z) and the reference angles (Roll, Pitch, Yaw) acquired in steps S13 and S14. The reference coordinate values (X, Y, Z) and the reference angles (Roll, Pitch, Yaw) of the camera 14 are determined (step S16). By the above processing, the position and orientation of the camera 14 in the reference coordinate space set by the marker 20A are accurately specified by using the plurality of markers 20 of the reference marker unit 12. At this stage, the CPU 151 is a reference coordinate value specifying means and a reference posture specifying means for obtaining a reference coordinate value and a reference angle indicating the position and orientation of the imaging means in the reference coordinate space set based on the image of the reference marker 20A. Is functioning as.

Next, the CPU 151 takes out one image of the reference marker 13 in the frame image, and the position (coordinates U, V, W) and the posture (angle) of the camera 14 in the coordinate space determined by the ID of the reference marker and the reference marker 13. Roll, Pitch, Yaw) is obtained and stored (step S17). At this stage, the CPU 151 is a reference coordinate value specifying means for obtaining a reference coordinate value indicating the position and orientation of the imaging means and a reference posture (reference angle) in the reference coordinate space set based on the image of the reference marker 13. It functions as a reference posture identification means.

The reference coordinate value (X, Y, Z) of the camera 14 obtained in step S16 = the reference coordinate value (U, V, W) of the camera 14 obtained in step S17. Based on this, the CPU 151 obtains the reference coordinate values (X, Y, Z) and the reference angles (Roll, Pitch, Yaw) in the reference coordinate space of the reference marker 13 (step S18). Therefore, at this stage, the CPU 151 functions as a position specifying means and a posture specifying means for specifying the position and the posture of the reference marker 13 in the reference coordinate space.

Next, the CPU 151 determines whether or not the processing for whether or not the image of the reference marker 13 that can be read (unprocessed) remains is completed (step S19). If the image of the reference marker 13 to be processed remains (step S19: Yes), the process returns to step S17, one unprocessed image of the reference marker 13 is selected, and the same process is repeated.

On the other hand, if the processing of the images of all the reference markers 13 is completed (step S19: No), the process proceeds to step S20.

In step S20, the CPU 151 determines whether or not the processing for the frame image at all the timings to be processed is completed (step S20). If the process is not completed (step S20: No), T = T + ΔT (ΔT = minute time) (step S21), the process returns to step S12, and the same process is repeated. If the processing for all timings is completed (step S20: Yes), the current processing is completed.

Thus, as illustrated in FIG. 8, the storage device 154 has timings TD, TD + ΔT, TD + 2 · ΔT. .. .. In, a history of the position and posture of each measurement point of the subject 11 in the reference coordinate space with reference to the position of the reference marker unit 12 is formed.

The CPU 151 displays the measurement point position history information illustrated in FIG. 8 on the display device 155, and outputs the measurement point position history information to another device via the communication device 157, if necessary.

In this way, in the present embodiment, the movement of each measurement point of the subject 11 can be obtained from the history of the reference coordinate values in the reference coordinate space set in the imaging space SP.

In the process of FIG. 7B, when the position and orientation of the reference marker unit 12 are fixed and the position and orientation of the camera 14 are fixed, the values obtained in steps S13 and S14 do not change. Therefore, steps S13 to S15 may be skipped except when the flow is first passed. That is, the reference coordinate value and the reference posture of the camera 14 are obtained at least once, and the position of the reference marker 13 in the reference coordinate space based on the same reference coordinate value and the reference posture and each reference coordinate value and the reference posture. You just have to ask for the posture.

Further, in the above embodiment, the example in which the position of the camera 14 is fixed is shown, but the position of the camera 14 may be fixed or variable. However, when the position and angle of the camera 14 are changed, it is desirable to execute steps S13 to S15 in the first process after the position change. That is, it is desirable to obtain the reference coordinate value and the reference posture and the reference coordinate value and the reference posture each time.

In the above embodiment, in the processing of steps S4, S17, and S18, the reference coordinate values (U, V, W) and the reference angles (Roll, Pitch, Yaw) of the camera 14 are obtained from the image of the reference marker 13. Based on the reference coordinate values (U, V, W) and the reference posture (angle) (Roll, Pitch, Yaw), the reference coordinate values and the reference angles of the reference marker 13 are obtained. The present invention is not limited to this.

In the configuration of FIG. 1, relative reference to the relative reference coordinate value (relative reference position) (ΔX, ΔY, ΔZ) in the reference coordinate space of the reference marker 13 based on the position and orientation of the camera 14 from the image of the reference marker 13. The posture (ΔRoll, ΔPitch, ΔYaw) may be obtained, and the reference coordinate value and the reference posture of the reference marker 13 may be obtained from these values.

Specifically, in the configuration of FIG. 1, the CPU 151 first sets the reference coordinate values (X, Y, Z) and the reference angles (Roll, Pitch, Yaw) of the camera 14 in step S2 or FIG. 7B of FIG. 7A. Find step S14.

Next, the CPU 151 determines the relative reference coordinate values (ΔX, ΔY, ΔZ) and the relative reference attitudes (ΔRoll, ΔPitch) in the reference coordinate space of the reference marker 13 based on the position and orientation of the camera 14 from the image of the reference marker 13. , ΔYaw). The method of obtaining the relative reference coordinate values (ΔX, ΔY, ΔZ) and the relative reference attitudes (ΔRoll, ΔPitch, ΔYaw) is arbitrary. For example, the distance from the camera 14 to the reference marker 13 can be obtained by comparing the size of the quadrangle formed by connecting the detection reference portions 23 at the four corners in the image of the reference marker 13 with the known reference size. .. Further, the direction in the reference coordinates of the reference marker 13 can be known from the inclination of the virtual line from the imaging point of the camera 14 toward the center of the reference marker 13 toward each axis of XYZ, and the distance to the reference marker 13 and the reference marker can be known. The relative reference coordinate values (ΔX, ΔY, ΔZ) can be obtained from the direction toward 13. Further, the posture with respect to the camera 14 can be obtained from the ratio of the lengths of the sides of the quadrangle formed by connecting the detection reference units 23, and the relative reference posture (ΔRoll, ΔPitch, ΔYaw) can be obtained from this. At this stage, the CPU 151 obtains a relative reference coordinate value and a relative reference posture indicating the relative position and posture of the reference marker 13 with respect to the imaging means in the reference coordinate space based on the image of the reference marker 13. It functions as a coordinate value specifying means and a relative reference posture specifying means.

Next, the CPU 151 obtains the reference coordinate value (X, Y, Z) of the camera 14 + the relative reference coordinate value (ΔX, ΔY, ΔZ) = (X + ΔX, Y + ΔY, Z + ΔZ), and uses this as the reference of the reference marker 13. Let it be a coordinate value. Similarly, the reference posture (Roll, Pitch, Yaw) + relative reference posture (ΔRoll, ΔPitch, ΔYaw) = (Roll + ΔRoll, Pitch + ΔPitch, Yaw + ΔYaw) of the camera 14 is obtained, and this is used as the reference angle of the reference marker 13. At this stage, the CPU 151 determines the position and posture of the reference marker 13 in the reference coordinate space based on the reference coordinate value and the reference posture of the camera 14 and the relative reference coordinate value and the relative reference posture of the reference marker 13. It functions as a means and a posture identification means.

Further, in the above embodiment, an example in which one camera 14 is used is shown, but as shown in FIG. 9, the number of cameras 14 may be a plurality of cameras. In this case, it is desirable that each camera 14 always captures the reference marker unit 12 within the angle of view. In this way, the absolute positions of the two cameras 14 in the reference coordinate space can be specified from the image of the reference marker unit 12. Therefore, when superimposing two images taken by a plurality of cameras 14, generating a stereo image from the two images, averaging the positions and orientations of the reference markers 13 obtained from the plurality of images, and the like. , Easy to process. Further, it is possible to take a picture of the reference marker 13 which cannot be taken by one camera 14 without omission by a plurality of cameras 14.

In order to photograph a subject with a plurality of cameras 14 and acquire effective data, it is desirable to arrange the cameras 14 at a certain interval and angle and to photograph the subject from the entire circumferential direction. Therefore, it is desirable that the reference marker unit 12 can also capture (recognize) a plurality of markers from all directions.

Further, when a plurality of cameras 14 are used, it is also possible to specify the reference position and the reference posture of the reference marker 13 in the reference coordinate space by using the stereo technique. The processing in this case is as follows.

First, the CPU 151 obtains the reference coordinate values (X, Y, Z) and the reference posture (Roll, Pitch, Yaw) of each camera 14 based on the image of the marker 20 in the reference marker unit 12.

Next, with respect to the reference marker 13 to be processed, two cameras 14 used for stereo viewing are specified. From the reference coordinate values (X, Y, Z) of the two cameras 14, the distance between the two cameras 14 is obtained for each axial direction.

Next, the parallax of the two cameras 14 with respect to the reference marker 13 is obtained for each axial direction, and the reference coordinate value of the camera 14 and the relative reference coordinate value of the reference marker 13 with respect to the reference posture are relative to the reference coordinate value of the camera 14 from the distance and parallax of the two cameras 14. Find the reference posture.

Next, the reference coordinate value (X, Y) of the reference marker 13 to be processed in the reference coordinate space is added by adding the relative reference coordinate value and the relative reference posture of the reference marker 13 to the reference coordinate value and the reference posture of the camera 14. , Z) and the reference angle (Roll, Pitch, Yaw).

When the positions and orientations of the camera 14 and the reference marker unit 12 are fixed, the processing for obtaining the reference coordinate value and the reference orientation of the camera 14 may be performed only an arbitrary number of times, and the processing may be performed using the same values. Etc. are the same in these modified examples.

The present invention is not limited to the above embodiment, and various modifications and applications are possible. For example, instead of the reference marker unit 12, one marker may be set at a position where it is difficult to hide behind the subject 11. The accuracy is reduced, but it is still measurable.

As the reference marker 13, it is also possible to use a reflective ball or a light emitting ball as described in Patent Document 1. However, a plurality of cameras 14 are arranged, and the reference coordinate value of the light emitting ball is acquired by using the stereo technique described above. In the case of a light emitting ball, the angle cannot be obtained.

There may be a plurality of reference marker units 12. By arranging a plurality of marker units whose positions are fixed, it is possible to take a picture of a moving subject with a moving camera and measure its movement.

The shape and arrangement of the reference marker unit can be changed arbitrarily, and various shapes and arrangements are possible, and it may be selected according to the application.

For example, as the reference marker unit 12, the number of faces on which the markers 20 that define the coordinates having the same origin are arranged may be two or more, and the number of faces of the support 24 is three, four, five or more. All faces (for example, 6 faces in the case of the support 24) may be used. In the case of the cube support 24, since at least one surface of the marker 20 is an installation surface on an object, the maximum number of surfaces on which the marker 20 can be detected is five. However, by arranging the marker 20 on the entire surface of the support 24, it is possible to easily install the marker 20 on an object without confirming whether or not the surface is the surface on which the marker 20 is arranged.

The reference marker unit 12 may be attached to the subject 11 in the same manner as the reference marker 13. Further, a part of the reference marker 13 may be substituted as a reference marker.

10 (A) to 10 (D) show examples of marker units having different support shapes.
In the reference marker unit 12A of FIG. 10A, the markers 20 are arranged on two adjacent surfaces on the support 24A of a right triangle prism. Specifically, in the reference marker unit 12A, one marker 20 is arranged so that one corner of the marker 20 is located in a right-angled region on the upper surface of the right triangle of the support 24A. Further, the other marker 20 adjacent thereto may be arranged in a region indicated by a dotted line, that is, a plane between the bases of the right triangle, a plane between the hypotenuses of the right triangle, and the like. The reference marker unit 12A can be arranged, for example, in the same corner as in FIG. 2B. Specifically, the bottom surface is in contact with the floor, and the side surface extending downward from the hypotenuse of the right triangle is in contact with the wall. Can be arranged to do so. In this case, the reference marker unit 12A preferably has, for example, the apex V1 as the origin. Further, other than the vertex V1, that is, the vertices V2, V3, etc. can be set as the origin OP.

In the reference marker unit 12B of FIG. 10B, markers 20 are arranged on the side surfaces of two adjacent rectangles on the support 24B of a right triangle prism. The reference marker unit 12B can be arranged so that, for example, the surfaces connecting the bases of the right triangles are in contact with the floor or the wall. In this case, the reference marker unit 12B can have, for example, the apex surrounded by a circle in FIG. 10B as the origin OP, and specifically, for example, at the corner where the reference marker unit 12B is aligned. Depending on this, the vertex can be selected as the origin OP.

In the reference marker unit 12C of FIG. 10C, the markers 20 are arranged on two adjacent surfaces of the support 24C including the cubic region and the quadrangular pyramid region. For example, when the object to be installed has a quadrangular pyramid-shaped recess, the reference marker unit 12C can be arranged by inserting the quadrangular pyramid region of the marker unit 2C into the recess. In this case, the reference marker unit 12C preferably has, for example, the apex surrounded by a circle in FIG. 10C as the measurement origin.

In the reference marker unit 12D of FIG. 10D, markers 20 are arranged on two surfaces of the support 24D including a cubic region and a triangular prism region. For example, when the object to be installed has a triangular prismatic recess, the reference marker unit 12D can be arranged by inserting the triangular prism region of the reference marker unit 12D into the recess. In this case, the reference marker unit 12D can have, for example, the apex surrounded by a circle in FIG. 10 (D) as the origin, and for example, the apex can be set as the origin according to the corner where the reference marker unit 12D is aligned. You can choose.

Further, the number of reference marker units arranged in the imaging space SP is not limited to one, and is arbitrary. 11 (A) and 11 (B) show measurement examples using a plurality of reference marker units 12.

FIG. 11A shows a first example in which a plurality of reference marker units 12B are arranged in the imaging space. Specifically, the imaging space SP is a room 40, and a square object 41 is arranged in the room 40. The reference marker unit 12 is arranged so that the vertices coincide with the corners (for example, corner corners) of various boundaries between the wall 401, the floor 402, and the object 41 of the room 40. In this example, it is assumed that the apex of the reference marker unit 12 that coincides with the corner corner is the origin. In such a state, when a plurality of reference marker units 12 are photographed by the camera 14 from a certain direction, the markers 20 on at least one surface of each reference marker unit 12 are reflected. Since the plurality of markers 20 included in one reference marker unit 12 have the same origin in common, for example, the origins of the plurality of markers 20 do not shift in one reference marker unit 12, and the markers It is possible to obtain a measurement result in which the deviation between 20 is suppressed. Further, the reference marker unit 12 can arrange its vertices (or sides) so as to coincide with the vertices (or sides) in space. Therefore, the target position in the space and the origin of the reference marker unit 12 can be easily matched. As a result, the deviation between the origin of the reference marker unit 12 and the target position of the object to be measured is suppressed, and a more accurate measurement result can be obtained.

On the other hand, with a conventional marker unit in which one marker is arranged on one support and the center of the recognition code portion on the surface of the marker is the measurement origin, it is difficult to perform measurement in which the influence of such deviation is suppressed. FIG. 11B shows a schematic view of a state in which the conventional marker unit 100 is arranged at the same position as in FIG. 11A. Each of the marker units 100 has its origin at the center of the recognition code portion on the surface. Therefore, when the target position of the space to be measured is, for example, the corner of the corner, there is a discrepancy between it and the measurement limitation of the marker unit 100. Further, in the image taken by the camera, a marker that does not appear in the camera may occur depending on the position of the camera. In that case, the marker that does not appear in the camera cannot be measured, so that the position information cannot be obtained. On the other hand, in the reference marker unit 12, at least one of the markers 20 on any surface is captured regardless of the direction in which the camera 14 is photographed, and one reference marker unit 12 captures a plurality of markers 20 having the same origin. By having it, such a problem can be easily solved.

Further, in the conventional marker unit having only one marker on one support, depending on how the markers are arranged with respect to the object, some markers are captured by the camera and some are not captured by the camera. There is. Then, since the marker that does not appear in the camera cannot be measured, the position information cannot be obtained. Therefore, in the reference marker unit 12 of the embodiment, by arranging the markers 20 on each of the two or more surfaces of the support 24, at least one of the markers 20 is captured by the camera. Therefore, for example, the camera 14 The degree of freedom of placement can be increased.

The camera 14 is not an essential configuration of the motion capture system 10. If images with different acquisition timings including a stationary reference marker and a moving reference marker are supplied, the camera itself is unnecessary from the system configuration.

Although the present invention has been described above with reference to the embodiments and examples, the present invention can be modified in various ways that can be understood by those skilled in the art within the scope of the invention. In addition, all the contents described in documents such as patent documents and academic documents cited in this specification shall be incorporated into this specification by citation.

10 Motion capture system 11 Subject 12 Reference marker unit 13 Reference marker 14 Camera 15 Image processing device 20, 20A, 20B Marker 24, 24A, 24B, 24C, 24D Support 21 Recognition code unit 22 VMP unit 23 Detection reference unit 24 , 24A, 24B, 24C, 24D support

Claims (15)

It is an image including a reference marker arranged in the imaging space and a reference marker arranged in the subject to be motion-captured, and is set based on the reference marker based on the image of the reference marker in the image. A reference coordinate value specifying means for obtaining a reference coordinate value indicating the position of the imaging means in the reference coordinate space to be obtained, and
A relative reference coordinate value specifying means for obtaining a relative reference coordinate value indicating a relative position of the reference marker with respect to the imaging means in the reference coordinate space based on the image of the reference marker in the image.
A position specifying means for obtaining the position of the reference marker in the reference coordinate space based on the reference coordinate value of the imaging means and the relative reference coordinate value of the reference marker.
Motion capture system with. A reference posture specifying means for obtaining a reference posture indicating the posture of the imaging means in the reference coordinate space based on the image of the reference marker in the image.
A relative reference posture specifying means for obtaining a relative reference posture indicating the posture of the reference marker with respect to the imaging means in the reference coordinate space based on the image of the reference marker in the image.
A posture specifying means for obtaining the posture of the reference marker in the reference coordinate space based on the reference posture of the imaging means and the relative reference posture of the reference marker.
The motion capture system according to claim 1. It is an image including a reference marker arranged in the imaging space and a reference marker arranged in the subject to be motion-captured, and is set based on the reference marker based on the image of the reference marker in the image. A reference coordinate value specifying means for obtaining a reference coordinate value indicating the position of the imaging means in the reference coordinate space to be obtained, and
Based on the image of the reference marker in the image, the reference coordinate value specifying means for obtaining the reference coordinate value indicating the position of the imaging means in the reference coordinate space set based on the reference marker, and
A position specifying means for obtaining the position of the reference marker in the reference coordinate space based on the reference coordinate value and the reference coordinate value of the imaging means.
Motion capture system with.

A reference posture specifying means for obtaining a reference posture indicating the posture of the imaging means in the reference coordinate space based on the image of the reference marker in the image.
A reference posture specifying means for obtaining a reference posture indicating the posture of the imaging means in the reference coordinate space based on the image of the reference marker in the image.
A posture specifying means for obtaining the posture of the reference marker in the reference coordinate space based on the reference posture and the reference posture of the imaging means.
The motion capture system according to claim 3. The reference marker includes a polyhedron and a plurality of markers arranged on a plurality of faces of the polyhedron.
The motion capture system according to any one of claims 1 to 4. The plurality of markers constituting the reference marker define a plurality of coordinate spaces having substantially the same origin.
The motion capture system according to claim 5. The plurality of markers constituting the reference marker have different identification units, and can be identified by the image captured by the imaging means.
The motion capture system according to claim 5 or 6. A plurality of the reference markers are arranged in the imaging space.
The motion capture system according to any one of claims 1 to 7. Further provided is a means for obtaining a history of the positions of the reference markers at different timings in the reference coordinate space.
The motion capture system according to any one of claims 1 to 8. The position and orientation of the imaging means are fixed, the reference coordinate value specifying means obtains the reference coordinate value at least once, and the position specifying means obtains the same reference coordinate value and the individual relative reference coordinates. Find the position of the reference marker in the reference coordinate space based on the value or the reference coordinate value.
The motion capture system according to any one of claims 1 to 9. The position of the imaging means is variable, and the reference coordinate value specifying means and the relative reference coordinate value specifying means or the reference coordinate value specifying means each time the reference coordinate value and the relative reference coordinate value or the reference coordinate. Find the value
The motion capture system according to any one of claims 1 to 9. An image obtained by imaging the reference marker and the reference marker placed on the subject to be motion-captured is acquired.
Based on the image of the reference marker, the reference coordinate value indicating the imaging position in the reference coordinate space set based on the reference marker is obtained.
Based on the image of the reference marker, the relative reference coordinate value indicating the relative position of the reference marker with respect to the imaging position is obtained.
Find the coordinates in the reference coordinate space of the reference marker based on the reference coordinate value and the relative reference coordinate value.
Motion capture method. An image obtained by imaging the reference marker and the reference marker placed on the subject to be motion-captured is acquired.
Based on the image of the reference marker, the reference coordinate value of the imaging position in the reference coordinate space set based on the reference marker is obtained.
Based on the image of the reference marker, the reference coordinate value indicating the imaging position in the reference coordinate space set based on the reference marker is obtained.
Find the coordinates in the reference coordinate space of the reference marker based on the reference coordinate value and the reference coordinate value.
Motion capture method. On the computer
A process of acquiring an image obtained by imaging a reference marker and a reference marker placed on a subject to be motion-captured.
A process of obtaining the reference coordinate value of the imaging position in the reference coordinate space set based on the reference marker based on the image of the reference marker.
A process of obtaining a relative reference coordinate value indicating the relative position of the reference marker with respect to the imaging position based on the image of the reference marker.
The process of finding the coordinates in the reference coordinate space of the reference marker based on the reference coordinate value and the relative reference coordinate value,
A program that executes. On the computer
A process of acquiring an image obtained by imaging a reference marker and a reference marker placed on a subject to be motion-captured.
A process of obtaining the reference coordinate value of the imaging position in the reference coordinate space set based on the reference marker based on the image of the reference marker.
A process of obtaining a reference coordinate value indicating the coordinates of an imaging position in the reference coordinate space set based on the reference marker based on the image of the reference marker.
The process of finding the coordinates in the reference coordinate space of the reference marker based on the reference coordinate value and the reference coordinate value,
A program that executes.

Images