JP5983132B2 - POSITION DETERMINING DEVICE, POSITION DETERMINING SYSTEM, POSITION DETERMINING METHOD, AND PROGRAM - Google Patents

Description

  The present invention relates to a position determination device, a position determination system, a position determination method, and a program.

  2. Description of the Related Art Conventionally, as a method for generating motion information indicating the motion of a subject, a motion capture system that generates motion information based on a plurality of moving image data in which a subject with a marker is captured from a plurality of directions is known. (For example, patent document 1).

Japanese Patent Laid-Open No. 10-222668

  By the way, when generating motion information from an image obtained by imaging a subject from a predetermined direction, if the subject performs a rotation operation or the like and a part of the subject blocks the imaging device and the marker, the position of the marker is changed. There is a problem that detection cannot be detected.

An object of the present invention is to provide a position determination device, a position determination system, a position determination method, and a program capable of determining the positions of all markers attached to a human based on an image obtained by imaging a human from a predetermined direction. Is to provide.

In order to solve the above-described problem, the position determination device of the present invention acquires the image in a position determination device that determines the position of a marker in an image captured by a human with a plurality of markers taken from a predetermined direction. Obtaining means, first detection means for detecting a marker in the image obtained by the obtaining means, second detection means for detecting a face image of the human image included in the image, and the second detection Calculating means for calculating a rotation angle about the axis in a predetermined direction of the face image detected by the means, and based on the rotation angle calculated by the calculation means, the axis in the predetermined direction of the human being a first generating means for generating a rotation information relating to rotational motion to, on the basis of the rotation information, and estimating means for estimating the position of a marker of the non-detected by the first detecting means, before Symbol human all of which are attached Comprising determining means for determining the position of the manufacturers, and the determination means, all using the detection result and the estimated position of the markers by the estimating means of the marker according to the first detecting means, attached to said human The position of the marker is determined .

The position determination system of the present invention includes an imaging device for imaging a plurality of markers attached person from a predetermined direction, and a positioning device according to any one of claims 1 to 5 In the position determination system, the acquisition unit of the position determination apparatus acquires an image captured by the imaging apparatus.

Further, the position determination method of the present invention is a position determination method using a position determination device that determines the position of a marker in an image in which a human with a plurality of markers is imaged from a predetermined direction. Acquiring an image, detecting a marker in the acquired image, detecting a face image of the human image included in the image, and centering an axis in a predetermined direction of the detected face image A step of calculating a rotation angle, a step of generating rotation information related to a rotation operation about the axis of the predetermined direction of the human based on the calculated rotation angle, and the rotation based on the information, estimating a position of the marker undetected, before SL has a step of determining the position of all the markers attached to the person, and determining a position of said all markers , Using the marker detection result obtained by the step of detecting a marker in the acquired image and the marker position estimated by the step of estimating the position of the non-detected marker, all markers attached to the human being are detected. The position is determined .

Further, the program of the present invention is acquired by means of a computer for a position determination device that determines a position of a marker in an image captured by a human with a plurality of markers from a predetermined direction, and that acquires the image. First detection means for detecting a marker in the image, means for detecting a face image of the human image included in the image, and calculating a rotation angle about an axis in a predetermined direction of the detected face image. It means, based on the calculated rotation angle, wherein the predetermined direction means to generate a rotation information relating to the rotational motion about the axis of the person, based on the rotation information, the first detection estimating means for estimating the position of a marker of the non-detected by means, determining means for determining the position of all the markers attached to the person, to function as the determining means, the detection result of the marker according to the first detection means Using the position of the estimated markers by fine said estimating means, and determining the position of all the markers attached to the person.

According to the present invention, it is possible to determine the positions of all markers attached to a human based on an image obtained by imaging the human from a predetermined direction.

It is a block diagram which shows schematic structure of the position determination system of one Embodiment to which this invention is applied. It is a block diagram which shows schematic structure of a position determination apparatus. It is a flowchart which shows an example of the operation | movement which concerns on an operation | movement information generation process. It is a flowchart which shows an example of the operation | movement which concerns on a marker position determination process. It is a figure which shows an example of the to-be-photographed object to which the marker was attached | subjected to several movable parts. It is a figure which shows an example of the correspondence of a face image and the rotation angle of a face image. It is a figure which shows an example of a to-be-photographed image when the to-be-photographed object which rotates is imaged. FIG. 7A is a diagram illustrating a case where the rotation angle of the subject is 0 [degree]. FIG. 7B is a diagram illustrating a case where the rotation angle of the subject is −45 [degrees]. FIG. 7C is a diagram illustrating a case where the rotation angle of the subject is −90 [degrees]. It is a figure which shows the relationship between the to-be-photographed object and imaging device shown in FIG. FIG. 8A is a diagram illustrating a case where the rotation angle of the subject is 0 [degree]. FIG. 8B is a diagram illustrating a case where the rotation angle of the subject is −45 [degrees]. FIG. 8C is a diagram illustrating a case where the rotation angle of the subject is −90 [degrees]. It is a figure which shows an example in case there exists a non-detection marker in the to-be-photographed image which is not performing the rotation operation.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.
FIG. 1 is a block diagram showing a schematic configuration of a position determination system 100 according to an embodiment to which the present invention is applied.
As illustrated in FIG. 1, the position determination system 100 according to the present embodiment includes an imaging device 1 and a position determination device 2.

First, the imaging device 1 will be described.
The imaging device 1 images a subject P (see FIG. 5) in which markers are attached to a plurality of movable parts from a predetermined direction.
Specifically, for example, the imaging device 1 directs the subject so that the optical axis of the imaging lens is directed in a predetermined direction in which the subject P exists (for example, the direction V1; see FIGS. 8A to 8C). It is fixed at a predetermined position where P can be imaged. In the subject P, markers (for example, markers M1 to M16, etc .; see FIG. 5) are attached in advance to a plurality of movable parts.
Then, the imaging apparatus 1 generates image data obtained by imaging the subject P to which the marker is attached and records it on the recording medium C. Specifically, the imaging apparatus 1 continuously captures an image of the subject P for a predetermined time, generates moving image data having a predetermined frame rate, and records it on the recording medium C.
Note that the imaging apparatus 1 may, for example, capture the subject P at a plurality of different timings within a predetermined time, generate still image data captured at each timing, and record the still image data on the recording medium C. .

Here, the marker will be described.
The markers are attached to a plurality of movable parts existing in the subject. For example, the position of the marker varies depending on the movement of the subject P, such as the position indirectly corresponding to the person who is the subject P, the tip of a portion that swings according to the motion of the subject P, or the vicinity of the tip. It is attached to the place of.
Specifically, for example, as shown in FIG. 5, the marker is attached to the face, shoulder, both legs, and both arms on the side (front side) of the subject P. Here, two markers M1 and M2 are attached to the face so as to be lined up and down. One marker M3 is attached to the base of the neck. Further, one marker M4 is attached above the crotch. In addition, one marker M5, M6, and M7 is attached to each of the knee of the right leg, the base of the shin, and the toe. Also, one marker M8, M9, and M10 is attached to the left leg knee, shin base and toe, respectively. Further, one marker M11, M12, and M13 is attached to the base of the right shoulder, right elbow, and right hand, respectively. Also, one marker M14, M15, and M16 is attached to the base of the left shoulder, left elbow, and left hand, respectively.

The marker is, for example, an index having a predetermined shape (for example, circular shape), a predetermined color, or the like, and features that are distinguishable from each other. Various characteristics of the marker such as shape and color are determined so as to facilitate identification with the subject P to which the marker is attached, in particular, visual identification.
In addition, various characteristics of the marker, such as the number of markers attached to the subject P, the positional relationship between the markers, the relationship between the movable parts of the subject P to which each marker is attached, and the like. Information shown (marker information) is recorded in predetermined recording means (not shown) of the position determining device 2.

  Note that a known device can be applied to the imaging device 1, and includes, for example, not only a digital camera having a main function as an imaging function but also a mobile phone having an imaging function that is not a main function.

Next, the position determination device 2 will be described with reference to FIG.
The position determination device 2 is configured by, for example, a personal computer or the like, and reads out the image data recorded by the imaging device 1 from the recording medium C and generates motion information of the subject P.

FIG. 2 is a block diagram illustrating a schematic configuration of the position determination device 2.
As shown in FIG. 2, the position determination device 2 specifically includes a recording medium control unit 201, an image acquisition unit 202, a marker detection unit 203, a rotation angle calculation unit 204, and a rotation information generation unit. 205, a marker position determination unit 206, an operation information generation unit 207, a display unit 208, an operation input unit 209, a central control unit 210, and the like.

  The recording medium control unit 201 is configured so that the recording medium C is detachable, and controls reading of data from the loaded recording medium C and writing of data to the recording medium C. That is, the recording medium control unit 201 reads out image data (for example, moving image data) related to the marker position determination process (described later) from the recording medium C that is detached from the imaging apparatus 1 and attached to the image acquisition unit 202. Output.

Based on the image data output from the recording medium control unit 201, the image acquisition unit 202 is an acquisition unit that acquires image data obtained by imaging a subject P with markers attached to a plurality of movable parts from a predetermined direction. Function.
Specifically, the image acquisition unit 202 acquires, for example, moving image data including a plurality of frame images obtained by continuously capturing the subject P for a predetermined time, and each of the plurality of frame images constituting the moving image data. Are individually extracted as still image data, and still image data corresponding to each frame image is acquired. Here, as shown in FIGS. 7A to 7C and FIG. 9, the still image data includes subject images (for example, FIGS. 7A to 7C and FIGS. 7A and 7B) captured when moving image data is generated. 9 includes subject images P1 to P4) and a marker image attached to the subject P.
When still image data obtained by imaging the subject P with a marker is recorded on the recording medium C, the image acquisition unit 202 acquires the still image data output from the recording medium control unit 201.

  Note that the still image data acquired by the image acquisition unit 202 is, for example, image data including image regions D having respective resolutions (horizontal resolution and vertical resolution) in two directions orthogonal to each other, as shown in FIG. Yes, the image area D includes a subject image (for example, subject image P4). 7A to 7C, the image area D of the still image data is not shown, and only the subject images P1 to P3 are illustrated. However, the subject images P1 to P3 are the same as the image area D in FIG. Contained within.

The marker detection unit 203 functions as a first detection unit that detects a marker in the image acquired by the image acquisition unit 202.
Specifically, the marker detection unit 203 determines the marker based on the difference in pixel values between the marker and its surroundings based on the marker characteristics (for example, visual characteristics such as the shape and color of the marker) indicated by the marker information. Edge detection processing for detecting an outline (edge) and an image area surrounded by the outline is performed. Then, the marker detection unit 203 outputs the marker detection result to the marker position determination unit 206.
In addition, the marker detection method by the marker detection unit 203 is an example, and is not limited thereto, and can be arbitrarily changed as appropriate.
Further, since the edge detection process is a known technique, detailed description thereof is omitted here.

The rotation angle calculation unit 204 calculates a rotation angle about an axis in a predetermined direction of at least a part of the subject image included in the image.
Specifically, the rotation angle calculation unit 204 includes a face image detection unit 204a, and calculates a rotation angle around the axis in a predetermined direction of the face image detected by the face image detection unit 204a.

The face image detection unit 204a functions as a second detection unit that detects a face image of the subject image.
Specifically, the face image detection unit 204a generates a plurality of detection candidate areas of a predetermined size (for example, 24 × 24 [pixel]) as face image candidate areas from the image acquired by the image acquisition unit 202. . Then, for example, the face image detection unit 204a calculates the degree of similarity between the image data of each detection candidate region and the image data of a predetermined reference image using output calculation by a predetermined algorithm (for example, adaboost (adaboost) or the like). Then, an image of a detection candidate area whose calculated similarity is equal to or greater than a predetermined threshold is detected as a face image.

The image data of the predetermined reference image is, for example, various identification information for determining whether or not the detection candidate area is a face image, recorded in a predetermined recording unit (not shown). Examples of identification information include information based on shapes such as “human eyes”, “human nose”, “human mouth”, and bright (high-pitched) colors with high brightness and saturation. For example, a color tone such as whether or not it is a skin color, and a color size based on the ratio of the entire angle of view to the image (for example, half or more).
Note that the face image detection method by the face image detection unit 204a is an example and is not limited to this, and can be arbitrarily changed as appropriate.

The rotation angle calculation unit 204 functions as a calculation unit that calculates a rotation angle around an axis in a predetermined direction of at least a part of a subject image (for example, a face image) included in an image.
Specifically, the rotation angle calculation unit 204, for example, includes a plurality of feature points (for example, “human eye”, “human nose”, “human mouth” of the face image detected by the face image detection unit 204a. Etc.) is detected by the feature point detection process, and the correlation indicating the positional relationship between the feature points is specified. Then, the rotation angle calculation unit 204 has a direction perpendicular to the ground on the basis of a result of matching between the correlation between the feature points specified in the face image detected by the face image detection unit 204a and predetermined correlation data. The rotation angle of the face image around the axis is calculated. Further, the rotation angle calculation unit 204 outputs information indicating the calculated rotation angle to the rotation information generation unit 205.
Here, the predetermined correlation data is recorded in, for example, a predetermined recording unit (not shown), and the rotation angle of the face image and the feature points of the face image that change in accordance with the rotation angle of the face image. This is data including various types of information related to the calculation of the rotation angle of the face image, such as information associated with the correlation of the positional relationship (see FIG. 6). Note that the rotation angle of the face image captured with the face of the subject P facing the front of the lens of the imaging device 1 is set as a reference (0 [degree]), and one rotation direction (for example, FIG. 6). The right rotation direction) is the + rotation direction, and the other rotation direction (for example, the left direction in FIG. 6) is the − rotation direction. Can be changed.

Moreover, the calculation method of the rotation angle by the rotation angle calculation part 204 is an example, and is not restricted to this, It can change arbitrarily arbitrarily.
Since the feature point detection process is a known technique, detailed description thereof is omitted here.

The rotation information generation unit 205 generates first rotation information related to the rotation operation about the axis in the predetermined direction of the subject P based on the rotation angle calculated by the rotation angle calculation unit 204. Functions as a means.
Specifically, the rotation information generation unit 205 determines, for example, whether or not the subject P is rotating about an axis perpendicular to the ground (for example, the axis O; see FIG. 8A and the like). Rotation information including information indicating information indicating the rotation angle of the subject P, and the like is generated. Then, the rotation information generation unit 205 outputs the generated rotation information to the marker position determination unit 206.

Further, the rotation information generation unit 205 determines whether the subject P is rotated based on the rotation angle calculated by the rotation angle calculation unit 204. Specifically, the rotation information generation unit 205 rotates the subject P based on a comparison result between the rotation angle value calculated by the rotation angle calculation unit 204 and a predetermined angle value, for example. It is determined whether or not. Here, when it is determined that the subject P is rotating, the rotation information generation unit 205 includes rotation information that includes the rotation angle calculated by the rotation angle calculation unit 204 as the rotation angle of the subject P. Is generated. On the other hand, when it is determined that the subject P has not rotated, the rotation information generation unit 205 does not include the rotation angle calculated by the rotation angle calculation unit 204 as the rotation angle of the subject P. Is generated.
The predetermined angle can be arbitrarily changed as appropriate. For example, the rotation information generation unit 205 may determine that the subject P is rotating when the absolute value of the rotation angle exceeds 0 [degrees], or a predetermined value with a larger absolute value. It may be determined whether or not the subject P is rotating based on the angle (for example, 45 [degrees]).

The marker position determination unit 206 determines a position of all markers attached to the subject P based on the marker detection result by the marker detection unit 203 and the rotation information generated by the rotation information generation unit 205. Function as.
Specifically, the marker position determination unit 206 determines the position of the marker detected by the marker detection unit 203. Here, the marker position determination unit 206 detects the image region position of the marker with respect to the entire image region (for example, the image region D; see FIG. 9) of the image acquired by the image acquisition unit 202, for example. The marker position determination unit 206 uses the detected marker position as position information in a predetermined format (for example, coordinate information using a predetermined point in the image region D as an origin).
The marker position determination unit 206 includes a first determination unit 206a, a second determination unit 206b, and a position estimation unit 206c.

The first determination unit 206a functions as a first determination unit that determines whether all markers have been detected by the marker detection unit 203.
Specifically, the first determination unit 206a acquires the marker detection result output from the marker detection unit 203, and specifies the number of markers detected by the marker detection unit 203. Then, the first determination unit 206a, based on the comparison result between the number of identified markers and the number of markers indicated by the marker information, all markers attached to the subject P (for example, markers M1 to M16, etc.) ) Is detected by the marker detection unit 203.
If it is determined that all the markers have been detected, the marker position determination unit 206 determines the positions of all the markers attached to the subject P based on the marker detection result by the marker detection unit 203 ( (See FIGS. 7A and 7B). That is, the marker position determination unit 206 determines the position of the marker by detecting the position of each marker detected by the marker detection unit 203.

  On the other hand, when it is determined that all the markers are not detected, that is, when it is determined that there are undetected markers, the marker position determination unit 206 detects the marker detection result and the rotation information by the marker detection unit 203. Based on the rotation information generated by the generation unit 205, the positions of all markers attached to the subject P are determined (see FIG. 7C). Specifically, for example, the marker position determination unit 206 estimates the position of the non-detected marker by the position estimation unit 206c based on the determination process related to the rotation operation of the subject P by the second determination unit 206b and the determination result. Processing is performed sequentially.

The second determination unit 206b functions as a second determination unit that determines whether or not the subject P is rotating based on the rotation information.
Specifically, the second determination unit 206b acquires the rotation information output from the rotation information generation unit 205, and the subject rotates depending on whether or not the rotation information includes information indicating the rotation of the subject P. Determine if it is moving. For example, as shown in FIG. 8C, when the rotation angle of the subject P is −90 [degrees], the rotation angle of the face image is −90 [for the subject image P3 (FIG. 7C). Degrees] is calculated by the rotation angle calculation unit 204. Then, since the rotation information including the rotation angle is generated by the rotation information generation unit 205, the second determination unit 206b determines that the subject is rotating.

If it is determined that all the markers have not been detected, the position estimation unit 206c estimates the position of the non-detected marker based on the rotation information.
That is, if it is determined that all the markers are not detected by the marker detection unit 203 and the subject P is rotating, the position estimation unit 206c is not detected based on the rotation information. It functions as an estimation means for estimating the position of the marker.
Specifically, for example, as shown in FIG. 8C, the position estimation unit 206c detects the markers M1, M7, etc., detected by the marker detection unit 203, and at the marker position determined by the marker position determination unit 206. Based on this, the front position of the subject P where the non-detected marker is estimated to exist (for example, the position indicated by the line L1) is estimated. Further, the position estimation unit 206c estimates the direction (for example, the direction V2) in which the front side of the subject is facing based on the rotation information. Then, the position estimation unit 206c determines a position that is most likely to contain a non-detected marker from the positional relationship between the markers indicated by the marker information, and the like (for example, markers M3, M6, etc.). ) For each.
In FIG. 7C, the positions of the markers M3 and M6 are indicated by arrows. In FIG. 8C, the detected markers M1 and M7 are indicated by black dots, and the markers M11 to M16 whose positions are estimated are indicated by white dots.

Further, when the second determination unit 206b determines that the subject P is not rotating, the position estimation unit 206c estimates the position of the non-detected marker based on the detected marker.
For example, as shown in FIG. 9, for example, when the rotation angle of the face image in the subject image P4 is 0 [degree], some markers (for example, the marker M4) are moved to the body of the subject P by the operation of the subject P. If the image is hidden by a part of the image and not captured, the first determination unit 206a determines that all the markers have not been detected by the marker detection unit 203, and the second determination unit 206b Judge that it is not moving. In this case, the position estimating unit 206c, for example, the positional relationship of the three markers M3, M5, and M8 adjacent to the marker M4 (for example, the triangle T1 shown in FIG. 5) and each of the three markers The positional relationship of the marker M4 is acquired from the marker information. Further, the position estimation unit 206c detects the marker M4 from the non-detected subject image P4, and the positional relationship between the markers M3, M5, and M8 whose positions are determined by the marker position determination unit 206 (for example, the triangle T2 shown in FIG. 9). ) To get. Then, the position estimating unit 206c estimates the position of the marker M4 in the subject image P4 based on these positional relationships.

  The marker position determination unit 206 determines the positions of all markers using the positions of undetected markers estimated by the position estimation unit 206c. That is, the marker position determination unit 206 determines the position of the undetected marker estimated by the position estimation unit 206c as a position where the undetected marker will exist, and each of the markers detected by the marker detection unit 203. The positions of all the markers are determined together with the detection results of the positions.

  The marker position determination unit 206 determines the positions of all markers attached to the subject P for a plurality of images captured at different timings within a predetermined time. Here, the position estimating unit 206c is different from the one of the plurality of images in that the marker corresponding to the non-detected marker in any one of the plurality of images by the marker detecting unit 203 is different from the one of the plurality of images. When detected from an image, the position of a non-detected marker may be estimated based on the position of the marker detected from another image.

Here, a case will be described in which the positions of all the markers are determined for a plurality of still image data obtained by imaging the subject P to be rotated at the respective timings shown in FIGS. In this case, the subject images of the plurality of still image data acquired by the image acquisition unit 202 are subject images P1 to P3 shown in FIGS.
When the subject P rotates from the timing of FIG. 8A to the timing of FIG. 8B by −45 [degree] from the rotation angle of 0 [degree] around the axis O, the rotation operation and rotation Rotation information indicating the angle is generated. In this state, as shown in FIGS. 7A and 7B, all the markers attached to the subject P are detected by the marker detection unit 203 based on the subject images P1 and P2.
Further, when the subject P further rotates by −45 [degree] from the timing of FIG. 8B to the timing of FIG. 8C, the rotation angle becomes −90 [degree], and the rotation operation and Rotation information indicating the rotation angle is generated. In this state, as shown in FIG. 7C, markers (for example, markers M3, M6, M11 to M16) that are not detected by the marker detection unit 203 are generated based on the subject image P3. In this case, for example, the position estimation unit 206c uses the marker in the subject image P2 detected by the marker detection unit 203 as a reference, and based on the degree of change in the marker position changed from the subject image P1 to the subject image P2. The position of the non-detected marker may be estimated in the image P3.

In addition, the marker position determination unit 206 outputs information (marker position information) indicating the position of each marker for all markers whose positions have been determined.
Specifically, the marker position determination unit 206 uses, for example, each marker position as coordinate information in a coordinate area with one point in a predetermined area as an origin, and includes coordinate information corresponding to all marker positions. The obtained information is output to the motion information generation unit 207 as marker position information. Here, the coordinate area may be, for example, for managing coordinates corresponding to a three-dimensional space, or for managing coordinates corresponding to other dimensions (for example, two-dimensional). There may be.

The motion information generation unit 207 functions as a second generation unit that generates motion information indicating the motion of the subject P based on the positions of all the markers related to the plurality of images determined by the marker position determination unit 206.
Specifically, the motion information generation unit 207 uses the marker position information output from the marker position determination unit 206 for each of a plurality of still image data, and the pose of the subject P at the timing when each still image data is captured. This data is shown. Then, the motion information generation unit 207 generates data (motion data) including the motion information by associating data indicating the pose of the subject P at each timing along the time axis of the imaging timing of each still image data. To do.
Further, the motion information generation unit 207 records the motion data in a recording device such as the recording medium C.

The display unit 208 includes a display such as an LCD or a CRT (Cathode Ray Tube), for example, and displays various information on the display screen under the control of the CPU of the central control unit 210.
That is, the display unit 208 displays, for example, still image data acquired by the image acquisition unit 202. The display unit 208 displays information indicating the marker detected by the marker detection unit 203. The display unit 208 displays information indicating the positions of all the markers determined by the marker position determining unit 206. Further, when there is a non-detected marker, the display unit 208 displays the position of the marker whose position is estimated by the marker position determination unit 206 and the position corresponding to the marker detected by the marker detection unit 203 in a distinguishable manner. You may do it. Further, the display unit 208 may display the movement of the subject P indicated by the operation information side by side at each timing, or reproduces the operation of the subject P according to the operation information using screen transition such as a slide show. Display may be performed.

The operation input unit 209 includes, for example, a keyboard or mouse configured with data input keys for inputting numerical values, characters, and the like, up / down / left / right movement keys for performing data selection, feeding operations, and the like, and various function keys And a key press signal and a mouse operation signal pressed by the user are output to the CPU of the central control unit 210.
Note that a touch panel (not shown) may be disposed on the display screen of the display unit 208 as the operation input unit 209, and various instructions may be input according to the touch position of the touch panel.

The central control unit 210 controls each unit of the position determination device 2. Specifically, the central control unit 210 includes a CPU, a RAM, and a ROM (all not shown), and performs various control operations according to various processing programs (not shown) for the position determination device 2 recorded in the ROM. . At that time, the CPU stores various processing results in a storage area in the RAM, and displays the processing results on the display unit 208 as necessary.
The RAM includes, for example, a program storage area for expanding a processing program executed by the CPU, a data storage area for storing input data, a processing result generated when the processing program is executed, and the like.
The ROM stores a program stored in the form of computer-readable program code, specifically, a system program that can be executed by the position determination device 2, various processing programs that can be executed by the system program, and these various processing programs. Records data used for execution.

Next, operation information generation processing by the position determination device 2 will be described with reference to FIGS.
FIG. 3 is a flowchart illustrating an example of an operation related to the operation information generation process. FIG. 4 is a flowchart illustrating an example of an operation related to the marker position determination process.

First, the marker position determination unit 206 sets a counter N that counts the number of images to be processed with a predetermined initial value (for example, N = 1) (step S1).
Next, the image acquisition unit 202 acquires an image in which the subject P with markers attached to a plurality of movable parts is captured from a predetermined direction (step S2). Specifically, for example, the image acquisition unit 202 individually extracts each of a plurality of frame images constituting the moving image data output from the recording medium control unit 201 as still image data, and corresponds to each frame image. Get still image data.

  Next, the marker detection unit 203 detects a marker included in the image acquired by the image acquisition unit 202 (step S3). Specifically, the marker detection unit 203 extracts, for example, an image region indicating the marker feature based on a feature (for example, the shape or color of the marker) predetermined for identifying the marker. Identify the markers included in the image.

Next, the face image detection unit 204a detects a face image included in the image acquired by the image acquisition unit 202 (step S4). Specifically, the face image detection unit 204a, for example, selects a plurality of detection candidate areas of a predetermined size (for example, 24 × 24 [pixel]) as candidate face image areas from the image acquired by the image acquisition unit 202. Generate and calculate the similarity between the image data of each detection candidate region and the image data of the predetermined reference image by using an output calculation by a predetermined algorithm (for example, adaboost), and the calculated similarity is An image of a detection candidate area that is equal to or greater than a predetermined threshold is detected as a face image.
Next, the rotation angle calculation unit 204 calculates a rotation angle around the axis in a predetermined direction of the face image detected by the face image detection unit 204a (step S5). Specifically, the rotation angle calculation unit 204, for example, includes a plurality of feature points (for example, “human eye”, “human nose”, “human mouth” of the face image detected by the face image detection unit 204a. )) Is detected by the feature point detection process, the correlation indicating the positional relationship between the feature points is specified, and the face image is detected based on the result of matching between the correlation of the specified face image and predetermined correlation data. For the face image detected by the unit 204a, the rotation angle about the axis in the direction perpendicular to the ground is calculated (see FIG. 6).

  Next, the rotation information generation unit 205 generates rotation information related to the rotation operation about the axis in the predetermined direction of the subject P based on the rotation angle calculated by the rotation angle calculation unit 204. (Step S6). Specifically, the rotation information generation unit 205 includes, for example, rotation information including information indicating whether or not the subject P is rotating about the axis perpendicular to the ground and the rotation angle of the subject P. Is generated.

Next, the marker position determination unit 206 performs marker position determination processing (step S7).
Hereinafter, the marker position determination unit 206 will be described with reference to the flowchart of FIG.
First, the marker position determination unit 206 determines the position of the marker detected by the marker detection unit 203 (step S21). Specifically, the marker position determination unit 206 detects, for example, the image region position of the marker with respect to the entire image region of the image acquired by the image acquisition unit 202, and the detected marker position is determined as position information in a predetermined format. (See FIG. 9 and the like).

Next, the first determination unit 206a determines whether or not all markers attached to the subject P have been detected by the marker detection unit 203 (step S22). Specifically, for example, the first determination unit 206a compares the number of markers already detected by the marker detection unit 203 with the number of markers detected by the marker detection unit 203, and based on the comparison result, It is determined whether or not the marker is detected.
When it is determined in step S22 that all markers attached to the subject P have been detected by the marker detection unit 203 (step S22; YES), the marker position determination unit 206 determines the marker detection result by the marker detection unit 203. Based on the above, the positions of all markers attached to the subject P are determined (step S23). That is, the marker position determination unit 206 determines the position of the marker by detecting the position of each marker detected by the marker detection unit 203.

On the other hand, when it is determined in step S22 that all markers attached to the subject P have not been detected by the marker detection unit 203 (step S22; NO), the marker position determination unit 206 determines that the value of the counter N is It is determined whether or not 1 (N = 1) (step S24). If it is determined that N = 1 (step S24; YES), the second determination unit 206b determines whether or not the subject P is rotating based on the rotation information (step S25). . In other words, the second determination unit 206b determines whether or not the rotation information indicates that the subject is rotating.
If it is determined in step S25 that the subject P is rotating based on the rotation information (step S25; YES), the position estimation unit 206c determines the non-detected marker based on the rotation information. The position is estimated (step S26). Specifically, the position estimation unit 206c, for example, the position estimation unit 206c is detected by the marker detection unit 203 among all the markers attached to the subject P, and the marker position determined by the marker position determination unit 206. In addition, the position of the non-detected marker is estimated from the rotation angle of the subject P indicated by the rotation information and various information related to the marker indicated by the marker information (see FIGS. 7C and 8C).
On the other hand, when it is determined in step S25 that the subject P is not rotating based on the rotation information (step S25; NO), the position estimation unit 206c is not detected based on the detected marker. The position of the marker is estimated (step S27). Specifically, the position estimation unit 206c acquires, for example, a positional relationship between a marker that is not detected by the marker detection unit 203 and a marker that is detected by the marker detection unit 203, and the positional relationship and the marker detection unit 203 Based on the detected marker position determined by the marker position determining unit 206, the position of the non-detected marker is estimated (see FIG. 9).
After the process of either step S26 or step S27, the marker position determination unit 206 determines the positions of all markers using the positions of the non-detected markers estimated by the position estimation unit 206c (step S23). That is, the marker position determination unit 206 determines the position of the undetected marker estimated by the position estimation unit 206c as a position where the undetected marker will exist, and each of the markers detected by the marker detection unit 203. The positions of all the markers are determined together with the detection results of the positions.
Upon completion of the process in step S23, the marker position determination unit 206 completes the marker position determination process.

  Returning to FIG. 3, after the marker position determination process (step S7) is completed, the marker position determination unit 206 determines whether or not the determination of the positions of all the markers has been completed for all the still image data to be processed (step S7). Step S8). Here, when it is determined that determination of the positions of all the markers has not been completed for all the still image data to be processed (step S8; NO), the marker position determination unit 206 changes the value of the counter N. (For example, +1) is performed (step S9), and the process proceeds to step S2.

In the marker position determination process (see FIG. 4) performed after the count process of step S9 is performed once or more, it is determined in step S24 that the value of the counter N is not 1 (step S24; NO). In this case, the second determination unit 206b determines whether or not the subject P is rotating based on the rotation information (step S28).
If it is determined in step S28 that the subject P is rotating based on the rotation information (step S28; YES), the position estimation unit 206c determines the position of the rotation information and the marker of another image. Based on this, the position of the non-detected marker is estimated (step S29). Specifically, for example, when performing the processing in the same flow as the processing in step S26, the position estimation unit 206c is further undetected in any one of the plurality of images by the marker detection unit 203. When a marker corresponding to a marker is detected from another image different from one of the plurality of images, the position of the non-detected marker is estimated based on the position of the marker detected from the other image (Refer to Drawing 7 (a)-(c) and Drawing 8 (a)-(c)).
On the other hand, when it is determined in step S28 that the subject P is not rotated based on the rotation information (step S28; NO), the position estimation unit 206c detects the detected marker and the marker of another image. The position of the non-detected marker is estimated based on the position (step S30). Specifically, for example, when performing the processing in the same flow as the processing in step S27, the position estimation unit 206c is further undetected in any one of the plurality of images by the marker detection unit 203. When a marker corresponding to a marker is detected from another image different from one of the plurality of images, the position of the non-detected marker is estimated based on the position of the marker detected from the other image .
After the processing in either step S29 or step S30, the marker position determination unit 206 determines the positions of all markers using the positions of the non-detected markers estimated by the position estimation unit 206c (step S23), Complete the position determination process.

Returning to FIG. 3, after the marker position determination process (step S7) is completed, the marker position determination unit 206 determines whether or not the determination of the positions of all the markers has been completed for all the still image data to be processed (step S7). Step S8). Here, when it is determined that all the marker positions have been determined for all the still image data to be processed (step S8; YES), the motion information generation unit 207 is determined by the marker position determination unit 206. Based on the positions of all the markers related to the plurality of images, motion information indicating the motion of the subject P is generated (step S10). Specifically, the motion information generation unit 207, for example, displays the marker position information output by the marker position determination unit 206 for each of a plurality of still image data upon completion of the marker position determination process. Data including motion information by associating data indicating the pose of the subject P at each timing along the time axis of the imaging timing of each still image data as data indicating the pose of the subject P at the captured timing ( Motion data).
Upon completion of the process in step S10, the position determination device 2 completes the motion information generation process.

As described above, according to the position determination system 100 of the present embodiment, the positions of all the markers attached to the subject P are determined based on the marker detection result and the rotation information. The position of the marker can be determined on the basis of the information related to. That is, when the subject P is imaged from one direction, all markers attached to the subject P may not be included in the image and all the markers may not be detected. In this case, no marker is detected. In addition, the position of the marker can be determined by taking into consideration the rotation state of the subject P (the presence or absence of a rotation operation, the rotation angle, etc.). Therefore, the positions of all markers attached to the subject P can be determined based on an image obtained by capturing the subject P from a predetermined direction.
Furthermore, the positions of all the markers can be determined at a lower cost compared to the case where the subject is imaged from a plurality of directions as in the prior art.

In addition, when it is determined that all the markers have not been detected, the position of the non-detected marker is estimated based on the rotation information. Therefore, when there is an undetected marker, the rotation information indicates the position. Based on the presence / absence and rotation angle of the subject P, the position of the non-detected marker can be determined by estimating the marker position, whereby the positions of all the markers attached to the subject P can be determined. Can be determined.
Further, when it is determined that the subject P is rotating, the position of the non-detected marker is estimated based on the rotation information. Therefore, the position of the marker is determined based on whether or not the subject P is rotating. It is possible to estimate, and the position of the non-detected marker whose position estimated by the presence / absence of the rotating motion changes can be estimated with higher accuracy.

  In addition, when a marker corresponding to an undetected marker is detected from another image different from one image of the plurality of images in any one of the plurality of images, Since the position of the non-detected marker is estimated based on the position of the detected marker, the position of the non-detected marker can be estimated based on the position of the marker existing in another image. The position of the marker can be estimated with higher accuracy.

  In addition, since the motion information indicating the motion of the subject P is generated based on the positions of all the markers related to the determined plurality of images, the motion information is further reduced based on the image obtained by capturing the subject P from a predetermined direction. Operation information can be generated at a cost.

  Also, since the rotation angle about the axis of the face image in the predetermined direction is calculated, for example, the rotation angle can be calculated using a face image having feature points such as eyes, nose and mouth. The moving angle can be calculated more appropriately.

  The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.

For example, in the above-described embodiment, the imaging device 1 and the position determination device 2 are provided separately. However, the position determination device 2 includes the same imaging unit as the imaging device 1, and images captured by the imaging unit are displayed. You may make it acquire.
When the position determination device 2 includes the imaging unit, a unit (for example, the recording medium C) for transferring data between the imaging unit (for example, the imaging device 1) and the acquisition unit (for example, the image acquisition unit 202). And the delivery work can be omitted, and the processing for determining the positions of all the markers can be performed more efficiently.

In the above embodiment, data is exchanged between the imaging device 1 and the position determination device 2 via the recording medium C. However, this is an example, and the present invention is not limited to this. For example, the position determination system may be configured such that the imaging device 1 and the position determination device 2 are connected by a communication line, a predetermined interface, or the like.
Moreover, in the said embodiment, although the face image detection part 204a is provided in the rotation angle calculation part 204, it is an example and is not restricted to this. For example, the face image detection unit 204a may be provided independently of the rotation angle calculation unit 204. Similarly, each unit included in the marker position determination unit 206 may be provided independently of the marker position determination unit 206.

In the above-described embodiment, the rotation angle calculation unit 204 calculates the rotation angle of the face image detected by the face image detection unit 204a. However, the rotation angle calculation unit 204 is not limited to this example. The rotation angle may be calculated based on other parts of the image. For example, the rotation angle may be calculated based on any one or more of a hand image, a foot image, an image of a part of the limb, a torso image, and other images of all parts of the subject P.
In the above embodiment, although the vertical axis to the ground serving as the center in a predetermined direction axial of the rotational operation, the was not limited to this example, Ru can be arbitrarily changed der appropriately .

  In the above embodiment, the position estimation unit 206c determines the position of the non-detected marker based on the position of the marker in the still image data captured at a timing before the still image data including the non-detected marker. Although the position is estimated, it is an example and the present invention is not limited to this. For example, the position estimation unit 206c may estimate the position of the non-detected marker based on the position of the marker in still image data captured at a later timing than the still image data with the non-detected marker. Alternatively, the position of the undetected marker may be estimated based on the marker position at both the preceding and following timings.

  Further, the position estimation unit 206c, when there are markers that are not detected by the marker detection unit 203 in two or more images among a plurality of images captured at different timings within a predetermined time, the two or more images Among them, based on the position of the non-detected marker estimated for any one of the images, the position of the non-detected marker of another image that has not yet been estimated for the position may be estimated.

In addition, the marker position determination unit 206 may perform processing related to the rotation operation of the subject P indicated by the rotation information generated by the rotation information generation unit 205 in connection with generation and output of the marker position information. Good.
Specifically, the marker position determination unit 206 detects, for example, all the markers by the marker detection unit 203, and the marker position information of still image data in which the rotation information is not used in determining the positions of all the markers. On the other hand, based on the rotation angle of the subject P indicated by the rotation information corresponding to the still image data, calculation of the position of each marker in the coordinate region of the three-dimensional space, adjustment and correction of the position of each marker, etc. Any one or more of the various processes may be performed.

  Further, the determination by the second determination unit 206b may be omitted, and the position of the non-detected marker may be estimated based on the rotation angle indicated by the rotation information. In this case, for example, in the process of estimating the position of the non-detected marker, the position estimation unit 206c acquires the value of the rotation angle indicated by the rotation information as one of the variables, but the rotation angle is 0 [ In the case of [degree], the correction of the estimated position of the non-detected marker based on the rotation angle is eliminated (set to 0), thereby estimating the position of the marker when the subject P is not rotating.

In the above embodiment, the image acquisition unit 202, the marker detection unit 203, the rotation angle calculation unit 204, the rotation information generation unit 205, and the marker position determination unit are controlled by the CPU of the central control unit 210. However, the present invention is not limited to this, and may be realized by executing a predetermined program or the like by the CPU.
That is, a program including an image acquisition routine, a marker detection routine, a rotation angle calculation routine, a rotation information generation routine, a marker position determination routine, an operation information generation routine, etc. is recorded in a program memory (not shown) for recording the program. Keep it. Then, the CPU may acquire an image in which a subject with markers attached to a plurality of movable parts is captured from a predetermined direction by an image acquisition routine. Further, the CPU may detect a marker in the acquired image by a marker detection routine. Further, the rotation angle calculation routine may cause the CPU to calculate a rotation angle about an axis in a predetermined direction of at least a part of the subject image included in the image. Further, the rotation information generation routine may cause the CPU to generate rotation information related to the rotation operation about the axis in the predetermined direction of the subject based on the calculated rotation angle. Further, the CPU may determine the positions of all markers attached to the subject based on the marker detection result and the rotation information by the marker position determination routine. Further, the CPU may generate motion information indicating the motion of the subject based on the positions of all the markers related to the determined plurality of images by the motion information generation routine.

  Furthermore, as a computer-readable medium storing a program for executing each of the above processes, a non-volatile memory such as a flash memory or a portable recording medium such as a CD-ROM is applied in addition to a ROM or a hard disk. Is also possible. A carrier wave is also used as a medium for providing program data via a predetermined communication line.

Although several embodiments of the present invention have been described, the scope of the present invention is not limited to the above-described embodiments, but includes the scope of the invention described in the claims and equivalents thereof.
The invention described in the scope of claims attached to the application of this application will be added below. The item numbers of the claims described in the appendix are as set forth in the claims attached to the application of this application.
[Appendix]
<Claim 1>
In a position determination device that determines the position of a marker in an image in which a subject with markers attached to a plurality of movable parts is captured from a predetermined direction,
Obtaining means for obtaining the image;
First detection means for detecting a marker in the image acquired by the acquisition means;
Calculating means for calculating a rotation angle about an axis in a predetermined direction of at least a part of the subject image included in the image;
First generation means for generating rotation information related to a rotation operation about the axis in the predetermined direction of the subject based on the rotation angle calculated by the calculation means;
Determining means for determining the positions of all markers attached to the subject based on the marker detection result by the first detecting means and the rotation information generated by the first generating means;
A position determining apparatus comprising:
<Claim 2>
The determining means includes
First determination means for determining whether or not all the markers have been detected by the first detection means;
An estimation means for estimating the position of a non-detected marker by the first detection means based on the rotation information when it is determined by the first determination means that all the markers have not been detected; The position determination apparatus according to claim 1, further comprising:
<Claim 3>
The determination means further includes second determination means for determining whether or not the subject is rotating based on the rotation information,
The estimation means estimates the position of a non-detected marker by the first detection means based on the rotation information when the second determination means determines that the subject is rotating. The position determining apparatus according to claim 2, wherein
<Claim 4>
The acquisition means acquires a plurality of images captured at different timings within a predetermined time,
The estimation means includes another image in which a marker corresponding to a non-detected marker in any one of the plurality of images by the first detection means is different from the one image in the plurality of images. 4. The position determination device according to claim 2, wherein the position of the non-detected marker is estimated based on the position of the marker detected from the other image.
<Claim 5>
The determining means determines the positions of all the markers attached to the subject for a plurality of images captured at different timings within a predetermined time,
5. The apparatus according to claim 1, further comprising second generation means for generating motion information indicating the motion of the subject based on the positions of all the markers related to the plurality of images determined by the determination means. The position determination apparatus as described in any one of.
<Claim 6>
A second detecting means for detecting a face image of the subject image;
The said calculation means calculates the rotation angle centering on the axis | shaft of the said predetermined direction of the face image detected by the said 2nd detection means, The Claim 1 characterized by the above-mentioned. Positioning device.
<Claim 7>
An image pickup means for picking up an image of the subject from the predetermined direction;
The position determination apparatus according to claim 1, wherein the acquisition unit acquires an image captured by the imaging unit.
<Claim 8>
An imaging device for imaging a subject having markers attached to a plurality of movable parts from a predetermined direction;
A position determination system comprising the position determination device according to any one of claims 1 to 6,
The position determination system, wherein the acquisition unit of the position determination apparatus acquires an image captured by the imaging apparatus.
<Claim 9>
A position determination method using a position determination device that determines a position of a marker in an image in which a subject with markers attached to a plurality of movable parts is captured from a predetermined direction,
Obtaining the image;
Detecting a marker in the acquired image;
Calculating a rotation angle about an axis in a predetermined direction of at least a part of a subject image included in the image;
Generating rotation information related to a rotation operation about the axis of the predetermined direction of the subject based on the calculated rotation angle;
Determining the positions of all markers attached to the subject based on the marker detection result and the rotation information;
A position determining method characterized by comprising:
<Claim 10>
A computer of a position determination device that determines the position of a marker in an image in which a subject with markers attached to a plurality of movable parts is imaged from a predetermined direction.
Means for acquiring the image;
Means for detecting a marker in the acquired image;
Means for calculating a rotation angle about an axis in a predetermined direction of at least a part of a subject image included in the image;
Means for generating rotation information relating to a rotation operation about the axis of the predetermined direction of the subject based on the calculated rotation angle;
Means for determining the positions of all the markers attached to the subject based on a marker detection result and the rotation information;
A program characterized by functioning as

DESCRIPTION OF SYMBOLS 100 Position determination system 1 Imaging device 2 Position determination apparatus 201 Recording medium control part 202 Image acquisition part 203 Marker detection part 204 Rotation angle calculation part 204a Face image detection part 205 Rotation information generation part 206 Marker position determination part 206a 1st determination Unit 206b second determination unit 206c position estimation unit 207 motion information generation unit C recording medium M1 to M16 marker O axis P subject P1 to P4 subject image

Claims (9)

In a position determination device that determines the position of a marker in an image in which a human with a plurality of markers is imaged from a predetermined direction,
Obtaining means for obtaining the image;
First detection means for detecting a marker in the image acquired by the acquisition means;
Second detection means for detecting a face image of the human image included in the image;
Calculating means for calculating a rotation angle about an axis in a predetermined direction of the face image detected by the second detecting means ;
Based on the rotation angle calculated by the calculation means, first generation means for generating rotation information related to the rotation operation about the axis of the human being in the predetermined direction ;
Estimating means for estimating a position of a non-detected marker by the first detecting means based on the rotation information;
Comprising determining means for determining the position of all the markers attached to the front Symbol person, and
The determining means includes
A position determination apparatus , wherein positions of all markers attached to the human are determined using a marker detection result by the first detection unit and a marker position estimated by the estimation unit. The position determination apparatus according to claim 1, wherein a plurality of the markers are attached to the plurality of movable parts of the human . Based on the prior Kikaido information comprises determining whether turning determination means the person is rotated,
The estimation means estimates the position of the non-detected marker by the first detection means based on the rotation information when the rotation determination means determines that the human is rotating. The position determining device according to claim 2 . The acquisition means acquires a plurality of images captured at different timings within a predetermined time,
The estimation means includes another image in which a marker corresponding to a non-detected marker in any one of the plurality of images by the first detection means is different from the one image in the plurality of images. when it is detected from the position determination apparatus according to claim 2 or 3, characterized in that estimating the position of a marker of the non-detected based on the detected position of the marker from the other image. The determining means determines the positions of a plurality of markers attached to the human for a plurality of images captured at different timings within a predetermined time,
Based on the positions of a plurality of markers according to the plurality of images determined by said determining means, according to claim 1 to 4, further comprising a second generation means for generating operation information indicating the operation of the human The position determination apparatus as described in any one of. Further comprising imaging means for imaging the human from the predetermined one direction;
It said acquisition means, the position determining device according to any one of claim 1 to 5, characterized in that obtaining the image captured by the imaging means. An imaging device for imaging a person with a plurality of markers from a predetermined direction;
A position determination system comprising the position determination device according to any one of claims 1 to 5 ,
The position determination system, wherein the acquisition unit of the position determination apparatus acquires an image captured by the imaging apparatus. A position determination method using a position determination device that determines a position of a marker in an image obtained by imaging a human with a plurality of markers from a predetermined direction,
Obtaining the image;
Detecting a marker in the acquired image;
Detecting a face image of the human image included in the image;
Calculating a rotation angle about an axis in a predetermined direction of the detected face image ;
Generating rotation information related to a rotation operation around the axis of the predetermined direction of the human based on the calculated rotation angle ;
Estimating a position of a non-detected marker based on the rotation information;
Comprising determining the position of all markers attached to the front Symbol person, and
Determining the position of all the markers,
Using the marker detection result by the step of detecting the marker in the acquired image and the marker position estimated by the step of estimating the position of the non-detected marker, all the markers attached to the human being position determination method characterized by determining the position. A computer of a position determination device that determines the position of a marker in an image in which a human with a plurality of markers is imaged from a predetermined direction.
Means for acquiring the image;
First detection means for detecting a marker in the acquired image;
Means for detecting a face image of the human image included in the image;
Means for calculating a rotation angle about the axis in a predetermined direction of the detected face image ;
Based on the calculated rotation angle, means to generate a rotation information relating to the rotational motion about a predetermined axis of said human,
Estimating means for estimating a position of a non-detected marker by the first detecting means based on the rotation information;
Functioning as a determining means for determining the positions of all markers attached to the human ,
The determining means includes
A program for determining the positions of all markers attached to the human using the marker detection results by the first detection means and the marker positions estimated by the estimation means .

Images