JPH11283040A - Operation controller and computer readable recording medium for recording operation analysis program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To automatically estimate a position and movement relating to respective parts of a photographing object in real time independent of a background, and to operate a control object based on the estimated movement. SOLUTION: A person as a photographing object 5 is photographed from plural photographing angles by plural image sensors 7 and 8. In a computer 8 for operation analysis, the difference in two successively photographed images is calculated for the respective plural image sensors, difference images are obtained for the respective image sensors, the area of the photographing object is estimated, the movement inside the respective areas is calculated further and the three-dimensional movement of the respective parts is estimated by using the difference images simultaneously obtained for the respective plural image sensors. In a computer 11 for operation reproduction, a human body model 12 is operated based on the estimated movement for the respective parts.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an operation control device for photographing the motion of a specific object such as a human, estimating the motion of the object for each part from an obtained image, and operating the control object based on the motion. , And a recording medium on which an operation analysis program is recorded.

[0002]

2. Description of the Related Art Image recognition technology for capturing the movement of a person with an image sensor such as a CCD and extracting the movement of each body part such as the head and arms from the captured image has a wide field of application. In particular, the technology to reflect the movement of the human being who is the shooting target in the movement of the human model represented by computer graphics is in the entertainment field such as video games by gesture input, and a new human being that replaces the existing keyboard and mouse. Enables construction of interfaces.

[0003] As a typical prior art related to this,
There is an example of an optical motion capture published in the publication "Imidas'97" (pp. 846-847). According to this technology, the movement of each body part can be captured by attaching a light reflection marker or the like to the surface of the human body to be photographed and following the marker position at each time, but the attached marker is required. There is a problem that the movement of each body part cannot be captured in real time. In order to overcome this, a technique for capturing the movement of the photographing target only by image processing in real time without using an attached marker or the like is required.

[0004] As another conventional technique which does not use an attached marker or the like, see IEICE Transactions Vol. J79-DI
I, No. 1, pp. Yamamoto et al., “Three-Dimensional Moving Image Tracking of Human Motion Based on Robot Model,” which is published in 71-83. According to this technology, the motion of a human being to be photographed is recorded on a video tape in advance, and each body part of the photographing target included in the initial frame is set.
The human is manually matched with the three-dimensional robot model, and the matching between the robot model and the human in the subsequent frame is automatically performed by a motion parameter estimation algorithm. The problem with this method is that the previously captured image is the target of analysis and that the matching with the model is performed manually in the initial frame, and the movement of the target cannot necessarily be reproduced in real time.
The scope of application is limited.

On the other hand, as an example of a conventional technique for extracting a three-dimensional movement in real time while limiting the body part to only the head, a magazine "IEEE Trans. PA"
MI15 (6), 1993 "(pp. 602-605,
"Visually" by Azarbayejani et al.
Controlled Graphics "). These include the outer corners of the eyes, the iris,
Extract two-dimensional displacements on the image by extracting points where the luminance value change is more characteristic than the surroundings, such as the nostrils, and using these points as feature points to calculate the two-dimensional displacement on the image.Based on this, rotate and translate with the Kalman filter This is a method for estimating the three-dimensional movement of the head.

[0006]

The problem with such a conventional method is that the direction in which the face is turned is restricted because the feature points to be dealt with must always be detected. For example, when the nostrils are hidden when facing down, or when the position of the iris cannot be specified when facing down, it is difficult to deal with feature points. Also, if the image includes not only the head but also other body parts such as the torso and both arms, it is necessary to cut out the head region from the image before detecting the rotation of the head, but the method to realize this is included. Not.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems, and automatically and in real time estimates the position and movement of each part to be photographed from an image captured every hour, regardless of the background. It is an object of the present invention to provide a method and a computer-readable recording medium storing an operation control device and an operation analysis program for operating a control target based on an estimated motion.

[0008]

Means for Solving the Problems The above-mentioned problems are solved,
In order to achieve the object, in the operation control device according to the present invention, an image input unit that obtains a plurality of two-dimensional images by capturing an image including a movement of a capturing target from a plurality of capturing angles using a plurality of image sensors. Calculating the difference between two temporally consecutive images photographed for each of the plurality of image sensors, analyzing the plurality of difference images obtained for each image sensor, and And a region estimation unit that estimates an image region for each difference image, and calculates a motion in each region estimated by the region estimation unit, using a difference image obtained simultaneously for each of a plurality of image sensors. A part motion estimating unit for estimating a three-dimensional movement of each part, and a target operation unit for operating a control target based on the motion of each part estimated by the part motion estimating unit.

According to the present invention, when a human body is moved, for example, when the head is shaken and the head is moved, the part region estimating unit analyzes the input images from the plurality of image sensors, thereby obtaining the head. The position of the part is specified, and the part motion estimating unit extracts three-dimensional motion information such as rotation. Also,
If necessary, the analysis contents of the region estimation unit and the region motion estimation unit are displayed on the monitor. Then, based on the three-dimensional motion information extracted by the part motion estimating unit, the target operation unit moves, for example, a person model in computer graphics expression in real time.

In the operation control device according to the next invention, the part region estimating unit binarizes each pixel value of the difference image to obtain a silhouette of the object to be photographed, analyzes the width of the obtained silhouette, and analyzes the width of the obtained silhouette. Means for estimating an image area including the part.

According to the present invention, the obtained difference image is binarized by the threshold value to obtain, for example, a human silhouette (black pixel, white pixel), and the width of the silhouette composed of black pixels That is, by calculating the width of each of the horizontal pixel columns, the black pixel positions appearing first when viewed from the left end and the right end as the two ends of the silhouette, and calculating the width of each of these columns, the image including the portion to be captured Estimate the area.

[0012] In the operation control device according to the next invention, the object operation section includes means for reproducing the operation of the photographing object by the movement of the model represented by computer graphics.

According to the present invention, the rotation angle of the head model is represented by the body part moved by a human, for example, the output regarding the presence / absence of the rotation detection of the head rotation and the frequency thereof. Move the head of the model etc. in real time.

According to another aspect of the present invention, there is provided a recording medium storing an operation analysis program according to the present invention, wherein a computer is provided with two temporally continuous two-dimensional images of a plurality of two-dimensional images taken from a plurality of photographing angles by a plurality of image sensors. Calculating a difference between the images, analyzing a plurality of difference images obtained for each image sensor, and estimating an image region including a part to be imaged for each difference image, a procedure for estimating each of the estimated regions. This is a program for recording a program for executing a procedure of calculating a motion, associating each difference image with each other, and estimating a three-dimensional motion of each part.

According to the present invention, when a human body is moved, for example, when the head is moved by shaking the head, a computer capable of reading this program analyzes the input images from the plurality of image sensors. By doing
The position of the head is specified, and three-dimensional motion information such as rotation is extracted. Based on the extracted three-dimensional motion information, a human model or the like in computer graphics is moved in real time.

In the operation control device according to the next invention, an image input section for capturing a plurality of two-dimensional images by capturing an image including a movement of a capturing target from a plurality of capturing angles using a plurality of image sensors; Calculate the difference between two temporally consecutive images taken for each image sensor, analyze the plurality of difference images obtained for each image sensor, and calculate the image area including the part to be imaged. A part region estimating unit for estimating each difference image, and each pixel value constituting two temporally continuous images is projected in the horizontal and vertical directions, and a primary image is obtained for each image sensor based on the obtained projection data. A part motion estimator for calculating the original optical flow, calculating the motion in each region estimated by the part region estimator from the obtained one-dimensional optical flows, and estimating a three-dimensional motion of each part, Action And the target operation section for operating the controlled object based on the movement of each region estimated in section, in which with a.

According to the present invention, when a human body part is moved, for example, when the head is shaken and the head is moved, the part region estimating part analyzes the input images from the plurality of image sensors, thereby obtaining the head part. Identify the position of the part, and further, the part motion estimating unit projects each pixel value constituting two temporally continuous images in the horizontal and vertical directions, and for each image sensor based on the projection data obtained, The original optical flow is calculated, and three-dimensional motion information such as rotation is extracted from the obtained one-dimensional optical flows. If necessary, the analysis contents of the region estimation unit and the region motion estimation unit are displayed on the monitor. Then, based on the three-dimensional motion information extracted by the part motion estimation unit, the target operation unit
For example, a person model in computer graphics expression is moved in real time.

In the operation control device according to the next invention, the projection data is directly obtained by an image sensor equipped with a semiconductor integrated circuit having an image processing function (corresponding to an artificial retinal LSI in an embodiment described later). It is.

According to the present invention, for example, an artificial retinal LSI operating as a semiconductor integrated circuit converts horizontal and vertical projection data relating to the two-dimensional image from data of the two-dimensional image captured by the image sensor. Output automatically.

In the operation control device according to the next invention, the part region estimating unit binarizes each pixel value of the difference image to obtain a silhouette of a photographing target, analyzes the width of the obtained silhouette, and obtains the silhouette. The apparatus includes means for estimating an image area including a target part.

According to the present invention, the obtained difference image is binarized by the threshold value to obtain, for example, a human silhouette (black pixel, white pixel), and the width of the silhouette composed of black pixels That is, by calculating the width of each of the horizontal pixel columns, the black pixel positions appearing first when viewed from the left end and the right end as the two ends of the silhouette, and calculating the width of each of these columns, the image including the portion to be captured Estimate the area.

In the operation control device according to the next invention, the object operation section has means for reproducing the operation of the photographing object by the movement of the model represented by computer graphics.

According to the present invention, the rotation angle of the head model is expressed by the body part moved by a human, for example, the output related to the presence or absence of detection of the rotation of the head rotation and the frequency of the output. Move the head of the model etc. in real time.

[0024] In the recording medium storing the operation analysis program according to the next invention, the computer is provided with two temporally continuous two-dimensional images of a plurality of two-dimensional images photographed from a plurality of photographing angles by a plurality of image sensors. Calculating a difference between the images, and analyzing a plurality of difference images obtained for each image sensor, and estimating an image region including a part to be imaged for each difference image. Each pixel value constituting one image is projected in the horizontal and vertical directions, a one-dimensional optical flow is calculated for each image sensor based on the obtained projection data, and estimated from the obtained plurality of one-dimensional optical flows. A program for calculating a motion in each region and estimating a three-dimensional motion of each part is recorded.

According to the present invention, when a human body moves, for example, when the head is shaken and the head is moved, a computer capable of reading this program analyzes input images from a plurality of image sensors. By specifying the position of the head by that, further, each pixel value constituting two images that are temporally continuous is projected in the horizontal and vertical directions,
A one-dimensional optical flow is calculated for each image sensor based on the obtained projection data, three-dimensional motion information such as rotation is extracted from the obtained one-dimensional optical flows, and the extracted three-dimensional motion information is obtained. On the basis of,
Move a human model in computer graphics expression in real time.

In a recording medium on which an operation analysis program according to the next invention is recorded, a computer is provided with a plurality of semiconductor integrated circuits having image processing functions (corresponding to an artificial retinal LSI in an embodiment described later). Calculating the difference between two temporally consecutive images of a plurality of two-dimensional images photographed from a plurality of photographing angles by the image sensor, and analyzing the plurality of difference images obtained for each image sensor. ,
Procedure for estimating an image area including a part to be imaged for each difference image, calculating a one-dimensional optical flow for each image sensor based on projection data directly obtained by the image sensor, and obtaining a plurality of one-dimensional optical flows. , A procedure for calculating the estimated motion in each region from the above, and estimating the three-dimensional motion of each part.

According to the present invention, when a human body is moved, for example, when the head is shaken and the head is moved, a computer capable of reading this program analyzes the input images from the plurality of image sensors. Identify the position of the head, calculate the one-dimensional optical flow for each image sensor based on the directly obtained projection data, and extract three-dimensional motion information such as rotation from the obtained one-dimensional optical flows Then, based on the extracted three-dimensional motion information, a human model or the like in computer graphics is moved in real time.

[0028]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of a computer-readable recording medium on which an operation control device and an operation analysis program according to the present invention are recorded will be described in detail with reference to the drawings.

FIG. 1 is a block diagram showing a configuration of an operation control apparatus according to an embodiment of the present invention. The image input unit 1, the part region estimating unit 2, the part motion estimating unit 3, the motion reproducing unit (object operation unit) ) 4).

The image input unit 1 includes a plurality of image sensors, and captures the movement of an object from different angles for each image sensor. In the present embodiment, the image input unit 1 uses two image sensors.
An example of a case where a table is provided will be described. The part region estimation unit 2 calculates a difference (absolute value) between corresponding pixels between two temporally consecutive images for each image sequentially output from each image sensor of the image input unit 1 ( Hereinafter, the obtained image is referred to as a difference image), and the image area of each body part is estimated based on the difference image. The part motion estimator 3 analyzes the motion in each area estimated by the part area estimator 2, and estimates three-dimensional motion such as rotation in the depth direction. The motion reproducing unit 4 simulates a person model simulated by computer graphics or the like based on the motion of the part estimated by the part motion estimating unit 3.

FIG. 2 shows an example in which a human model is simulated by the operation control device according to the present embodiment, and shows the positional relationship between a person to be photographed 5 and the image sensors 6 and 7 and a region area. A motion analysis computer 8 responsible for the processing contents of the estimation unit 2 and the part motion estimation unit 3, a motion analysis result display monitor 9 for displaying the analysis contents, and an output data of the motion analysis computer 8 Network 10, a motion reproduction computer 11 for reproducing the motion of a human being to be photographed using transmission data as input, and a motion reproduction monitor (computer graphics display monitor) for expressing the reproduced motion as a human model 12. ) 1
And 3.

In the configuration of FIG. 2, when a body part moved by a human, such as a head, is shaken and moved, the computer 8 analyzes the input images from the image sensors 6 and 7, thereby The position of the head can be specified, and three-dimensional motion information such as rotation can be extracted. If necessary, the analysis contents of the computer 8 can be displayed on the monitor 9. In FIG. 2, by transmitting the three-dimensional motion information extracted by the computer 8 via the network 10, the computer 11 coupled to the network transmits the computer graphics expression on the attached monitor 13. The human model 12 can be moved in real time.

Here, a human body part estimating algorithm processed by the part area estimating unit 2 will be described.
An example of processing based on the head region estimation algorithm will be described with reference to FIG. 3, and details of the head region estimation algorithm will be described with reference to the flowchart in FIG. Further, an example of the processing content based on the torso and both arms region estimation algorithm will be described with reference to FIG. 5, and details of the torso and both arms region estimation algorithm will be described with reference to the flowchart in FIG.

FIGS. 3A to 3C show the processing contents of the part region estimating section 2 based on the head region estimating algorithm.
This is an example in which a human head region is estimated by analyzing a difference image obtained from each output image of the image sensors 6 and 7, and shows a process relating to one of the two image sensors. FIG. 3A shows a difference image obtained from an input image including human motion, and FIG.
The result of binarizing each pixel value of the difference image by a threshold value and the analysis contents of a head region estimation algorithm are shown. FIG. 3C shows the head position obtained based on the above algorithm on the input image, and the white line frame A indicates the position of the estimated head rectangular area in the input image.

Here, in order to explain the processing contents of FIG. 3B, steps ST1 to ST of the flowchart of FIG.
6 will be described. First, in step ST1, the difference image of FIG.
The human silhouette 14a obtained by the movement shown in FIG. Next, in step ST2, the width of the silhouette 14a composed of black pixels is calculated. In particular,
This width is calculated for each horizontal pixel column, with the black pixel position first appearing from the left end and the right end as viewed from both ends of the silhouette 14a. This silhouette width reflects the approximate width of the human outline from the head to the torso, but if the required silhouette width does not reflect the actual width depending on the location, such as the width around the eyes of the silhouette 14a. There is.

Next, by smoothing the silhouette width distribution in step ST3, it is possible to obtain a smoothed silhouette width distribution such as 14b reflecting the actual human outline width to some extent. The upper end of the smoothed silhouette width distribution obtained in this way is registered as the estimated top position. Also, in particular, when a human moves left and right with his arms down, a depression can be formed around the neck position as in the smoothed silhouette width distribution 14b. If this position is specified, the vertical position of the human neck can be calculated. Can be estimated.

Next, in step ST4, this hollow search is performed. That is, the width value is checked from the upper end (estimated crown position 16) of the silhouette width distribution 14b, the next smallest position is found from the first largest position, and registered as the estimated neck position 15. If the silhouette width distribution is smooth as in 14b, it can be seen that this minimum position reflects the position of the neck.

Next, in step ST5, in order to estimate both end positions of the head region, the binarized difference image between the estimated head position 16 and the estimated neck position 15 is projected in the vertical direction, and the projected image is projected. Both ends are registered as both ends (left end 17, right end 18) of the estimated head region. From the above, the position of the head region is estimated (step ST6).

FIGS. 5 (a) to 5 (c) show the processing contents of the part region estimating unit 2 similar to FIG. 3 for estimating the torso and both arms regions. FIG. 5A shows a difference image obtained from an input image including human motion, and FIG.
Shows the result of binarizing each pixel value of the difference image by the threshold value and the analysis contents of the torso and both arms region estimation algorithm. FIG. 5C shows the estimated head rectangular area (white line frame A), torso rectangular area (white line frame B), and both arms rectangular area (white line frames C and D) in the input image. Indicates the position.

Here, in order to explain the processing contents of FIG. 5B, steps ST11 to ST11 of the flowchart of FIG.
The correspondence with T16 will be described. First, step ST1
1, the difference image of FIG. 5A is binarized by a threshold value, and the human silhouette 14 due to the motion shown in FIG.
a is obtained.

Next, in step ST12, the above-mentioned rectangular head region is estimated. Next, in step ST13, both end positions of the shoulder are estimated using the estimated head rectangular area as a clue. Specifically, the position of the shoulder is assumed to be several pixels below the lower end of the rectangular region in the head (two pixels below in the example of FIG. 5).

Next, in step ST14, a torso rectangular region is extracted on the basis of the positions of the left end 20 and the right end 21 of the shoulder estimated in step ST3, assuming that the width of the torso is equal to the shoulder width. Also, the upper end of the estimated torso rectangular region is equal to the lower end 15 of the estimated head rectangular region,
In the example of FIG. 5, only the upper body is assumed to be the range of the image with respect to the lower end, so the lower end is set to the lowermost end of the image. If both legs are within the range of the image, it can be estimated that the horizontal width of the torso rectangular region is the shoulder width and the vertical width is a fixed multiple of the shoulder width.

Next, in step ST15, using the torso rectangular region estimated in step ST14 as a clue,
Assuming that the black pixels present on both sides are both arms, a rectangular shape surrounding them is estimated as a two-arm rectangular area. In FIG. 5B, the left arm is estimated as a rectangular area having a horizontal width between 19 and 20 and a vertical width between 15 and the lowermost end of the image, and the right arm is a rectangular area having a horizontal width between 21 and 22. Estimated as type region.

Finally, in step ST16, an approximate square approximating the distribution of black pixels existing in the rectangular region of both arms estimated in step ST15 is calculated, and the inclination of the moving arms is estimated. In addition, as a specific calculation method of the approximate quadrangle, the international conference proceedings “IEEE2nd International”
Tional Conference on Auto
magic Face and Gesture Re
cognition, October 1996, "C
output Vision for Computing
er Games "". In this example, when a hand included in an image has various inclinations, the area of the hand is approximated by a rectangle, and the inclination is determined at the same time. If this algorithm is applied to the case where both arms are calculated as approximate squares, an approximate square reflecting the inclination of both arms can be calculated as shown in FIG. 5C.

Next, an algorithm for estimating the motion of the body part processed by the part motion estimating unit 3 will be described with reference to FIGS. Hereinafter, details of the head rotation estimation algorithm for estimating the three-dimensional rotation of the head with the body part whose motion is estimated as the head will be described.

FIG. 7 is a flowchart showing a head rotation estimation algorithm. FIGS. 8 and 9 are explanatory diagrams showing human head rotation about vertical and horizontal rotation axes, respectively. In particular, the correspondence with steps ST21 to ST26 in the flowchart of FIG. 7 is described. FIGS. 8 and 9A show the rotation of the head of the person 5 about the vertical rotation axis 28X and the horizontal rotation axis 28Y. 8 and 9 (b).
(C) is an input image from the two image sensors, and white line frames A ₁ and A ₂ in the figure indicate the head rectangular region estimated by the above-described region estimation unit 2 (step ST21 and step ST21). ST22).

Next, in step ST23, an optical flow is calculated within the estimated head rectangular area. A specific calculation method of the optical flow is described in, for example,
No. 1077, there is a method using a neural network.
Velocity vector (motion vector) of each pixel from spatial change
Is calculated, and the direction and intensity of the motion for each pixel are calculated. Specifically, the optical flow can be obtained by minimizing the functional ε, that is, ε = ε ₁ + ε ₂ . Note that ε ₁ is a condition that should satisfy the optical flow to be obtained, and ε ₂ is a constraint condition for the velocity vector to change smoothly and spatially.

Here, a method of calculating an optical flow using the neural network will be briefly described. The specific configuration for calculating the optical flow is as shown in FIGS. 15A and 15B. For example, the x-direction velocity neurons 102a and y for obtaining the x component and the y component at each point of the moving object, respectively. A set of directional velocity neurons 102b (see (b)), a plurality of arranged bidirectional neural networks 102 ((a)
), And a line process 102c (see (b)) for detecting the edge of the moving object (taking an analog value) by arranging it between the velocity neurons.

Next, the operation will be described with reference to the configuration of FIG. 15 and the flowchart of FIG. First,
When the input image 101 is input to the bidirectional neural network 102 (step ST51), the x-direction velocity neuron 2a substitutes the x component of the optical flow, and similarly, the y-direction velocity neuron 2b substitutes the y component of the optical flow, Calculate each. That is, each velocity neuron calculates a velocity vector (step ST5).
2). Then, the line process 102c calculates the horizontal and vertical line processes (step ST5).
3) By the output of the bidirectional neural network 102 when the functional ε of the equation converges (step ST5)
4), a velocity vector (optical flow) of the moving object to be obtained is obtained (step ST55).

The calculation of the optical flow is performed by, for example, the apparatus shown in FIG. 17 (the image sensor 6 shown in FIG. 2).
7, including the operation analysis computer 8). Note that a method other than the above document may be used as a method for calculating the optical flow. In FIG. 17, reference numeral 104 denotes a CCD camera for capturing input image data, 105 denotes a camera interface, 106 denotes a processor (CPU) for performing the above-described processing, and 107 denotes input image data captured by the CCD camera 104. An image memory 108 for storing the image data is an interface for outputting the result obtained by the CPU 106 to the outside.

Therefore, 23a ₁ in FIGS. 8 (d) and 9 (d) is obtained by the method described above with reference to FIGS. 8 (b) and 9 (b).
Estimation and shows the head rectangular type region A optical flows calculated in the _1, 23a ₂ in FIG. 8 (e) and FIG. 9 (e) by the method described above, FIG. 8 (c) and FIG. 9
Shows the calculated optical flow in the estimated head rectangular type region A ₂ of (c).

Next, in step ST24 in FIG. 7, the optical flow obtained in step ST23 is averaged and projected in the horizontal and vertical directions. FIG. 8 (d)
And 24a ₁ and 25a ₁ in FIG. 9 (d) shows the result obtained by projecting the optical flow 23a ₁ by averaging it it horizontally and vertically, 8 (e) and FIG. 9
(E) 24a ₂ and 25a ₂ are the results of averaging and projecting the optical flow 23a ₂ in the horizontal and vertical directions, respectively.

Next, in step ST25, the horizontal projection 24 of the optical flow obtained in step ST24 is performed.
a ₁ , 24a ₂ and the vertical projections 25a ₁ , 25a ₂ are compared between the respective image sensors to estimate the three-dimensional movement of the head region including the depth direction. 8 (f) (g) and FIG.
(F) Reference numerals 26 and 27 in (g) denote cross-sections of the human head viewed from above and to the right. Each arrow row indicates the head direction including the depth direction estimated in step ST25 viewed from each direction. Indicates movement. 8 (f) and FIG.
Each row of arrows in (f) is an averaged vertical projection 25a.
₁ and 25a are arrows columns each motion vector obtained by combining in each column in _2, the arrows row in FIG. 8 (g) and FIG. 9 (g) were averaged horizontal projection 24a ₁ and 24a ₂
5 is an arrow sequence obtained by synthesizing each motion vector in each column.

Finally, at step ST26,
Depth of vertical and horizontal sections obtained by ST25
Input the motion vector pattern including the direction
Determine if rotation has been detected for one of these axes
You. As a specific rotation judgment method, each vertical and horizontal
For each motion vector including the depth direction obtained on the cross section,
And appears most strongly in the forward (facial) and backward direction of the head
Vector pair P₁, Q ₁, And P_Two, Q_TwoAnd select
Their magnitude is greater than or equal to a preset threshold, and
If there are restrictions on the angle between them,
Rotation with respect to the horizontal rotation axis can be determined.

FIG. 10 shows the result of the estimation of the head rotation with the horizontal direction as the rotation axis, estimated by the part motion estimating unit 3, where the clockwise direction is the positive direction and the counterclockwise direction when viewed from the right side of the head. Is a time series of rotation detection in the case where is taken in the negative direction. Each vertical line indicates the presence or absence of rotation detection for each frame, and the density of these arrangements indicates the magnitude of the rotation angle of the head. For example, if one detection signal indicates a rotation of 10 degrees, FIG.
First 10 degrees clockwise, then slowly 2 counterclockwise.
It can be estimated that the head is rotated 0 degrees, then 50 degrees clockwise, and then 10 degrees counterclockwise. It should be noted that the estimation of the head rotation with the vertical direction as the rotation axis can be similarly estimated.

Finally, how the motion of the human body is reproduced in the motion reproducing section 4 will be described. When moving a human model represented by computer graphics or the like based on the head rotation estimated by the part motion estimating unit 3, it is desirable that the rotation of the head model reflects the actual movement of the head. There are two types of output data expression formats of the part motion estimating unit 3 regarding the rotation angle of the head rotation. One is a method in which the rotation angle of the head model is represented by the output regarding the presence or absence of rotation detection and its frequency as shown in FIG. 10, and it is not necessary to estimate the rotation angle. it can.

The other is the output data of the part motion estimating unit 3.
Not only the presence / absence of rotation detection but also the rotation angle
It is a method of expressing, for example, various vectors
Vs. P ₁, Q₁, And P_Two, Q_TwoThe rotation angle in advance
Determined, corresponding to the position and size of the obtained vector pair
The rotation angle is output. In this way, FIG.
Output data from the motion analysis computer 8
The signal about the presence or absence of detection and the numerical data about the rotation angle
Can be sent as one message,
Through this, the operation reproducing computer 11 receiving the
You can receive a message with detection, and
Rotating computer graphics with a rotation angle
it can.

In this expression method, since the estimation of the rotation angle depends on the illumination condition, if the illumination condition changes, the reproduced motion may not reflect the actual motion of the object to be photographed. As a result, the response speed from the human eyes regarding the reproducibility of the motion is improved as compared with the above-described expression method. Regardless of which expression method is adopted for the output data of the part motion estimating unit 3, the human head model represented by computer graphics by the motion reproducing unit 4 can be operated by the motion of the person himself.

In the above-described embodiment, an example has been shown in which the part motion estimating unit 3 estimates the rotation of the human head. However, in the case of parallel movement, the horizontal projection of the optical flow and the By comparing the vertical projection among the image sensors, it is possible to easily estimate the three-dimensional movement of the head including the depth direction, and it is possible to analyze a more complicated movement of the head by combining these.

The above-described operation analysis operation of the photographing target is realized by an operation analysis program, and the program is provided as a recording medium. For example, in the above embodiment, the data is stored in the hard disk of the operation analysis computer 8 in FIG.

As described above, according to the present embodiment, the region estimation is performed by using the difference image regardless of the background, and furthermore, the rotation and the like are determined by comparing the optical flow analysis in the estimation region obtained from the plurality of image sensors. A complex three-dimensional motion can be estimated, and the motion of a person or the like can be reproduced in real time using a computer graphics model.

In this embodiment, an example has been described in which the image input unit 1 has two image sensors 6 and 7 to obtain a stereo image. However, the present invention is not limited to this configuration. A configuration using an arbitrary pair of two sensors may be used.

Further, for example, Japanese Patent Application Laid-Open Nos. 8-292998, 8-275059, 8-27
If an image sensor equipped with a light receiving element as described in JP-A-5069 and JP-A-8-56011 is used, a difference image can be obtained as an output from an image input unit, and the operation speed of motion analysis can be further improved. .

Further, in the present embodiment, an example of reproducing the movement of the head of the person to be photographed has been described, but other body parts such as the torso and both arms may be subjected to the motion reproduction.
The imaging target is not limited to a human, but may be another imaging target such as an animal. The analysis algorithm of the region estimation unit differs depending on the characteristics of the imaging target.

Further, in the present embodiment, an example has been described in which the processing functions of the part motion estimating unit 3 and the motion reproducing unit 4 are realized by separate computers, but both processing functions are realized by a single computer. You may.

Further, the reproduced motion information of the object to be photographed may be transmitted to a game machine, a personal computer, or the like, and the operation of a joystick, a mouse, or the like may be performed as an input interface. By doing this,
A person with an obstacle in both hands or a person who constantly engages in work using both hands can operate any control target by moving the head.

In this embodiment, FIG. 8B and FIG.
Horizontal and vertical within the estimated head rectangular area shown in FIG. 9 (b)
As a method of estimating the direction motion, first, two-dimensional
Optical flow 23a₁・ A_TwoAnd then
The individual motion vectors that make up the physical flow are
And vertical projection (for vectors arranged in columns or rows)
Calculate the average)
Motion vector 24a₁・ A_TwoAnd vertical motion vector
25a₁・ A_TwoHas been shown as an example, but another method
Each pixel value that constitutes the image in the horizontal and vertical directions.
Shadow, and based on these projection data,
Calculating the cull flow, 24a₁・ A _TwoAnd 25a₁・
A motion vector similar to a2 can be obtained.

Here, the one-dimensional optical flow algorithm and specific calculations will be described with reference to FIGS. FIG. 11 is a diagram for calculating a one-dimensional motion vector. For example, each element of the pixel value projection is regarded as one pixel value, and a contrast formed by three adjacent pixel values (hereinafter referred to as a light-dark edge). ) Show four examples that move left and right. It should be noted that the thin line in the drawing indicates the projection at the previous time, and the thick line indicates the projection at the current time.
The brighter pixel value is 50, and the darker pixel value is 10.

For example, FIGS. 11 (a) and (c)
11B shows a change in the projection value when the light-dark edge moves from left to right, and FIGS. 11B and 11D show a change in the projection value when the light-dark edge moves from right to left. .
From these, it can be seen that there are the following three fixed rules regarding the movement direction at a certain projection position between the change in the projection value (pixel value) and the movement of the projection.

(I) If the change in the projection value is 0 at a certain projection position, or if there is a change in the projection value, the current time or the previous time is selected as the reference time, and the adjacent two projection values at the reference time are selected. Have the same projection value, the motion direction is estimated to be 0 (no motion). (II) At a certain projection position, when the change of the projection value is a positive value, the movement direction is estimated to be the direction of the smaller projection value among the two adjacent projection values at the reference time. (III) At a certain projection position, when the change in the projection value is a negative value, the motion direction is estimated to be the direction of the larger projection value among the two adjacent projection values at the reference time.

FIG. 12 is a diagram showing an example of the projection at the previous time and the projection at the current time, and the estimation direction of the one-dimensional motion vector. Next, using the rules (I) to (III), how a one-dimensional motion vector (that is, one-dimensional optical flow) is obtained from a change in the projection value between the previous time and the current time will be described. 12 will be described. FIG.
(A) and (b) are examples of projection values each having eight elements, and show projection values at the previous time and the current time, respectively. FIG. 12C is a table for specifically showing how the above rule is applied to each projection element. Each element takes a value of 10 to 50 for simplicity. Although the reference time will be described in the case of the previous time, a similar result can be obtained even if the current time is selected as the reference time separately from the present embodiment.

First, regarding the element # 1, since the difference between the projection value of the previous time and the current time is 0, the rule (I) is applied, and the estimation direction becomes 0. Next, regarding the element # 2, since the difference is a negative value, the rule (II
Applying I), the estimation direction becomes the direction from # 2 to # 3.
The same applies to elements # 3 and # 4, and the estimation directions are # 3 → # 4 and # 4 → # 5, respectively.

Next, regarding the element # 5, the difference is a positive value, but since the elements # 4 and # 6 have the same value in the projection at the previous time, the rule (I) is applied. The estimated direction is 0. Next, regarding the element # 6, since the difference is a positive value, the rule (II) is applied, and the estimation direction is from # 6 to # 7. Similarly, for the element # 7, the estimation direction is # 7 → # 8. Finally, regarding # 8, the difference is a positive value, but the direction cannot be estimated because there is no adjacent element in the projection at the previous time.

The principle of the one-dimensional motion vector calculation has been described above. Next, the operation of the one-dimensional optical flow algorithm utilizing this principle will be described with reference to the drawings.

FIG. 13 is a flowchart showing the principle of the one-dimensional motion vector calculation. First, a projection value is obtained at the current time, and this is
It is stored in r_proj (step ST31). Next, a difference between the array curr_proj and the array prev_proj in which the projection at the previous time is stored is calculated, and the result is stored in the array diff_proj (step ST32).

Next, the array diff_proj and the array p
Using rev_proj, a one-dimensional motion vector is calculated for each element, and all the results are stored in an array vector
_Proj (step ST33). Next,
After calculating all the one-dimensional motion vectors, the array curr_p
roj is the projection of the previous time, the array prev_proj
(Step ST34). Finally, step S
In order to reduce noise of the one-dimensional motion vector obtained in T33, the array vector_proj is smoothed and output (step ST35).

Here, the calculation of the one-dimensional motion vector for each projected element in step ST33 will be described in detail using the above-mentioned rule. Here, the process of calculating the projection array index k will be described with reference to the flowchart in FIG.

First, prev_proj [k-1] and p
The magnitude of rev_proj [k + 1] is compared (step ST41). If these values are equal (step S
T42, YES), 0 is substituted for vector_proj [k] (step ST43), and the calculation is terminated (corresponding to rule (I)). On the other hand, if they are not equal (step S
T42, NO), of the array indices k and k + 1
The smaller value of prev_proj is L, and the larger value is G
(Step ST44).

Next, the sign of diff_proj [k] is checked, and if it is 0, for example (step ST4
5, YES), substituting 0 into vector_proj [k] (step ST43), and terminating the calculation (corresponding to law (I)). If it is a negative value (step ST45, NO, step ST46, YES) , The direction k → G is selected (step ST48, corresponding to the rule (III)), and if it is a positive value (step ST45, NO, step ST46, N
O), the direction k → L is selected (step ST47, corresponding to the rule (II)).

Then, steps ST48 and ST47
In, if G (or L) is, for example, k−1, −1 is substituted for vector_proj [k], and if G (or L) is k + 1, +1 is added to vector_proj [k].
Is assigned. Finally, the value of the vector_proj [k] is weighted by the absolute value of the diff_proj [k] (step ST49), and the calculation of the one-dimensional motion vector ends.

As described above, FIG.
Alternatively, one-dimensional optical flows corresponding to 24a ₁ · a ₂ and 25a ₁ · a ₂ in FIG. 9 can be calculated. In addition, processing data can be reduced by using one-dimensional optical flow calculation using projection data, as compared with using two-dimensional optical flow calculated from an image. Furthermore, an image processing function (artificial retina L
8 (b) and FIG. 9 by an image sensor having
If it is possible to obtain a projection of direct pixel values from the image data of the estimated head region A ₁ in (b), each pixel value forming the image it is not necessary to project the horizontal and vertical directions, the above calculation process It is further accelerated.

[0082]

As described above, according to the operation control apparatus of the present invention, an image including a motion of a photographing target is photographed from a plurality of photographing angles by a plurality of image sensors to obtain a plurality of two-dimensional images. An input unit and calculates a difference between two temporally consecutive images captured for each of the plurality of image sensors,
Each of the plurality of difference images obtained for each image sensor is analyzed, and a region estimation unit that estimates an image region including the imaging target region for each difference image, and a region region estimation unit that estimates the region. A part motion estimating unit that calculates the motion in each region and estimates a three-dimensional motion of each part using a differential image obtained simultaneously for each of the plurality of image sensors; and for each part estimated by the part motion estimating unit. And a target operation unit that operates the control target based on the movement of the target, so that the position of the part to be shot can be specified regardless of the background, and the control target can be determined based on the movement of the shot target in various situations. Can be operated in real time.

According to the operation control device of the next invention, the region estimation unit binarizes each pixel value of the difference image to obtain a silhouette of the object to be photographed, and analyzes the width of the obtained silhouette. Since there is provided a means for estimating the image area including the part to be imaged, it is possible to easily estimate each part of the object to be imaged.

According to the operation control device of the next invention, the object operation unit includes means for visually reproducing the operation of the object to be photographed by the movement of the model represented by computer graphics. This has the effect that the movement of the photographing target can be reproduced on the model. In addition, when oneself or another person is a shooting target, there is an effect that the movement of the target itself and the movement of the model can be intuitively compared.

According to the recording medium storing the operation analysis program according to the next invention, the difference between two temporally continuous two-dimensional images of a plurality of two-dimensional images photographed from a plurality of photographing angles by a plurality of image sensors. Is calculated, and a plurality of difference images obtained for each image sensor are analyzed, and an image region including a part to be imaged is estimated for each difference image, and a motion of each estimated region is calculated. Then, the procedure for estimating the three-dimensional movement of each part, and the procedure for estimating the three-dimensional movement of each part, were performed by the computer, so that the position of the part to be photographed can be specified regardless of the background, This has the effect that the control target can be operated in real time based on the movement of the shooting target in various situations.

According to the operation control apparatus of the next invention, an image input unit for obtaining a plurality of two-dimensional images by capturing images including a movement of a subject by a plurality of image sensors from a plurality of shooting angles, The difference between two temporally consecutive images photographed for each image sensor is calculated, and a plurality of difference images obtained for each image sensor are analyzed, and an image region including a part to be photographed is determined for each image sensor. A part region estimating unit for estimating each difference image, and each pixel value constituting two temporally continuous images is projected horizontally and vertically, and one-dimensionally for each image sensor based on the obtained projection data. A part motion estimator for calculating an optical flow, calculating a motion in each region estimated by the region estimator from a plurality of obtained one-dimensional optical flows, and estimating a three-dimensional motion of each part, and a part motion Estimation And the target operation section for operating the control target in on the basis of the motion of each estimated site, since with a,
It is possible to specify the position of the part to be photographed irrespective of the background, and it is possible to operate the control object in real time based on the movement of the photographing object in various situations. In addition, processing data can be reduced by using one-dimensional optical flow calculation using projection data as compared with using two-dimensional optical flow calculated from an image.

According to the operation control device of the next invention, the projection data is directly obtained by the image sensor equipped with a semiconductor integrated circuit having an image processing function (corresponding to an artificial retinal LSI). There is an effect that the motion estimation can be further speeded up.

According to the operation control apparatus of the next invention, the region estimation unit binarizes each pixel value of the difference image to obtain a silhouette of the object to be photographed, and analyzes the width of the obtained silhouette. Since there is provided a means for estimating the image area including the part to be imaged, it is possible to easily estimate each part of the object to be imaged.

According to the operation control device of the next invention, the object operation unit has means for reproducing the operation of the object to be photographed by the movement of the model represented by computer graphics. Can be reproduced on the model at a higher speed. In addition, when oneself or another person is a shooting target, there is an effect that the movement of the target itself and the movement of the model can be intuitively compared.

According to the recording medium storing the operation analysis program according to the next invention, the difference between two temporally continuous two-dimensional images of a plurality of two-dimensional images photographed from a plurality of photographing angles by a plurality of image sensors. Calculating a plurality of difference images obtained for each image sensor, and estimating an image region including a part to be imaged for each difference image, and two temporally continuous images. Are projected in the horizontal and vertical directions, a one-dimensional optical flow is calculated for each image sensor based on the obtained projection data, and each region estimated from a plurality of one-dimensional optical flows is obtained. Calculate the movement of, and the procedure of estimating the three-dimensional movement of each part, and let the computer to execute, it is possible to identify the position of the part of the imaging target regardless of the background, You can operate the controlled object in real time based on the motion of the imaging target in variety of situations, an effect that. In addition, processing data can be reduced by using one-dimensional optical flow calculation using projection data as compared with using two-dimensional optical flow calculated from an image.

According to the recording medium on which the operation analysis program according to the next invention is recorded, a plurality of image sensors equipped with a semiconductor integrated circuit having image processing functions (corresponding to an artificial retinal LSI) photograph from a plurality of photographing angles. The difference between two temporally continuous images of the obtained two-dimensional images is calculated, the plurality of differential images obtained for each image sensor are analyzed, and an image region including a part to be imaged is calculated. The procedure for estimating each difference image, and the image sensor calculates a one-dimensional optical flow for each image sensor based on the directly obtained projection data. Calculate the movement of
The procedure for estimating the three-dimensional movement of each part is made to be executed by the computer, so that the position of the part to be imaged can be specified regardless of the background, and based on the movement of the imaging target in various situations Thus, the control target can be operated in real time. In addition, processing data can be reduced by using one-dimensional optical flow calculation using projection data as compared with using two-dimensional optical flow calculated from an image. Further, since the projection data is directly obtained by the image sensor, there is an effect that the motion estimation of each part can be further speeded up.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an operation control device according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating an implementation example of an operation control device according to an embodiment of the present invention.

FIG. 3 is an explanatory diagram for explaining processing contents of a part region estimating unit in the embodiment of the present invention in the case of head region estimation.

FIG. 4 is a flowchart illustrating a head region estimation algorithm of a part region estimation unit according to the embodiment of the present invention.

FIG. 5 is an explanatory diagram for describing processing performed by a part region estimating unit according to the embodiment of the present invention in the case of estimating both arms and a torso region.

FIG. 6 is a flowchart illustrating a body part region extraction algorithm of a part region estimating unit according to the embodiment of the present invention.

FIG. 7 is a flowchart illustrating a head rotation estimation algorithm of a part motion estimation unit according to the embodiment of the present invention.

FIG. 8 is an explanatory diagram for explaining processing contents of a part motion estimating unit according to the embodiment of the present invention when a human shakes his / her head.

FIG. 9 is an explanatory diagram for explaining processing contents of a part motion estimating unit according to the embodiment of the present invention when a human shakes his / her head vertically.

FIG. 10 is a diagram showing time-series data of a head rotation having a horizontal axis as a rotation axis, which is output from a region motion estimation unit according to the embodiment of the present invention.

FIG. 11 is a diagram for calculating a one-dimensional motion vector.

FIG. 12 is a diagram illustrating an example of a projection at a previous time and a projection at a current time, and an estimation direction of a one-dimensional motion vector.

FIG. 13 is a flowchart illustrating the principle of one-dimensional motion vector calculation.

FIG. 14 is a flowchart illustrating a process of calculating a projection array index k.

FIG. 15 is a diagram showing a specific configuration for calculating an optical flow.

FIG. 16 is a flowchart illustrating a specific method of calculating an optical flow.

FIG. 17 is a diagram showing a specific device for calculating an optical flow.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 Image input part, 2 part area | region estimation part, 3 part operation | movement estimation part, 4 operation reproduction part, 5 imaging | photography object, 6,7 image sensor, 8 operation analysis computer, 9 operation analysis result display monitor, 10 network, 11 Motion reproduction computer, 12 human model, 13 motion reproduction monitor, 14
a silhouette, 14b smoothed silhouette width distribution, 15 estimated neck position, 16 estimated head position, 17 estimated head region left end, 18 estimated head region right end, 19 estimated right arm rectangular region left end, 20 estimated torso region left end, 21 Estimated torso region right end, 22 Estimated left bowl rectangular region right end, 23a
₁ , 23a ₂ optical flow, 24a ₁ , 24a ₂
Averaged horizontal projection, 25a ₁ , 25a ₂ Averaged vertical projection, 26 horizontal section of human head, 27 vertical section of human head, 28X vertical axis of rotation, 28Y horizontal axis of rotation.

Claims (10)

[Claims] An image input unit that obtains a plurality of two-dimensional images by photographing an image including a movement of a photographing target from a plurality of photographing angles by a plurality of image sensors; Calculating the difference between two consecutive images, analyzing each of the plurality of difference images obtained for each image sensor, and estimating an image region including the imaging target region for each difference image An estimating unit, estimating a motion in each area estimated by the estimating unit, and estimating a three-dimensional motion of each part using a difference image obtained simultaneously for each of a plurality of image sensors. An operation control device comprising: a unit; and a target operation unit that operates a control target based on the motion of each part estimated by the part motion estimation unit. 2. The part region estimating unit binarizes each pixel value of a difference image to obtain a silhouette of a photographing target, analyzes the width of the obtained silhouette, and analyzes an image region including the part of the photographing target. The operation control device according to claim 1, further comprising a unit that estimates 3. The operation control device according to claim 1, wherein the target operation unit includes a unit configured to reproduce a motion of the imaging target by a motion of a model represented by computer graphics. 4. A computer calculates a difference between two temporally continuous two-dimensional images of a plurality of two-dimensional images photographed from a plurality of photographing angles by a plurality of image sensors, and obtains a difference for each image sensor. A procedure for analyzing each of the plurality of difference images and estimating an image region including a part to be imaged for each of the difference images, calculating a motion of each of the estimated regions, and simultaneously obtaining a difference image for each of the plurality of image sensors A computer-readable recording medium on which a program for executing the steps of: estimating a three-dimensional movement of each part using; 5. An image input unit for photographing an image including a movement of a photographing target from a plurality of photographing angles by a plurality of image sensors to obtain a plurality of two-dimensional images, a temporal image photographed for each of the plurality of image sensors. Calculating the difference between two consecutive images, analyzing each of the plurality of difference images obtained for each image sensor, and estimating an image region including the imaging target region for each difference image Estimating unit, projecting each pixel value constituting the two temporally continuous images in the horizontal and vertical directions, calculating a one-dimensional optical flow for each image sensor based on the obtained projection data, A motion in each region estimated by the region estimation unit from a plurality of one-dimensional optical flows to be calculated, and a region motion estimation unit for estimating a three-dimensional motion of each region; Part An operation control device, comprising: a target operation unit that operates a control target based on each movement. 6. The operation control device according to claim 5, wherein the projection data is directly obtained by an image sensor equipped with a semiconductor integrated circuit having an image processing function. 7. The part region estimating unit binarizes each pixel value of the difference image to obtain a silhouette of a photographing target, analyzes a width of the obtained silhouette, and analyzes an image region including the part of the photographing target. The operation control device according to claim 5, further comprising a unit that estimates 8. The operation control device according to claim 5, wherein the target operation unit includes means for reproducing a motion of the shooting target by a motion of a model represented by computer graphics. . 9. A computer calculates a difference between two temporally continuous two-dimensional images of a plurality of two-dimensional images photographed from a plurality of photographing angles by a plurality of image sensors, and obtains a difference for each image sensor. A procedure of analyzing each of the plurality of difference images and estimating an image area including a part to be imaged for each difference image; and calculating pixel values of the two temporally continuous images in horizontal and vertical directions. To calculate the one-dimensional optical flow for each image sensor based on the obtained projection data, calculate the motion in each of the estimated regions from the obtained one-dimensional optical flows, and calculate the three-dimensional A computer-readable recording medium on which a program for executing the steps of: 10. A computer, comprising: a plurality of image sensors equipped with a semiconductor integrated circuit having an image processing function; and a difference between two temporally continuous two-dimensional images of a plurality of two-dimensional images photographed from a plurality of photographing angles. Calculating a plurality of difference images obtained for each image sensor, and estimating an image region including a part to be imaged for each difference image, and projecting data directly obtained by the image sensor. Calculating a one-dimensional optical flow for each image sensor based on the above, calculating the motion in each of the estimated regions from the obtained one-dimensional optical flows, estimating the three-dimensional motion of each part, A computer-readable recording medium on which a program to be executed is recorded.