JP2000125196A - Input image processing method, input image processor, and recording medium recording input image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain an input image processor where an accurate depth of an image is measured in spite of a small calculation quantity so as to process a photographed image in real time and that copes with even a case that a background plane of the photographed image is wide. SOLUTION: The input image processor 100 is provided with camera sections 102, 103 that photograph an object moving on a background plane to acquire an input image, object extracted image generating sections 108, 109 that generate an object extracted image from the input image, a difference image generating section 110 that generates a difference image from the object extracted image, an occlusion image generating section 111 that generates an occlusion image from the object extracted image and the difference image and a distance measurement section that measures a position of the object with respect to the background plane based on the occlusion image. Thus, an accurate depth of the image is measured in spite of a small calculation quantity and the photographed image is processed in real time.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an input image processing method for processing an input image, an input image processing device, and a recording medium on which an input image processing program is recorded. More particularly, the present invention relates to an input image processing method for processing an image input from an image input unit such as a CCD camera, an input image processing device, and a recording medium on which an input image processing program is recorded.

[0002]

2. Description of the Related Art In recent years, an input image processing apparatus for inputting information using a camera such as a CCD camera and processing the input image using a camera such as a CCD camera instead of a mouse apparatus or a keyboard apparatus as an image input apparatus has been researched and developed. .

FIG. 11 shows a computer system including a conventional input image processing apparatus using a camera such as a CCD camera in an image input section. The computer system includes an input image processing device 1100 that processes an input image,
Image processing device 1 connected to input image processing device 1100
111.

In FIG. 11, an input image processing device 110
0 denotes a camera unit A1102 and a camera unit B1103 such as a CCD camera for taking an image, and a camera unit A1.
A synchronization signal generation unit 1101 for transmitting a synchronization signal for photographing to the camera unit 102 and the camera unit B 1103;
A which converts an analog signal of each image taken by the camera unit 02 and the camera unit B 1103 into a digital signal.
/ D converters 1104 and 1105 and A / D converter 11
Feature point extraction units 1106 and 11 for extracting feature points of the respective digital images output from 04 and 1105
07, a corresponding point search unit 1108 that searches for a one-to-one correspondence between the respective feature points extracted by the feature point extraction units 1106 and 1107, and three of the respective feature points associated by the corresponding point search unit 1108. A distance measuring unit 1109 for measuring dimensional coordinate values based on triangulation, and a distance measuring unit 1109
Of the image based on the distance measurement result, and a control signal based on the recognition, for example, a control signal similar to a click operation with a mouse, is transmitted to the image processing apparatus 111.
1 is provided with an input processing unit 1110 for inputting the data to the input unit 1.

[0005] The image processing apparatus 1111 receives a control signal from the input processing section 1110 and performs processing based on the control signal; a ROM (Read Only Memory) 1115 for storing a control program and the like;
RAM (Random Access Memory) for storing data
1116, a display unit 1114 that displays an image, and a display control unit 1113 that controls the display unit 1114.

In the image processing system including the input image processing device and the image processing device 1211 described above, a camera such as a CCD camera is used instead of a mouse device or a keyboard device as an input device, and an operator (user) is used. The operation (action) is photographed, the camera image is processed by the input image processing device, and the image processing device 1 is recognized by recognizing the action of the operator.
A control signal is input to 111.

Hereinafter, the operation of the input image processing apparatus will be described. Here, automatic input of a piano performance will be described. In this case, the camera unit A 1102 and the camera unit B
The image photographed in 1103 is subjected to image processing, and input processing of a control signal to the image processing apparatus 1111 is performed. For this reason, there is no need for a piano keyboard that actually produces a sound, and it is sufficient to have a picture that imitates the keyboard.

FIG. 12 is a conceptual diagram showing processing of a photographed image. FIGS. 12A1 and 12A2 are camera images obtained by photographing a piano being played by two camera units A1102 and B1103 provided in the vertical direction of the picture depicting the keyboard. 12 (a1) is the camera unit A
FIG. 12A shows an image of the camera unit B 1103 (right camera). FIG.
FIGS. 2 (b1) and (b2) are diagrams showing epipolar lines and feature points in the camera images of FIGS. 12 (a1) and (a2), respectively.

Here, a process for inputting performance data from a camera image to the image processing device 1111 will be described with reference to FIGS. First, the camera section A11
In order to synchronize the captured frames of the camera images (FIGS. 12A1 and 12A2) input from the camera unit B1103 and the camera unit B1103, the synchronization signal generation unit 1101
A synchronization signal is sent to the camera unit 1102 and the camera unit B 1104. Upon receiving the synchronization signal from the synchronization signal generation unit 1101,
The camera unit A 1102 and the camera unit B 1103 capture the situation of the piano performance, and obtain analog signals of the captured images 1211 and 1212. A / D converters 1104 and 1105
Converts the analog signals of the captured images 1211 and 1212 into digital signals (digital images).

Next, in order to detect the position of the keyboard 1214 pressed by the fingertip 1213, the camera unit A1102
And the fingertip 1213-3 from the position of the camera unit B1103.
Find the dimension depth.

In order to determine the value of the three-dimensional depth of each fingertip 1213, the feature point extracting units 1106 and 1107
The feature points of the fingertip 1213 are extracted from the digital image (FIGS. 12A and 12A). This feature point is extracted by previously determining the target of the feature point such as the fingertip 1213 in the captured images 1211 and 1212.

Next, the corresponding point searching unit 1108 generates the captured image 12 of the camera unit A 1102 and the camera unit B 1103.
11. Each feature point 120 extracted from 11 and 1212
A one-to-one correspondence between 1 and 1210 is searched. This corresponding point search is performed for each feature point 1201 to 1201 for each epipolar line 1215.
Search for 1210 and put one on one epipolar line 1215
A feature that has only two feature points 1201 to 1210 is extracted. As a result, the one-to-one correspondence points are
Corresponding points 1202 and 120 in (b1) and (b2)
7, 1203 and 1208, and 1204 and 1209. Also, feature points 1201, 1205, 1
As in 206 and 1210, the same epipolar line 1
When a plurality of entities are searched for on the reference point 215, the correspondence between the feature points cannot be determined only from the constraint condition of the epipolar line 1215.
Corresponding points 1201 and 1206 and 1205 and 1210 are determined based on the coordinates of 6 and 1210 and a new epipolar line due to the addition of a camera unit.

The distance measuring unit 1109 measures the three-dimensional coordinate values of the feature points 1201 to 1210 associated with each other based on triangulation.

Subsequently, based on the depth value of the fingertip 1213 measured by the distance measuring unit 1109, the input processing unit 111
A value of 0 determines whether or not the keyboard 1214 has been pressed, and outputs a signal indicating the position of the keyboard for which the determination has been made to the image processing apparatus 1111.

The control unit 1112 of the image processing device 1111
Is a ROM based on a signal from the input processing unit 1110.
It controls an image processing program and data stored in the RAM 1115, the RAM 1116, and the like. Display control unit 11
13 displays an image on the display unit 1114 based on the processing result of the control unit 1112.

[0016]

However, FIG.
In the conventional input image processing apparatus as shown in FIG.
Since it is necessary to extract 1 to 1210 by searching the entire epipolar line 1215 of the images 1211 and 1212, the amount of calculation in the extraction of the feature points 1201 to 1210 increases and the processing time is increased. For this reason, there is a problem that real-time properties in image processing are lacking.

If there are a plurality of target images having similar characteristics such as the fingertip 1213 in the same image,
Since the matching of the corresponding points may occur, there has been a problem that erroneous three-dimensional measurement based on the erroneous feature point association is performed.

Further, the range in which three-dimensional measurement is possible is limited to the overlapping portion of the image pickup areas of a plurality of cameras (two in FIGS. 11 and 12) used for photographing. There was a problem that the shooting range had to be narrowed.

Accordingly, an object of the present invention is to solve the above-mentioned problems, to measure the depth of a three-dimensional image accurately with a small amount of calculation and to process the captured image in real time. An object of the present invention is to provide an input image processing method, an input image processing device, and a recording medium on which an input image processing program is recorded, which can cope with a case where an image has a wide background surface.

[0020]

According to a first aspect of the present invention, there is provided an input image processing method for processing an image of a subject moving on a background surface. In, the subject moving on the background surface is photographed simultaneously by a plurality of photographing units to acquire a plurality of input images, and only the subject is extracted from each of the input images,
Generating a plurality of subject extraction images, generating a difference image from the generated plurality of subject extraction images, generating an occlusion image from an arbitrary subject extraction image and the difference image among the plurality of subject extraction images, and generating an occlusion image Measuring the position of the subject with respect to the background based on the background. At this time, the relative positions of the plurality of imaging units are known, and the optical axis directions of the imaging units are parallel to each other, and the respective imaging planes are set on a plane parallel to the background plane.

According to another aspect of the present invention, there is provided an input image processing apparatus for processing an image of a subject moving on a background. A plurality of photographing means for simultaneously photographing a subject moving on the to obtain a plurality of input images,
Extraction image generation means for extracting only the subject from the plurality of input images acquired by the imaging means to generate a plurality of subject extraction images, and generating a difference image from the plurality of subject extraction images generated by the extraction image generation means Image generating means for generating an occlusion image from an arbitrary object extracted image and a difference image among a plurality of object extracted images, and an object based on the occlusion image generated by the occlusion image generating means. And a distance measuring means for measuring a position with respect to the background surface. At this time, it is preferable that the relative positions of the plurality of photographing units are known, and the optical axis directions of the photographing units are parallel to each other, and the respective imaging planes are set on a plane parallel to the background surface.

Thus, when processing a motion image of a subject using a camera or the like, the depth of the subject can be accurately measured with a small amount of calculation. Thereby, high-speed and accurate input image processing capable of performing real-time processing becomes possible.

According to another aspect of the present invention, there is provided an input image processing method for processing an image of a subject moving on a background surface. A plurality of input images are obtained by simultaneously photographing a subject moving on the camera with a plurality of photographing units, a normalized image is generated from the plurality of input images, and only the subject is extracted from each normalized image. Generating a plurality of subject extraction images, generating a difference image from the generated plurality of subject extraction images, generating an occlusion image from any of the plurality of subject extraction images and the difference image, and generating an occlusion image The position of the subject with respect to the background plane is measured based on Here, the plurality of photographing units may be installed such that their relative positional relationships are known, their optical axis directions are known, and they are congested.

According to another aspect of the present invention, there is provided an input image processing apparatus for processing an image of a subject moving on a background surface. A plurality of photographing means for simultaneously photographing a subject moving on the to obtain a plurality of input images,
A normalized image generating means for generating respective normalized images from a plurality of input images obtained by the photographing means; and extracting only a subject from the respective normalized images generated by the normalized image generating means, Extraction image generation means for generating a subject extraction image; difference image generation means for generating a difference image from a plurality of subject extraction images generated by the extraction image generation means; and an arbitrary subject extraction image of the plurality of subject extraction images And an occlusion image generating means for generating an occlusion image from the difference image; and a distance measuring means for measuring a position of the subject with respect to a background surface based on the occlusion image generated by the occlusion image generating means. Here, the plurality of photographing units may be installed such that their relative positional relationships are known, their optical axis directions are known, and they are congested.

This makes it possible to process a motion image of a subject even when the background required for input is large.

According to a third aspect of the present invention, there is provided an input image processing method according to the third aspect of the present invention, further comprising the step of generating an occlusion image in the first and second aspects. Generating two or more occlusion images from two or more arbitrary subject extracted images of the plurality of subject extracted images and the difference image, and measuring the position of the subject is performed for each of the two or more occlusion images. Is characterized by a step of measuring a position with respect to a background surface and determining a position of a subject based on an average value of positions measured for each of two or more occlusion images.

According to a third aspect of the present invention, there is provided an input image processing apparatus according to the first and second aspects, wherein a plurality of occlusion image generating means are provided. The configuration is such that two or more occlusion images are generated from two or more arbitrary subject extracted images of the subject extracted images and the difference image, and the distance measuring means measures the position of the subject with respect to the background plane for each of the two or more occlusion images The position of the subject is determined based on the average value of the positions measured for each of two or more occlusion images.

Thus, the accuracy of three-dimensional measurement can be improved.

According to a fourth aspect of the present invention, there is provided an input image processing method according to the first to third aspects, wherein the position of the subject is measured. The measured position is corrected by a predetermined correction value. At this time, in the step of measuring the position of the subject, the position of the subject is determined by a value Z indicating the distance from the background surface to the subject.
h ′, the predetermined correction value is a width value Z0 in a direction perpendicular to the background surface of the subject, and the value of the measured position is Zh.
, The value Zh 'indicating the distance to the subject is Zh
'= Zh-Z0.

According to a fourth aspect of the present invention, there is provided an input image processing apparatus according to the first to third aspects, wherein the distance is measured by distance measuring means. The position is corrected by a predetermined correction value. Here, in the distance measuring means,
The position of the subject is a value Zh 'indicating the distance from the background surface to the subject, the predetermined correction value is a width value Z0 in a direction perpendicular to the background surface of the subject, and the measured position value is Zh.
h, the value Zh indicating the distance to the subject is Zh
'= Zh-Z0.

Thus, the accuracy of three-dimensional measurement can be improved.

According to another aspect of the present invention, there is provided an input image processing method according to any one of the first to fourth aspects.
The step of acquiring a plurality of input images may be a step of acquiring a plurality of input images by simultaneously photographing a subject moving on a background surface with a plurality of photographing units while synchronizing. In the step of generating a plurality of subject extraction images, it is preferable that only the subject is extracted based on the background image in which the subject is not shown. Further, in the step of generating the difference image, an exclusive OR (EX) for each corresponding pixel of the generated plurality of subject extraction images is provided.
OR: EXclusive-OR) to generate a difference image. Further, in the step of generating an occlusion image, an occlusion image may be generated by calculating a logical product (AND) of corresponding pixels of an arbitrary subject extracted image and a difference image. Furthermore, after the step of measuring the position of the subject,
Recognize the movement of the subject from the measured position of the subject,
You may do so.

Further, in order to solve the above-mentioned problem, in the input image processing apparatus according to the first to fourth aspects,
The plurality of photographing units may be configured to simultaneously photograph a subject moving on the background surface while synchronizing with each other to acquire a plurality of input images. Further, the output image generating means may extract only the subject based on the background image in which the subject is not shown. Further, the difference image generating means is an exclusive OR (EXOR: EXOR-OR) for each corresponding pixel of the plurality of generated subject extraction images.
, A difference image can be generated. Further, the occlusion image generating means may generate an occlusion image by taking a logical product (AND) of corresponding pixels of an arbitrary subject extracted image and a difference image. Further, it is possible to have means for recognizing the motion of the subject from the position of the subject measured by the distance measuring means.

The input image processing methods according to the first to fourth aspects of the present invention can be recorded as an input image processing program on a computer-readable recording medium.

[0035]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an input image processing method, an input image processing device, and a recording medium on which an input image processing program is recorded according to the present invention will be described with reference to the drawings.

<Embodiment 1> FIG. 1 is a block diagram of a computer system including an input image processing apparatus of the present invention. The computer system includes an input image processing device 100 that processes an input image and an input image processing device 1
00 and an image processing apparatus 114 connected to the image processing apparatus.

In FIG. 1, an input image processing device 100
Are two camera units A102 and B103 for photographing a moving subject on a background surface 120 having a texture, and two camera units A102 and B103.
Synchronizing signal generating unit 10 for transmitting a synchronizing signal for photographing to
1 and an A / D converter 104 for converting an analog signal of each image taken by the camera unit A102 and the camera unit B103 into a digital signal (digital image).
And 105, background image storage units 106 and 107 for storing only the background image of the background plane 120, and the A / D conversion unit 10
Only the subject is extracted based on each digital image output from each of the digital images 4 and 105 and each background image stored in the background image storage units 106 and 107, and the extraction result (signal) is binarized. And a difference image generation unit 110 that generates a difference image between the two subject extraction images generated by the subject extraction image generation units 108 and 109. An occlusion detecting section 111 for detecting an occlusion area from the subject extracted image generated by one subject extracted image generating section (the subject extracted image generating section 108 in FIG. 1) and the differential image generated by the differential image generating section 110; From the occlusion area detected by the occlusion detection unit 111, three-dimensional position measurement of the subject with respect to the background surface 120 is performed. And the Hare distance measurement unit 112, recognizes a like motion of the image based on the distance measurement result of the distance measurement unit 112, the image processing apparatus control signal based on the recognition 114
And an input processing unit 113 for inputting the information to the user.

Here, the camera unit A102 and the camera unit B
Reference numerals 103 have a known relative positional relationship, and are arranged on the same plane whose optical axis directions are parallel to each other and whose imaging plane is parallel to the background plane.

The image processing device 114 receives a control signal from the input processing unit 113 and performs processing based on the control signal, and a ROM (Read Only Memory) 119 for storing a control program and the like. (Random Access Memory) 118 for storing data, data, etc.
, A display unit 117 for displaying an image, and a display control unit 116 for controlling the display unit 117.

Hereinafter, image processing in the computer system having the above configuration will be described with reference to FIGS. This image processing will be described using an example of performance input using a picture of a piano keyboard.

FIG. 2 shows an input image processing apparatus 100 according to the present invention.
6 is a flowchart showing the operation of the first embodiment. Here, the background surface 1
Only 20 backgrounds (pictures of the piano keyboard) are stored in the camera section A1 in advance.
02 and the camera unit B103, digital conversion processing is performed by the A / D conversion units 104 and 105, and the digital images are stored in the background image storage units 106 and 107 as background images.

In FIG. 2, first, the operator performs a performance operation on a background surface 120 of a picture depicting a keyboard of a piano. At this time, the camera unit A102 and the camera unit B103
Captures an image of a finger within the angle of view as an analog signal in synchronization with a synchronization signal generated from the synchronization signal generator 101 (steps 201A and 201B).

This analog signal is supplied to the A / D converter 10
At 4 and 105, the signal is converted into a digital signal (digital image). Hereinafter, generation of a difference image for performing distance measurement by the distance measurement unit 112 will be described.

FIG. 3 is a conceptual diagram showing generation of a difference image. In FIGS. 1 and 3, first, the subject extraction image generation units 108 and 109 generate an image 3 of only the background (texture) 309 stored in the background image storage units 106 and 107.
01 and 303 and images (302) and 304 (304) containing a subject (fingers) 310 in a background 309 (A / D converters 104 and 105).
The difference is calculated by calculating the difference between each of the digital images converted in (Step 202A, 202).
B).

Then, based on the difference image, the portion where the subject 310 exists is “1”, and the portion of only the background 309 is “0”, and the binarization is performed.
06 is generated (steps 203A and 203B). FIG.
In the subject extraction images 305 and 306 of
The portion “1” where 10 is present is blacked out.

Next, the difference image generation unit 110 obtains an exclusive OR (EXOR: EXOR-OR) for each corresponding pixel of the subject extracted images 305 and 306, and obtains a difference image 307.
Is generated (step 204). Here, the positional relationship (correspondence relationship between pixels) between the subject extraction image 305 and the subject extraction image 306 when taking the difference is the texture (picture of a piano keyboard) 309 of the background surface 120 photographed in the background images 301 and 303. Are set at overlapping positions without displacement. In the difference image 307 of FIG. 3, the overlapped portion of the subject 310 in the subject extracted images 305 and 306 is “0”.
(Open).

Subsequently, the occlusion detecting section 111
An occlusion image 308 is generated by taking the logical product (AND) of one of the subject extracted images (the subject extracted image 305 in FIG. 3) and the difference image 307 for each corresponding pixel (step 205). Here, the positional relationship (correspondence relationship between pixels) between the subject extraction image 305 and the difference image 307 at the time of calculating the logical product is set to be the same as the positional relationship set by the difference image generation unit 110 described above.

Subsequently, the distance measuring unit 112 obtains the number of pixels of the width of the occlusion area 311 from the occlusion area 311 of the occlusion image 308 along the boundary line of the subject 310. Based on the number of pixels having the width of the occlusion area 311, a coordinate value of the subject 310 in a three-dimensional coordinate system defined by the background surface 120 is obtained (Step 206).

FIG. 4 is a diagram showing how to obtain the coordinate values of the subject 310. FIG. 4A shows a background surface 120 and a subject 310, and a coordinate system 403 defined there. FIG. 4B shows the camera focal points (pinhole positions) 404 and 405 and the imaging planes 406 and 40 viewed from above the Y axis.
7, the positional relationship between the subject 310 and the background surface 120 is shown.

Now, the camera unit A102 and the camera unit B10
3 are the coordinates (xc1, yc, -zc) and (xc1,
2, yc, -zc). In FIG. 4, a camera unit A102 and a camera unit B103
Is described as a pinhole camera model. The distance between the pinholes 404 and 405 is B, the distance from the pinholes 404 and 405 to the background 120 is L,
The respective imaging planes 40 from the pinholes 404 and 405
F is the distance (focal length) to 6, 407, p is the number of pixels in the width of the occlusion area 311 determined from the occlusion image 308 (FIG. 3), and occlusion area 3
Assuming that the actual length of the width of 11 is x0,

Equation 1 holds.

## EQU1 ## p: x0 = F: L∴x0 = (p × L) / F

When the distance from the background surface 120 to the object (in this case, the fingertip) 310 is Zh, the similarity between the triangles is as follows.

Equation 2 holds.

## EQU2 ## B: x0 = (L-Zh): Zh∴Zh = (L × x0) / (B + x0)

Where

X0 obtained by the following equation

By substituting into Equation 2,

Equation 3 is obtained.

Zh = L / [1 + {(B × F) / (p × L)}]

This

In Equation 3, since the values of L, p, F, and B are all known, the distance Zh from the background surface 120 to the subject 310 can be obtained.

The distance measuring unit 112

The distance Zh from the background plane 120 to the subject 310 is calculated by the following equation (3), and this value is calculated as
20 together with the xy coordinate values.

Subsequently, in the input processing unit 113, the background 1
Based on the distance Zh from the background 20 (picture depicting the keyboard of the piano) 309 to the subject (fingertip) 310, it is determined that the keyboard has been pressed (step 207).

When it is determined that the keyboard has been pressed based on a predetermined threshold or the like, the input processing unit 113 sets the subject (fingertip) 3
The (x, y) coordinate value indicating the position of the keyboard corresponding to 10 is sent to the image processing device 114 as a control signal (step 208). On the other hand, if it is not determined that the key is pressed, the process is continued (step 209).

Hereinafter, steps 201A and 201B to 209 described above are performed until the measurement processing of the image is completed (in the case described above, for example, until the performance of the piano is completed).
The processing up to is repeated (step 210).

The control unit 115 of the image processing device 114 determines the RO based on the coordinate value (signal) from the input processing unit 113.
It processes programs and data stored in M119, RAM 118, and the like. The control unit 115 controls the display control unit 116 based on the processing result, and
And an audio output unit (not shown) outputs an audio.

As described above, according to the input image processing apparatus and the input image processing method of the present invention, when an image is input using a camera, the distance between the background surface and the subject can be accurately calculated with a small amount of calculation. Can be measured. As a result, high-speed and stable input image processing capable of processing an image in real time becomes possible.

<Embodiment 2> Next, an input image processing apparatus and an input image processing method of the present invention will be described with reference to the drawings.

FIG. 5 is a block diagram of a computer system including the input image processing device of the present invention. In addition,
5, the same components as those in FIG. 1 are denoted by the same reference numerals.

In FIG. 5, the computer system includes an input image processing device 500 for processing an input image, and an image processing device 114 connected to the input image processing device 500.
And

The input image processing apparatus 500 includes two camera units A 502 and B 503 for photographing a moving object on the background surface 120 having a texture.
A synchronizing signal generating unit 101 for transmitting a synchronizing signal of photographing to the camera unit A502 and the camera unit B503; and a digital signal (digital image) of an analog signal of each image photographed by the camera unit A502 and the camera unit B503. A / D converters 104 and 105 for converting to
And the digital image converted by the A / D conversion units 104 and 105 is captured by two camera units whose optical axis directions are parallel to each other, and the same image (hereinafter referred to as an image whose imaging surface is parallel to the background surface 120) (Referred to simply as “normalized image”) and a background image storage unit 106 that stores only the background image of the background surface 120.
And 107, the respective normalized images output from the normalized image generators 506 and 507, and the background image storage 10
6 and 107, only the subject is extracted based on the respective background images stored therein, binarized extraction results (signals) are used to generate subject extraction images, and subject extraction image generation units 108 and 109 , Subject extraction image generation unit 1
A difference image generation unit 110 that generates a difference image between the two subject extraction images generated in 08 and 109 and one of the subject extraction image generation units (the subject extraction image generation unit 108 in FIG. 1). An occlusion detection unit 111 that detects an occlusion area from the subject extraction image and the difference image generated by the difference image generation unit 110, and measures a three-dimensional position of the subject with respect to the background surface 120 from the occlusion area detected by the occlusion detection unit 111. The image processing apparatus 114 includes a distance measurement unit 112 that performs the operation, and an input processing unit 113 that recognizes a motion of an image based on the distance measurement result of the distance measurement unit 112 and inputs a control signal based on the recognition to the image processing device 114. .

Here, the camera unit A 502 and the camera unit B
Reference numerals 503 are arranged such that their relative positional relationships are known, and their optical axis directions are known and congested.

The image processing device 114 receives a control signal from the input processing unit 113 and performs processing based on the control signal, a ROM 119 for storing a control program and the like, and stores data and the like. RAM
118, a display unit 117 for displaying an image, and a display unit 11
And a display control unit 116 for controlling the control unit 7.

The difference between the input image processing device 500 shown in FIG. 5 and the input image processing device 100 shown in FIG.
In the input image processing apparatus 500 of FIG.
And the relative positional relationship between the camera unit B503 and each other is known,
The point is that normalization image generation units 506 and 507 are arranged so that their optical axis directions are known and congested, and normalize and convert a digital image into a normalization image.

By configuring the input image processing apparatus of the present invention as described above, the camera unit A502 and the camera unit B5
The range of the background surface 120 that can be photographed at 03 can be expanded.

Hereinafter, the input image processing device 50 shown in FIG.
The operation of 0 will be described by taking a piano performance as an example as described above.

FIG. 6 is a flowchart showing image processing in the input image processing device 500. In FIG. 6, the same steps as those in FIG. 2 described above are denoted by the same step numbers. Here, the background surface 120
Only the background (picture of the piano keyboard) of the camera unit A102
And A / D converter 1
In steps 04 and 105, digital conversion processing is performed, and the digital images are stored in the background image storage units 106 and 107 as background images.

In FIG. 6, similar to FIG. 2, first,
The operator operates the background 120 of the picture of the piano keyboard.
Perform the performance operation above. At this time, the camera unit A502 and the camera unit B503 take in an image of the finger within the angle of view as an analog signal in synchronization with the synchronization signal generated from the synchronization signal generation unit 101 (steps 601A and 601B). The analog signals are converted into digital signals (digital images) by A / D converters 104 and 105.

Next, the normalized image generation units 506 and 507
Captures digital images converted by the A / D converters 104 and 105 based on the known optical axis directions in which the camera units A502 and B503 are congested by two camera units whose optical axis directions are parallel to each other. And its imaging surface is the background surface 12
Normalization conversion is performed to the same image (normalized image) as the image parallel to 0 (steps 602A and 602B). The generation of such a normalized image may be performed by a general normalization (rectification) process used in a binocular stereoscopic method or the like.

Thereafter, in the same manner as described with reference to FIG.
08 and 109 generate difference images (step 202).
A, 202B). Then, based on the difference image, 2
The binarization is performed to generate subject extraction images 305 and 306 (FIG. 3) (steps 203A and 203B). Next, the difference image generation unit 110 generates a difference image 307 (FIG. 3) based on the subject extraction images 305 and 306 (Step 204).

Subsequently, the occlusion detecting section 111
An occlusion image 308 (FIG. 3) is generated based on one of the subject extracted images (the subject extracted image 305 in FIG. 3) and the difference image 307 (step 205). Next, the distance measurement unit 112 obtains the coordinate value of the subject 310 (FIG. 3) based on the occlusion image 308 (Step 206).

Subsequently, in the input processing section 113, the object 3
Based on the coordinate values of 10, the keyboard is determined to be pressed by a predetermined threshold or the like (step 207). Input processing unit 113
Indicates that the subject (fingertip) 31
The coordinate value of the keyboard corresponding to 0 is sent to the image processing device 114 as a control signal (step 208). On the other hand, if it is not determined that the key is pressed, the process is continued (step 209).

Hereinafter, steps 601A and 601B to 209B until the image measurement processing ends (in the case described above, for example, until the piano performance ends).
The processing up to is repeated (step 210).

The control unit 115 of the image processing device 114 determines the RO based on the coordinate value (signal) from the input processing unit 113.
It processes programs and data stored in M119, RAM 118, and the like. The control unit 115 controls the display control unit 116 based on the processing result, and
And an audio output unit (not shown) outputs an audio.

As described above, according to the input image processing apparatus and the input image processing method of the present invention, it is possible to process an image showing the action of the operator even when the background required for inputting the image is wide. it can.

<Embodiment 3> Next, another embodiment of the present invention will be described. A feature of the present embodiment is that measurement accuracy is further improved by detecting a plurality of occlusion regions. Here, the input image processing device according to the present embodiment has the same configuration as input image processing device 100 shown in FIG. Next, the operation of the input image processing device according to the present embodiment will be described.

FIG. 7 is a flowchart showing image processing in the input image processing apparatus 100. Note that, in FIG. 7, the same steps as those in FIG. 2 described above are denoted by the same step numbers. Here, the background surface 120
Only the background (picture of the piano keyboard) of the camera unit A102
And A / D converter 1
In steps 04 and 105, digital conversion processing is performed, and the digital images are stored in the background image storage units 106 and 107 as background images.

In FIG. 7, similar to FIG. 2, first,
The operator operates the background 120 of the picture of the piano keyboard.
Perform the performance operation above. At this time, the camera unit A102 and the camera unit B103 capture an image of the finger within the angle of view as an analog signal in synchronization with the synchronization signal generated from the synchronization signal generation unit 101 (steps 201A and 201B). The analog signals are converted into digital signals (digital images) by A / D converters 104 and 105.

FIG. 8 is a conceptual diagram showing generation of a difference image. 1 and 8, first, the subject extraction image generation units 108 and 109 generate an image 3 of only the background (texture) 309 stored in the background image storage units 106 and 107.
01 and 303 and images (302) and 304 (304) containing a subject (fingers) 310 in a background 309 (A / D converters 104 and 105).
The difference is calculated by calculating the difference between each of the digital images converted in (Step 202A, 202).
B).

Then, based on the difference image, the portion where the subject 310 exists is “1”, and the portion of only the background 309 is “0”, and the binarization is performed.
06 is generated (steps 203A and 203B). FIG.
In the subject extraction images 305 and 306 of
The portion “1” where 10 is present is blacked out.

Next, the difference image generation unit 110 calculates the exclusive OR (EXOR: Exclusive-OR) for each corresponding pixel of the subject extraction images 305 and 306, and obtains the difference image 307.
Is generated (step 204). Here, the positional relationship (correspondence relationship between pixels) between the subject extraction image 305 and the subject extraction image 306 when taking the difference is the texture (picture of the piano keyboard) 309 of the background surface 120 captured in the background images 301 and 303. Are set at overlapping positions without displacement. In the difference image 307 of FIG. 8, the overlapped portion of the subject 310 in the subject extracted images 305 and 306 is “0”.
(Open).

Subsequently, the occlusion detecting section 111
The logical product (AND) of each corresponding pixel of the subject extraction image 305 and the difference image 307 is calculated to obtain the occlusion image 8
08 is generated (step 705A). Here, the positional relationship (correspondence relationship between pixels) between the subject extraction image 305 and the difference image 307 at the time of calculating the logical product is the same as the difference image generation unit 1 described above.
The position is set to be the same as the positional relationship set at 10. Similarly, the occlusion detecting unit 111 generates an occlusion image 809 by taking the logical product (AND) of each corresponding pixel of the subject extraction image 306 and the difference image 307 (step 705B).

Next, the distance measuring section 112 executes the occlusion images 308 and 30 in the same manner as described with reference to FIG.
Based on the occlusion areas 311A and 311B, the coordinate value Zh of the subject 310 is obtained (step 20).
6). The coordinate value Zh is the coordinate value Z calculated based on the respective occlusion areas 311A and 311B.
The average value of hA and ZhB may be used.

Subsequently, in the input processing unit 113, the subject 3
Based on the ten coordinate values Zh, the keyboard press is determined by a predetermined threshold or the like (step 207). Input processing unit 1
13 indicates a subject (fingertip) when it is determined that a key is pressed.
The coordinate value of the keyboard corresponding to 310 is sent to the image processing device 114 as a control signal (step 208). on the other hand,
If it is not determined that the key is pressed, the processing is continued (step 209).

Hereinafter, steps 201A and 201B to 209 are performed until the image measurement processing is completed (in the case described above, for example, until the performance of the piano is completed).
The processing up to is repeated (step 210).

The control unit 115 of the image processing device 114 determines the RO based on the coordinate value (signal) from the input processing unit 113.
It processes programs and data stored in M119, RAM 118, and the like. The control unit 115 controls the display control unit 116 based on the processing result, and
And an audio output unit (not shown) outputs an audio.

As described above, according to the embodiment of the present invention, since two occlusion images 808 and 809 are generated and the average value of the coordinate values Zh of the subject 310 is obtained from them, the accuracy of distance measurement is improved. Can be done.

<Embodiment 4> Next, another embodiment of the present invention will be described. In the input image processing of the present invention, the subject 310 (FIG. 4) is shifted from the background plane 120 (FIG. 4).
When determining the distance Zh to the distance, the thickness of the subject is taken into consideration, so that the accuracy of the image operation determination (press determination) can be improved. Therefore, the feature of the present embodiment is that the shape of the subject 310 is modeled in the input processing unit 113, and the operation of the image is determined based on the model. As a result, the determination accuracy is further improved.

Here, the input image processing apparatus according to the present embodiment has the same configuration as the input image processing apparatus 100 shown in FIG. Next, the operation of the input image processing device according to the present embodiment will be described.

FIG. 9 is a flowchart showing image processing in the input image processing apparatus 100. In FIG. 9, the steps that perform the same processing as in FIG. 2 described above are given the same step numbers. Here, the background surface 120
Only the background (picture of the piano keyboard) of the camera unit A102
And A / D converter 1
In steps 04 and 105, digital conversion processing is performed, and the digital images are stored in the background image storage units 106 and 107 as background images.

In FIG. 9, similar to FIG. 2, first,
The operator operates the background 120 of the picture of the piano keyboard.
Perform the performance operation above. At this time, the camera unit A102 and the camera unit B103 capture an image of the finger within the angle of view as an analog signal in synchronization with the synchronization signal generated from the synchronization signal generation unit 101 (steps 201A and 201B). The analog signals are converted into digital signals (digital images) by A / D converters 104 and 105.

Next, the subject extraction image generators 108, 10
9 is an image 301 of only the background (texture) 309 (FIG. 3) stored in the background image storage units 106 and 107;
303 (FIG. 3) and a subject (finger) 310 in the background 309
Images 302 and 304 (A / D conversion unit 1) containing (FIG. 3)
Digital digital image converted in 04, 105)
Are calculated to generate difference images (steps 202A and 202B). Then, based on the difference image, the portion where the subject 310 exists is “1”, and the portion of only the background 309 is “0”, and the binarization is performed to generate subject extraction images 305 and 306 (FIG. 3). (Step 2
03A, 203B).

Next, the difference image generation unit 110 calculates an exclusive OR (EXOR: Exclusive-OR) for each corresponding pixel of the subject extraction images 305 and 306, and obtains a difference image 307.
(FIG. 3) is generated (step 204). Subsequently, the occlusion detection unit 111 compares one of the subject extracted images (the subject extracted image 305 in FIG. 3) with the difference image 307.
Then, an occlusion image 308 (FIG. 3) is generated based on (step 205).

Next, based on the occlusion image 308, the distance measuring section 112 calculates the coordinate value Zh of the subject 310 (FIG. 3).

FIG. 8 shows an example in which the subject (fingertip) 310 has a thickness. As described above, when the subject 310 has a thickness, the coordinate value Zh is 0 even when the pressing operation is performed on the background surface.
It does not become. Therefore, the input processing unit 113 models the thickness of the subject 310 in advance and stores a correction value (Z0 in FIG. 8) corresponding to the thickness. Then, the input processing unit 113 subtracts the stored correction value Z0 from the coordinate value Zh, which is the output result of the distance measurement unit 112, to obtain the actual coordinate value Zh ′ (Zh ′ = Zh−Z0). (Step 906). Thereby, the subject 31
The correction for the thickness of 0 is performed. Note that the coordinate value of the subject 310 may be corrected by the distance measurement unit 112.

Subsequently, in the input processing unit 113, the subject 3
Based on the 10 coordinate values Zh 'and a predetermined threshold value, it is determined whether the keyboard has been pressed (step 907).

If the input processing section 113 determines that the keyboard has been pressed, it sends the coordinate value of the keyboard corresponding to the subject (fingertip) 310 to the image processing device 114 as a control signal (step 208). On the other hand, if it is not determined that the key is pressed, the process is continued (step 20).
9).

Thereafter, steps 201A and 201B to 209 described above are performed until the measurement processing of the image is completed (in the case described above, for example, until the performance of the piano is completed).
The processing up to is repeated (step 210).

The control unit 115 of the image processing device 114 determines the RO based on the coordinate value (signal) from the input processing unit 113.
It processes programs and data stored in M119, RAM 118, and the like. The control unit 115 controls the display control unit 116 based on the processing result, and
And an audio output unit (not shown) outputs an audio.

As described above, according to the embodiment of the present invention, since the shape of the subject is modeled and corrected, the accuracy of the motion judgment of the image can be improved.

The input image processing apparatus and the input image processing method of the present invention have been described above. However, the third embodiment or the fourth embodiment can be applied to the second embodiment. Further, both the third embodiment and the fourth embodiment can be applied to the second embodiment. Further, the fourth embodiment can be applied to the third embodiment.

The above-described input image processing method can be recorded on a recording medium as an input image processing program which can be executed by a computer.

[0106]

As described above, according to the input image processing apparatus, the input image processing method, and the recording medium on which the input image processing program is recorded, the depth of a three-dimensional image can be accurately measured with a small amount of calculation. Thus, the captured image can be processed in real time, and it is possible to cope with a case where the background surface of the captured image is wide.

Further, by taking the average value of the coordinate values of the subject, distance measurement with high measurement accuracy can be performed.
Further, by applying the modeled correction value to the coordinate value of the subject, it is possible to improve the accuracy of the image motion determination.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an input image processing device of the present invention.

FIG. 2 is a flowchart showing input image processing of the present invention.

FIG. 3 is a conceptual diagram illustrating generation of a difference image.

FIG. 4 is a diagram illustrating a method of obtaining a coordinate value of a subject.

FIG. 5 is a block diagram illustrating a configuration of an input image processing device according to the present invention.

FIG. 6 is a flowchart illustrating input image processing according to the present invention.

FIG. 7 is a flowchart showing input image processing of the present invention.

FIG. 8 is a conceptual diagram illustrating generation of a difference image.

FIG. 9 is a flowchart showing input image processing of the present invention.

FIG. 10 is a diagram illustrating a method of obtaining a coordinate value of a subject.

FIG. 11 is a block diagram illustrating a configuration of a conventional input image processing device.

FIG. 12 is a diagram showing processing of a conventional captured image.

[Explanation of symbols]

 100, 500, 1100 Input image processing device 101 Synchronous signal generation unit 102, 502, 1102 Camera unit A 103, 503, 1103 Camera unit B 104, 105, 1104, 1105 A / D conversion unit 106, 107 Background image storage unit 108 , 109 subject extraction image generation unit 110 difference image generation unit 111 occlusion detection unit 112, 1109 distance measurement unit 113, 1110 input processing unit 114, 1111 image processing device 115, 1112 control unit 116, 1113 display control unit 117, 1114 display unit 118, 1116 RAM 119, 1115 ROM 120 Background surface 301, 303 Background image 302, 304 Digital image 305, 306 Subject extraction image 307 Difference image 308, 808, 809 Occlusion image 311, 311A 311B Occlusion area 403 Coordinate system 404, 405 Camera focus 406, 407 Image plane 506, 507 Normalized image generator 1201 to 1210 Feature points 1211, 1212 Photographed image 1213 Fingertip 1214 Keyboard 1215 Epipolar line

Continued on the front page (72) Inventor Junko Saito Waseda University, 3-4-1 Okubo, Shinjuku-ku, Tokyo (72) Inventor Shunichi Takeuchi 1-21-1, Nishiwaseda, Shinjuku-ku, Tokyo Inside the Waseda Research Center (72) Inventor Hideyoshi Tominaga 3-4-1 Okubo, Shinjuku-ku, Tokyo F-term within Waseda University 2F065 AA04 DD03 DD06 FF01 FF05 JJ03 JJ26 QQ03 QQ24 QQ25 QQ42 UU05 5B057 CE09 CH08 DA07 AA01 AA06 AA08 AA10 AA34 AA38 BA01 BA02 CA01 DA02 DA03 DA08

Claims (36)

[Claims] 1. An input image processing method for processing an image of a subject moving on a background surface, wherein a plurality of input images are obtained by simultaneously photographing a subject moving on the background surface by a plurality of photographing units. Extracting only the subject from each input image to generate a plurality of subject extracted images; generating a difference image from the generated plurality of subject extracted images; selecting any of the plurality of subject extracted images Generating an occlusion image from the subject extracted image and the difference image, and measuring a position of the subject with respect to the background surface based on the occlusion image. 2. The plurality of photographing units have a known relative positional relationship, and their optical axis directions are parallel to each other, and their respective imaging planes are set on a plane parallel to the background plane. The input image processing method according to claim 1, wherein: 3. An input image processing method for processing an image of a subject moving on a background surface, wherein the subject moving on the background surface is simultaneously photographed by a plurality of photographing units to obtain a plurality of input images. Generating respective normalized images from the plurality of input images; extracting only the subject from the respective normalized images to generate a plurality of subject extraction images; and generating the plurality of subject extraction images. A difference image is generated from the plurality of subject extraction images, an occlusion image is generated from any of the subject extraction images and the difference image, and a position of the subject with respect to the background surface is measured based on the occlusion image. An input image processing method comprising: 4. The input image according to claim 3, wherein the plurality of photographing units are installed so that their relative positional relationship is known, their optical axis directions are known, and they are congested. Processing method. 5. The method according to claim 1, wherein the step of generating the occlusion image includes the step of generating two or more occlusion images from two or more arbitrary subject extracted images of the plurality of subject extracted images and the difference image. Measuring the position of the subject with respect to the background surface for each of the two or more occlusion images, and determining the position of the subject based on an average value of the positions measured for each of the two or more occlusion images. The input image processing method according to any one of claims 1 to 4, further comprising a step of determining. 6. The step of measuring the position of the subject,
6. The input image processing method according to claim 1, wherein the measured position is corrected by a predetermined correction value. 7. The step of measuring the position of the subject, wherein the position of the subject is a value Zh ′ indicating a distance from the background surface to the subject, and the predetermined correction value is the value of the background of the subject. When a value of the width in the direction perpendicular to the surface is Z0 and the value of the measured position is Zh, the value Zh indicating the distance to the subject is obtained by Zh ′ = Zh−Z0. The input image processing method according to claim 6. 8. The step of acquiring the plurality of input images is a step of acquiring a plurality of input images by simultaneously photographing a subject moving on the background surface with a plurality of photographing units while synchronizing. 8. The method according to claim 1, wherein
The input image processing method as described. 9. The method according to claim 1, wherein the step of generating the plurality of subject extracted images includes extracting only the subject based on a background image in which the subject is not shown.
9. An input image processing method according to any one of claims 1 to 8. 10. The step of generating a difference image includes generating a difference image by taking an exclusive OR (EXOR: EXOR-OR) for each corresponding pixel of the generated plurality of subject extraction images. Is the step of
10. The input image processing method according to claim 1, wherein: 11. The step of generating an occlusion image is a step of generating an occlusion image by taking a logical product (AND) of each of the corresponding subject extracted image and the difference image for each corresponding pixel. 11. The input image processing method according to claim 1, wherein: 12. The image according to claim 1, further comprising, after the step of measuring the position of the subject, recognizing a motion of the subject from the measured position of the subject. input method. 13. An input image processing apparatus for processing an image of a subject moving on a background surface, wherein a plurality of photographing means for simultaneously photographing a subject moving on the background surface to obtain a plurality of input images. Extraction image generation means for extracting only the subject from the plurality of input images obtained by the imaging means to generate a plurality of subject extraction images; and the plurality of subjects generated by the extraction image generation means A difference image generating unit configured to generate a difference image from the extracted image; an occlusion image generating unit configured to generate an occlusion image from an arbitrary subject extracted image of the plurality of subject extracted images and the difference image; and the occlusion image generating unit. Distance measuring means for measuring a position of the subject with respect to the background surface based on the generated occlusion image. Input image processing apparatus characterized by. 14. The plurality of photographing means have a known relative positional relationship, and their optical axis directions are parallel to each other, and their respective imaging planes are set on a plane parallel to the background plane. 14. The input image processing device according to claim 13, wherein: 15. An input image processing apparatus for processing an image of a subject moving on a background surface, wherein a plurality of photographing means for simultaneously photographing a subject moving on the background surface to obtain a plurality of input images. A normalized image generating means for generating respective normalized images from the plurality of input images obtained by the photographing means; and only the subject from the respective normalized images generated by the normalized image generating means Extracted image generating means for extracting a plurality of subject extracted images, a difference image generating means for generating a difference image from the plurality of subject extracted images generated by the extracted image generating means, and the plurality of subjects Occlusion image generation means for generating an occlusion image from an arbitrary subject extraction image of the extraction images and the difference image; and the occlusion image generation Input image processing apparatus characterized by comprising: a distance measuring means for measuring a position relative to the background surface of the object on the basis of the occlusion image generated by the stage, a. 16. The input image according to claim 15, wherein said plurality of photographing means are installed such that their relative positional relationship is known, their optical axis directions are known, and they are congested. Processing equipment. 17. The distance measuring means, wherein the occlusion image generating means generates two or more occlusion images from two or more arbitrary subject extracted images of the plurality of subject extracted images and the difference image. Is a configuration in which the position of the subject with respect to the background surface is measured for each of the two or more occlusion images, and the position of the subject is determined based on an average value of the positions measured for each of the two or more occlusion images. 17. The input image processing device according to claim 13, wherein: 18. The apparatus according to claim 1, wherein said distance measuring means corrects the measured position by a predetermined correction value.
18. The input image processing device according to any one of items 3 to 17. 19. The distance measuring means according to claim 1, wherein said position of said subject is a value Zh indicating a distance from said background surface to said subject.
', The predetermined correction value is a width value Z0 of the subject in a direction perpendicular to the background surface, and a value indicating the distance to the subject when the measured position value is Zh. 19. The input image processing apparatus according to claim 18, wherein Zh 'is obtained by Zh' = Zh-Z0. 20. The apparatus according to claim 13, wherein said plurality of photographing means are configured to simultaneously photograph a subject moving on said background plane while synchronizing to obtain a plurality of input images. 20. The input image processing device according to 19. 21. The input image processing apparatus according to claim 13, wherein said extracted image generating means extracts only said subject based on a background image in which said subject is not shown. 22. The difference image generating means generates a difference image by taking an exclusive OR (EXOR: EXOR-OR) for each corresponding pixel of the plurality of generated subject extracted images. 2. The method according to claim 1, wherein
22. The input image processing device according to any one of Items 3 to 21. 23. The occlusion image generating means generates an occlusion image by taking a logical product (AND) of each of the corresponding extracted pixels of the subject and the difference image for each corresponding pixel. The input image processing device according to claim 13. 24. The image input apparatus according to claim 13, further comprising means for recognizing a motion of said subject from a position of said subject measured by said distance measuring means. 25. A method of simultaneously photographing a subject moving on a background with a plurality of photographing units to obtain a plurality of input images, extracting only the subject from each input image, and extracting a plurality of subjects. Generating an image; generating a difference image from the generated plurality of subject extraction images; and generating an occlusion image from an arbitrary subject extraction image of the plurality of subject extraction images and the difference image. Measuring a position of the subject with respect to the background based on the occlusion image; and a computer-readable recording medium recording an input image processing program for causing a computer to execute an input image processing method. 26. The step of obtaining a plurality of input images, wherein the relative positional relationship between the input images is known, the optical axis directions are parallel to each other, and the respective imaging planes are set on a plane parallel to the background plane 26. The recording medium according to claim 25, further comprising a step of acquiring a plurality of input images photographed by said plurality of photographing units. 27. A step of simultaneously photographing a subject moving on a background with a plurality of photographing units to obtain a plurality of input images; and generating respective normalized images from the plurality of input images. Extracting only the subject from each of the normalized images to generate a plurality of subject extracted images; generating a difference image from the generated plurality of subject extracted images; and the plurality of subject extracted images Generating an occlusion image from an arbitrary subject extracted image and the difference image, and measuring a position of the subject with respect to the background surface based on the occlusion image. A computer-readable recording medium that stores an input image processing program to be executed. 28. The step of acquiring the plurality of input images, wherein the relative positional relationship between the plurality of input images is known, the directions of the optical axes are known, and the plurality of input images are photographed by the plurality of photographing units installed so as to be congested. 28. The recording medium according to claim 27, further comprising a step of acquiring a plurality of input images. 29. The step of generating the occlusion image is a step of generating two or more occlusion images from two or more arbitrary subject extracted images of the plurality of subject extracted images and the difference image. Measuring the position of the subject with respect to the background surface for each of the two or more occlusion images, and determining the position of the subject based on an average value of the positions measured for each of the two or more occlusion images. 29. The recording medium according to claim 25, which is a step of determining. 30. The recording medium according to claim 25, wherein the step of measuring the position of the subject corrects the measured position by a predetermined correction value. 31. In the step of measuring the position of the subject, the position of the subject is a value Zh ′ indicating a distance from the background surface to the subject, and the predetermined correction value is the background of the subject. When there is a width value Z0 in a direction perpendicular to the surface, and when the value of the measured position is Zh, the value Zh indicating the distance to the subject is obtained by Zh ′ = Zh−Z0. The recording medium according to claim 30. 32. The step of acquiring a plurality of input images is a step of acquiring a plurality of input images by simultaneously photographing a subject moving on the background surface with a plurality of photographing units while synchronizing. 32. The recording medium according to claim 25, wherein: 33. The recording medium according to claim 25, wherein the step of generating the plurality of subject extracted images extracts only the subject based on a background image not including the subject. 34. The step of generating a differential image includes generating a differential image by taking an exclusive OR (EXOR: EXOR-OR) for each corresponding pixel of the generated plurality of subject extraction images. Is the step of
The recording medium according to any one of claims 25 to 33, wherein: 35. The step of generating an occlusion image is a step of generating an occlusion image by taking a logical product (AND) of each of the corresponding subject extracted image and the difference image for each corresponding pixel. 35. The recording medium according to claim 25, wherein: 36. The recording method according to claim 25, further comprising, after the step of measuring the position of the subject, recognizing an operation of the subject from the measured position of the subject. Medium.