JP2001325069A - Device and method for detecting position - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a compact and lightweight position detecting device and a method therefor, which have an extremely high flexibility operability and by which positions to be detected on a display image are easily detected even in the partial image pickup range of a displayed image. SOLUTION: This position detecting device is provided with an image pickup means 1 for picking up the images of a plurality of marks displayed is advance at prescribed positions on the prescribed plane, by matching those marks and positions to be detected on the prescribed plane with the center of the image pickup image, a mark-extracting means 51 for extracting the marks picked up by the image pickup means, a mark-specifying means 52 for specifying the number of marks smaller than the number of the displayed marks from the plurality of marks extracted by the mark extracting means, and a position- calculating means 53 for calculating the positions to be detected on the prescribed plane, on the basis of the mark position specified by the mark specifying means.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a position detecting device for detecting a desired pointing position on a display screen, and more particularly to a position detecting device for detecting a pointing position based on an image provided with reference information displayed on the display screen. The present invention relates to a method and a position detecting device.

[0002]

2. Description of the Related Art Recently, it has been frequently performed to input coordinates directly on a display screen of a computer or to operate a cursor by projecting a computer screen onto a screen by a projector. In particular, a pointing device that can specify a pointing position on an image projected on a large screen with a laser pointer or the like and perform a command execution operation, editing, enlargement / reduction, or the like of a computer main body has attracted attention. As these conventional examples, for example, Japanese Patent Laid-Open No.
JP-A-3-176718, JP-A-4-305
687, JP-A-6-308879, JP-A-6-332612, JP-A-7-112293, JP-A-7-191797, JP-A-10-18734
0 and JP-A-11-143629.

[0003] Japanese Patent Laid-Open No. 2-3062, which is a typical conventional example,
No. 94 is composed of a screen projected by a projector, a laser pointer indicating a position to be detected on the screen, and a fixed CCD camera directed at the screen and detecting a bright point of the laser pointer. The CCD camera detects a laser luminescent spot on the screen at predetermined time intervals to detect the position of the luminescent spot on the screen.

[0004] Japanese Patent Application Laid-Open Nos. Hei 7-112293 and Hei 8-335136 disclose conventional examples of position detecting devices for extracting a marker image on a display screen by a camera and detecting the position. Japanese Unexamined Patent Publication No. 7-121293 discloses that a marker is put on a display screen for each defined frame, and only a marker image is extracted by differential image processing of an adjacent frame.
A position detection device that detects the position of the designated position based on the marker is disclosed.

Further, Japanese Patent Application Laid-Open No. 8-335136 discloses a method of judging whether or not a center position of image pickup is on a display screen by a marker image, and calculating the size and position of a screen area in the picked-up image. . JP-A-7-26
Japanese Patent No. 1913 is an apparatus for detecting a designated position on a display screen by a fixed camera, and photographing a mark displayed at a predetermined position on a display image by an image pickup means to obtain a plurality of mark positions in a photographing area. This is an information display method in which position calibration information is created from a plurality of designated positions and positions of a plurality of mark images corresponding to the designated positions, and distortion caused by an imaging position and lens aberration is reduced.

[0006]

However, a conventional position detecting apparatus using a marker image, for example, Japanese Patent Application Laid-Open No.
In JP-A-21293 and JP-A-8-335136, it is assumed that the instructor operates at a position in front of the display screen. In these methods, the operation of the position detection device is limited, and the degree of freedom of operation is poor.

Further, there is a problem that these mark image introducing methods cannot cope with a large display screen. In Japanese Patent Application Laid-Open No. Hei 7-261913, the calculation is performed by correcting the image position of the designated point using position calibration information created from the mark image position. As described above, such a method is possible in the stationary camera because the view screen for viewing the display screen is fixed. However, it is impossible in the case of a position detection device in which the pointing position and the camera are integrated.

An object of the present invention is to enable detection of a position to be detected on a display image even in a partial imaging range of a displayed image, and to provide a very small degree of operability. And a lightweight position detecting apparatus and method.

[0009]

According to a first aspect of the present invention, there is provided a position detecting device for detecting a position to be detected on a predetermined plane in a three-dimensional space. A plurality of marks previously displayed at a predetermined position on the predetermined plane, a detection position on the predetermined plane being aligned with a center of a captured image, and an imaging unit for imaging the plurality of marks; Mark extracting means for extracting the mark imaged by the mark, mark specifying means for specifying less than the displayed mark from among the plurality of marks extracted from the mark extracting means, and mark specifying means for specifying the mark. And a position and orientation calculating means for calculating the detected position on the predetermined plane based on the position of the mark.

According to a tenth aspect of the present invention, there is provided a position detecting device for detecting a detected position on a display screen on which an image is displayed, wherein the reference image is provided on the image for detecting the detected position. Display means for displaying, including, based on the reference image captured by the imaging means, and imaging means for capturing the image for detecting the detected position to the center of the captured image, including the reference image, An image processing means for detecting a detected position is provided.

Further, according to an eleventh aspect of the present invention, there is provided a position detecting device for detecting a detected position on a display screen on which an image is displayed, wherein the first screen is an image for detecting the detected position, Display means for displaying a second screen, which is a reference image, at a position corresponding to the first screen; and a detected position of the first screen being displayed on the second screen with the detected position of the first screen being aligned with the center of the image. An image pickup means for picking up an image including a reference image, and an image processing means for detecting a detected position on the first screen based on the reference image of the second screen picked up by the image pickup means. I do.

According to a seventeenth aspect of the present invention, there is provided a position detecting method for detecting a detected position on a predetermined plane placed in a three-dimensional space, wherein a plurality of positions are detected at predetermined positions on the predetermined plane. A display step of displaying a mark; an alignment step of aligning a detected position on the predetermined plane with a center of a captured image; and an imaging step of capturing an image including at least four of the displayed plurality of marks. When,
An extracting step of extracting the imaged mark;
A specifying step of specifying a mark for calculating the position and orientation of the predetermined plane from among the plurality of marks extracted in the extraction step, and a step of specifying the mark on the predetermined plane based on the mark position specified in the specifying step. And a calculating step of calculating the detected position.

[0013]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 is a conceptual configuration diagram of a position detecting device according to an embodiment. FIG. 1 is a configuration conceptual diagram of a position detection method in which an operator points a detected position Ps of a projected image on a screen plane projected by a projector from an arbitrary position and detects the detected position on the projected image. . 100 is a position detection device main body which is input means for position detection having an image pickup means, 110 is a screen, 1
20 is a personal computer and 130 is a projector. Reference numeral 111 denotes an image projected on a screen as a coordinate detection target. The position detection device main body 100 can indicate the detected position Ps on the projection image at an arbitrary operation posture position with respect to the screen plane 110. A broken line 101 is an optical axis extending vertically from the center of the imaging surface of the imaging unit 1 provided in the position detection device main body 100 to a detected position on a display image.

On the projected image, nine marks Ki are projected. These marks are reference images for detecting the detected position, and are characteristic points that characterize the rectangular shape of the coordinate detection target projection image 111. 2 and 3 are configuration block diagrams of a position detection device according to an embodiment of the present invention, and FIG. 4 is a configuration perspective view of a position detection device main body 100.

A position detecting device main body 1 according to the present embodiment.
Reference numeral 00 denotes a configuration including all the components of the position detecting device except the image processing unit 5. In the configuration block diagram of FIG. 2, reference numeral 1 denotes an imaging unit, and the imaging unit includes a lens optical system and an imaging element. In the present embodiment, CC
Although a digital still camera having a D imaging device was used,
It may be a video camera.

Further, a reference position is previously defined in the image pickup means 1 for specifying a position to be detected on a plane which is a subject. In the present embodiment, the reference position is the center of the imaging surface,
It is the origin Om of the image coordinate system (XY coordinate system). 2
A / D conversion means for converting image data picked up by the image pickup means into digital image data. Reference numeral 3 denotes a frame memory capable of temporarily storing A / D converted digital image data for each address corresponding to each pixel on the CCD imaging surface.

This frame memory has a capacity capable of storing about several tens of MB so that continuous imaging can be stored. 4
Is control means. The control means 4 is a ROM (not shown)
And stores a program for performing a perspective projection operation process, various control programs, and the like.

Reference numeral 5 denotes an image processing means. In this embodiment, the image processing is performed on the personal computer side in FIG. The image processing unit 5 extracts feature points that characterize a rectangular shape arranged anywhere on a plane in a three-dimensional space based on image data captured and captured.
1 and a position calculating means 52 for performing a detected position calculating process based on the coordinate positions of the extracted feature points.

The feature point extracting means 51 includes an extraction judging means (not shown) for judging whether or not the image data temporarily called into the frame memory has extracted feature points on a rectangular plane.
It has. By providing the extraction judging means, if the extraction judging means fails to extract a feature point, a warning sound is emitted and the operator can be instructed to take in the image again.

Further, the position calculating means 52 has a three-dimensional space (X
-YZ coordinate system), a plane attitude calculating means 521 for calculating the attitude position of a predetermined plane with respect to the imaging surface, and a coordinate calculation processing means 5 for calculating coordinates of a detected position on the predetermined plane.
22. FIG. 3 shows a plane posture calculating means 522.
FIG. 2 is a detailed block diagram of the configuration of FIG.

The plane attitude calculating means 521 comprises a vanishing point calculating means 5211, a vanishing line calculating means 5212, a vanishing feature point calculating means 5213, an image coordinate system converting means 5214, and a perspective projection converting means 5215. Reference numeral 6A denotes a light beam irradiation means, which uses an LED light emitting element or a semiconductor laser generator. The light beam irradiating means 2 may be any one that emits visible light that indicates a position to be detected, and may be a general-purpose laser pointer that can specify a position specified by an operator during a presentation or a meeting.

FIG. 5 shows the light beam irradiation means 6A shown in FIG.
1 is a first example of an optical system of a position detecting device main body using the present invention.
This is an example in which an infrared laser is used as a light beam irradiation unit. When the power is turned on, the laser light parallelized by the 61 collimating lens from the 60 light sources passes through the 62 mirror,
The light is reflected by 13 mirrors provided on the central axis of the imaging optical system, and is guided as a bright spot on a predetermined plane. The imaging optical system includes an imaging lens 12 and a CCD 11. The optical system is such that the optical axis of the pointing laser emitted from the position detecting device main body coincides with the optical axis of the imaging optical system. The mirror 13 is a half mirror that reflects an infrared laser and transmits visible light.

The pointing laser only needs to be able to confirm the position to be detected on the display screen, and is turned off during imaging. Therefore, the mirror 13 may have a mechanism that raises the mirror when capturing an image. The light beam irradiating means 6A and the imaging means 1 are provided in a predetermined positional relationship such that the detected position on a predetermined plane irradiated by the light beam irradiating means 6A matches the reference position on the imaging surface. I have.
In the figure, the optical axis of the imaging lens and the optical axis of the laser irradiation optical system are made to coincide with each other. However, since the laser light is only visually recognized, it is sufficient that the bright spot on the predetermined plane is near the detected position. , Do not necessarily have to match.

Reference numeral 7 denotes a laser irradiation button, and 8 denotes a shutter button (first operation means). Reference numerals 7 and 8 denote two-stage switches, and when the first stage is pressed, they are turned off simply by irradiating the infrared laser and pointing to the detected position. By further pressing the second stage, the shutter of the imaging means is released, and an image can be captured.

Reference numeral 9 denotes an output signal processing unit. Since the image processing means 5 is provided on the personal computer side as in the present embodiment, the output processing means sends the captured image data to an external device such as a personal computer. If a transmission means capable of transmitting an image data output signal as a wireless signal is used as the output signal processing means 9, the operability of the apparatus is greatly expanded and effective.

FIG. 6 shows a second example of the optical system of the position detecting device body provided with the collimating means 6B instead of the light irradiation beam means as a method of adjusting the reference position to the detected position. In the collimating means 6B, a cross line 74 is previously engraved at a position corresponding to the reference position on the imaging surface. The position is detected. By using a finder provided with collimating means, a light beam irradiating means, and the like at the time of image capturing, image capturing is performed in a state where the image matches a predetermined reference position on the image capturing surface. Therefore, the reference position is located at the point where the optical axis of the imaging lens cuts the imaging surface, that is, the center of the captured image. It is possible to perform the position detection operation only by turning the means in the direction of the detected position.

The configuration of the position detecting device main body according to the present embodiment is such that the position detecting device main body 100 provided with the imaging means 1 and the image processing means 5 are separated from each other, and the image processing means 5 is provided inside an external device such as a personal computer. Although the storage device and the external recording medium are provided, the position detection device main body 100 may include an imaging unit and an image processing unit.

Reference numerals 14 and 15 denote a command execution output button (second operation means) and a pop-up menu button.
The second operation means 14 corresponds to a double click of a normal mouse cursor left click button, and includes an icon on the screen,
This button is used to send a command execution signal by aligning the cursor position with an operation target object such as a figure or text.

The pop-up menu button 15 corresponds to a right-click button of a normal mouse cursor. This is an operation for displaying a pop-up menu at the position where the cursor is displayed. In the present embodiment, the shutter button (first operation means) 8 is used for a single click operation for determining a cursor position, which is a basic operation of a normal mouse cursor, and a drag operation for moving an object by aligning the cursor position with a target object. Done.

Next, the block configuration of a personal computer for inputting captured image data and command execution signals output from the position detection device body will be described. FIG. 7 is a configuration block diagram of a personal computer. Since the detailed description of the image processing means 5 has been described above, it is omitted here.

Reference numeral 120 denotes a main body of a personal computer (hereinafter referred to as a personal computer), and 121 is connected to a display. Reference numeral 130 denotes a projector, which is connected to the personal computer and projects the personal computer screen on the screen. At that time, the display 122 may not be provided.

Reference numeral 121 denotes a receiving unit that receives the captured image data and the command execution signal output from the position detecting device main body. The imaging data including the detected position signal received from the position detecting device main body is processed by the image processing means and output as the detected position signal.

Reference numeral 124 denotes cursor control means (corresponding to a mouse driver) for controlling a cursor operation. The cursor control means 124 includes a cursor position control means 125 for converting a detected position signal into a cursor position signal of a coordinate system on a personal computer screen and outputting the same, and a cursor display control means 126 for controlling the shape and color of the cursor. I have.

The cursor control means 126 is an OS
(Operating system) and application programs. Next, the basic operation of the position detecting device according to the present embodiment will be described. FIG.
5 is a flowchart illustrating a basic operation of the apparatus according to the present embodiment.

In step S100, the power of the detecting device is turned on.
I do. In step S101, a predetermined reference position on the imaging surface of the image coordinate system is adjusted to a detected position to be detected on a display plane including a plurality of feature points in a three-dimensional space. In the present embodiment, the reference position is a point where the optical axis of the imaging lens cuts off the imaging surface, that is, the center of the captured image.

In step S102, the shutter 8 of the imaging means is turned on in this state, and an image is taken.
The captured image is subjected to image data signal processing and stored in a frame memory. In step S103, geometric feature points to be extracted in advance are extracted, and the barycentric position coordinates of four marks characterizing a rectangular shape in an image coordinate system. q1, q2, q3, q4 are specified. At that time, step S
At 104, it is determined whether or not a predetermined geometric feature point has been correctly extracted. If not extracted correctly,
In step S105, a warning sound is emitted, and the operator is instructed to take in again by the imaging means. If a predetermined geometric feature point is extracted, the next step S10
Proceed to 6.

In step S106, the posture position of the predetermined plane in the three-dimensional space and the coordinate position of the detected position are calculated. The details of step S106 will be described later. In step S107, the calculated value is subjected to signal processing according to a display unit (not shown) or an external device, and output.

Next, each configuration and operation of the image processing means 5 will be described in detail. (A) Feature point extraction First, the detailed configuration and basic operation of the feature point extraction means 51 will be described. Here, a plurality of marks characterizing a rectangular shape correspond to feature points.

As shown in FIG. 2, the feature point extracting means 51 comprises a difference processing means 511, a binarization processing means 512, a feature point coordinate specifying means 513, a mark discriminating means 514, and a position detecting mark specifying means 515. . Difference processing means 51
In step 1, two types of images having different brightness or gray levels are captured, and the two images are subjected to a difference process. The binarization processing means 512 binarizes the image subjected to the difference processing over all pixels for each pixel by a predetermined threshold value and extracts a mark.

The feature point coordinate specifying means 513 calculates the area and the center of gravity of each mark and performs labeling processing for specifying the mark position coordinates. Further, the mark discriminating unit 514 compares the calculated mark with the area of the mark closest to the center position of the captured image, that is, the mark position closest to the detected position, as a reference, and judges whether or not to select the mark as a mark. I do.

The position detecting mark specifying means 515 specifies four marks characterizing a rectangular shape for a calculation to be executed by the perspective projection converting means from among the plurality of marks determined by the mark determining means. Next, the arrangement and color discrimination of mark original images having different densities (or luminance) used as reference images will be described in detail.

(A1) Reference Image and Display Method Thereof This embodiment includes at least four marks when capturing a part of a position detection target display screen on which a plurality of feature points (marks) are previously arranged. As described above, the position is detected by capturing an image and detecting the coordinates of these four marks.

First, a description will be given of a method of displaying a reference image provided with a mark array and marks arranged at predetermined positions in advance. 9A and 9B are two position detection reference images provided with a plurality of marks. FIG. 9A shows nine square marks K in a 3 × 3 grid pattern over the entire screen.
(I = 1 to 9), and a first position detection reference image (first frame reference image) in which at least four marks characterizing a rectangular shape can be captured from among the plurality of marks. It is. In the present embodiment, these nine mark arrangements include one G (green) mark at the center position K5, a B (blue) mark at the center column vertical position K2, K8,
An R (red) mark is arranged at the center row left and right positions K4 and K6, and a magenta (E) is arranged at the four corner positions K1, K3, K7 and K9. In the figure, τ is assigned to each of the four display screen areas.
1, τ2, τ3, τ4 are given. FIG. 9B is a second position detection reference image in which all nine BL (black) marks are arranged at positions corresponding to the marks of the first position detection reference image.

An image including these two reference images is displayed in a time-series manner, imaged by the position detecting device main body 100 provided with an image pickup means, and a difference process between these two picked-up images is performed. It is intended to detect only a plurality of marks. When the mark is provided on the reference image, the shape, color, number, and arrangement of the marks are appropriately selected according to the size of the position detection target image, camera characteristics, imaging conditions, and the like.

By arranging a plurality of marks at predetermined positions of the two reference images in this manner, four marks whose positional relationship between the display screen and the marks is known in advance without having to image the entire display screen. Can be identified, the coordinates of the detected position on the entire display screen can be specified. At the same time, it is possible to specify how the imaging surface of the position detection device is imaged on the display screen whose position is to be detected.

According to such a method, the cost can be reduced without using an ultra-wide-angle lens for the imaging lens used for the imaging means. Further, by devising the number and arrangement method of the marks, the present invention can be used in a wide range of fields other than this embodiment. The display method of the reference image of the present embodiment is as follows.
5 is a display screen in a state where a reference image in which a plurality of markers are arranged is displayed on a position detection target screen.

As shown in FIG. 1, the method of displaying the reference screen of this embodiment is such that the reference image K is superimposed on the target image 111 whose window display screen of the personal computer is to be detected on the screen by the projector. It is projected and displayed. Note that the window screen may be switched in synchronization with the imaging operation timing, with the position detection target screen serving as the first window screen and the reference image serving as the second window screen, without being superimposed and displayed.

(A2) Mark Extraction and Color Discrimination Next, the operation of the feature point extracting means for detecting at least four marks as position detection marks from among the plurality of marks will be described. FIG. 10 is a flowchart of the operation of the feature point extracting means from imaging four marks to identifying the color of each mark and specifying the coordinate value.

Steps S302 and S303 are operations for taking in a mark, which is a reference image projected on the screen, as a captured image. In the present embodiment, since the difference image processing method is used, 2
One frame image is captured and captured in time series, and only a position detection mark characterizing a rectangular shape is extracted.

In step S301, the first image including the first mark is captured and captured. In step S302, the second mark is captured and captured in the second image. In step S303, difference processing is performed on the two captured images, and in step S304, binarization processing is performed using a predetermined threshold value for each of the RGB colors and extracted. After the binarization processing, the color and shape of each mark are determined in step S305, and the area for calculating the center of gravity of each mark is calculated in step S306.

In step S307, after the area of each mark is calculated in step S306, mark determination processing is performed to determine whether or not the extracted mark can be used as a position detection mark. When imaging a plurality of marks, it often happens that only a part of the mark is imaged. Also,
In the imaging surface of this embodiment, the mark shape operated and detected at an arbitrary position with respect to the screen display surface is also affected by the perspective effect. As a result, the area of the mark has various values depending on the imaging position. Will be taken. In order to deal with these problems, in the discrimination processing method according to the present embodiment, a ratio C to a plurality of mark areas (S _Ki ) is determined based on the mark area (S _G ) closest to the center position of the captured image. Was determined in advance. For example, when Ci (= S _Ki / S _G ) is 50% or less, it is not regarded as a mark to be extracted. According to such a method, the determination can be made without being affected by the perspective effect accompanying the operation position.

In step S308, it is determined whether four or more marks for position detection have been extracted. next,
In step S309, the barycentric coordinate position and color of each mark are determined, and the mark position is specified. The mark position specifying process will be described later. The picked-up image data to be subjected to the mark extraction processing is, when the output signal from the imaging means is a video signal
The first and second images having different mark luminance states are two captured images in a time series in two frames (1/30 sec per frame).

In this embodiment, R (red), G (green), B
A first frame image including the (blue) and E (magenta) color marks and a second frame image including the BL (black) color mark are captured, and a difference process is performed. Thereafter, a color extraction process is performed. The video output signal is a composite video signal of a video signal and a synchronization signal, and is converted into a digital signal by A / D conversion means. The converted output signal is separated into an RGB output signal or a luminance signal / color difference signal by a matrix circuit (not shown). A color discrimination process is performed using these output signals. In this embodiment, RG
The output signal of either the luminance signal or the color difference signal may be used for the B output signal.

Next, the difference image obtained for each color is calculated based on the preset upper and lower threshold values Thu and Thb.
The value processing is performed, and a mark is detected for each color. Note that the color extraction method used in the present embodiment is a method according to the related art, so a detailed description is omitted.

Regardless of the color extraction method, when capturing an image of a projected or displayed mark, various problems such as shading, white balance, and color misregistration that occur due to a projection display device such as a projector and a use environment at the time of imaging. As a result, the captured color often differs from the predetermined color. In order to cope with such a problem, it is preferable that the setting of the upper and lower thresholds can be adjusted according to the use environment at the time of initial setting of the position detection operation.

In this embodiment, the position is determined by using four types of color marks. However, if the position can be determined, the colors need not be a plurality of colors. (A3) Mark Position Specifying Process Next, a process of specifying the coordinate values of four position detecting marks from among a plurality of detected marks will be described.

The plurality of marks provided in the reference image in advance are greatly affected by the operation position for detecting the position on the display screen, the lens performance, the photographing state, and the like, and the shape, area, number, and position of the detected marks change. . FIG. 11 and FIG. 12 are examples of typical captured images when capturing a part of the display screen including a plurality of marks from a certain operation position.

FIG. 11 shows that the position of the detected position Om is within the display area τ2 of the display screen, and the captured image is substantially within the imaging range of the entire display screen, but six marks as feature points are detected. However, two were not detected at all.
The remaining one is an example in which a mark is detected but a part of the mark is missing. Furthermore, the detected position Om is within the range of four marks characterizing the rectangular shape. In the figure, straight lines gu, gb, hr, and hl connecting between the coordinates of the center of gravity of the mark specified as the boundary between the display area and the non-display area of the display screen are shown.

FIG. 12 shows an example in which the detected position Om is outside the range of the four marks characterizing the rectangular shape.
In the present embodiment, in order to detect a detected position on a large screen of 100 inches, a predetermined G color mark, two R marks, two B marks, and four E marks, nine of four colors and nine, are determined. Display the mark image. Even if the imaging range to be imaged is a part of a large screen, it is possible to reliably detect four areas, and to determine which screen area of the entire display screen has been imaged, the number, position, and color of the detected marks. Based on this, it is necessary to accurately determine the posture of the position detection device main body with respect to the display screen.

FIG. 13 is a flowchart showing the basic processing of mark specifying means for performing position specifying processing of four marks used for position calculation. In this process,
When a plurality of marks of the same color are detected among the marks detected in the mark detection processing of FIG. 10, the position of the mark on the display screen is specified. In the detection processing of FIG. 10, four or more detection marks are always detected,
Moreover, at least one of the four colors must be different.

Since only one G color mark is previously arranged at the center of the reference image, it is detected in FIG. 28 and the coordinates are specified. However, since a plurality of detected R marks, B marks, and E marks are previously arranged in the reference image, it is necessary to specify the position of the mark on the display screen.

Step S3100 is processing for specifying the position of the detected R color mark on the display screen. The next step S3200 is processing for specifying the B-color mark as in step S3100. Step S33
00 is a process for specifying the detected E color mark. The process of specifying the R, B, and E color marks will be described later.

The next step S3400 is a process of specifying to which region on the display screen the center of the captured image belongs as the detected position. In the figure, a screen on which nine marks are displayed shows four areas τ1 to τ4 defined by the four marks. Step S3500 is processing for specifying which four of the plurality of detected marks are to be used for calculating the detected position.

FIGS. 14 and 15 show the detected R color and B color.
It is a flowchart of the specification processing of a color mark. FIG. 14 shows two R color marks displayed on the reference screen, and at least one of them is detected. It is a flowchart which specifies the position of the R color mark on the display screen where the R color mark is.

A process for specifying the position of the detected at least one R color mark on the display screen will be described. Now, it is assumed that two R color marks are detected, and (Xri, Yri) is set on the image coordinates of each mark. At this time, i =
1, 2. The two R color marks are arranged on a straight line passing through the center position of the display screen in parallel with the X axis of the screen coordinate system. Therefore, in step S323, the image coordinates (Xg, Yg) of the G color marker at the center position on the display screen
It is determined whether the R mark is a marker located on the right side or a left side based on the Xg coordinates of.

In steps S314 and S315, Xg <
If Xri, it is an R color mark where K6 is located, and Xg> Xri
If so, it is specified that the mark is an R color mark located at K4 on the display screen. FIG. 15 is a flowchart for the process of specifying the position of the B color mark on the display screen. The two B color marks are arranged on a straight line passing through the center position of the display screen in parallel with the Y axis of the screen coordinate system. Therefore, the determination in step S323 is based on the image coordinates (Xg, Y
This is performed based on the Yg coordinate of g). Other processing is shown in FIG.
The description is omitted because it is the same as the R mark.

FIG. 16 is a flow chart for specifying the position of the E-color mark among the E-color marks previously provided on the reference screen so as to be located at the four corners of the display screen. Let the coordinate data of the E-color mark obtained on the captured image be (Xei, Yei). i is the number of detected E-color marks.

Step S331 is a step of calculating a linear equation for position determination. The linear equation is a straight line parallel to the X axis of the screen coordinate system connecting the R color mark coordinate and the G color mark coordinate, and a straight line parallel to the Y axis of the screen coordinate system connecting the B color mark coordinate and the G color mark coordinate. hc 2
The position of the E-color mark is determined based on the two equations.

In step S334, it is determined whether or not the detected E mark is below the display screen, that is, below the straight line gc, based on the determination line gc. That is, by substituting the coordinate values of E color mark discriminant, Yei> proceeds to a _c Xei + b If if is _c E color marks is determined to be under the display screen side step S336, the display screen right Whether or not there is is determined by the straight line hc. In other words, Yei <d _c Xei +
When e _c (where d _c > 0), the E color mark is specified as Ebl located at the lower right side of the display screen, that is, at K7 of the display area τ2 of the reference screen. Further, the process proceeds at step _{S334 Yei <a c Xei + b} c, that is, when step S335 it is determined that the display screen upper, or left or right is determined by a linear equation hc. Yei <d _c Xei + e
_c , that is, if it is determined to be the upper left side of the display screen, the E color mark is determined to be the Eul mark located on the reference screen K1. Further, when it is determined that Yei> d _c Xei + e _c is determined that on the right side of the display screen, E color mark is identified as Eur mark located on the reference screen K3.

By the above processing, a plurality of detected R
The color, B and E color marks are sequentially processed and specified. Next, the process of determining which area on the display screen of the detected position is to be performed in step S3400 of FIG. 13 will be described.

In the present embodiment, the detected position on the display screen to be detected is detected as the center of the picked-up image. To calculate the detected position, an image having the center of the picked-up image as the origin Om (0, 0) is calculated. This is performed using a coordinate system. Therefore, the process of determining the area on the display screen where the origin is located is determined by E in FIG.
The same flowchart as that for specifying the position of the color mark on the display screen can be used. Position determination is step S3
31 is performed with a value obtained by substituting the origin (0, 0) into (X, Y) of the linear equation. For example, Ebl in step S337
In the steps until (K7) is specified, it is specified that the detected position is within the display screen area τ3 located at K7. In this way, the position of the imaging center as the detected position on the display screen is specified.

Next, a description will be given of a process for specifying four position detection marks used for calculating the detected position on the display screen. In this embodiment, if four marks, one for each of the G, R, B, and E colors, can be specified, the orientation of the display screen with respect to the imaging surface can be detected, and the detected position can be specified. That is, any one of the display screen regions τ1, τ2, τ3, and τ4 defined by the four colors of G, R, B, and E may be selected. As shown in FIG. 11, even if the detected position is within the display screen area τ3 and the four marks defining τ3 are not detected, the four marks defining the display screen area τ2 are detected. If so, the calculation of the detected position is possible.

Next, a process of determining whether or not the center of the image, which is the detected position, is outside the display screen will be described with reference to FIG. In step S351, based on the coordinates of the plurality of specified marks, a straight line formula g parallel to the X axis of the screen coordinate system of the display screen indicating a boundary line inside and outside the effective range of the display screen.
Calculate b, gu, parallel to the Y axis and hr, hl. Whether or not the detected position is within the display screen is determined based on these four equations.

Since the display screen area of the position to be detected is already specified in FIG. 16, it is determined here whether the area is within the display screen by two linear equations. In step S352, if there is a detected position in the display screen area τ2, if the discriminant equation bb ≦ 0 and er ≦ 0, it is determined that it is within τ2 within the display screen, and it is determined that it is outside the display screen.
In step S360, a warning is issued by an error display or a warning sound, and the process proceeds to a process of performing an image capturing operation again.

The detected position is the display screen area τ1, τ3, τ
4, the judgment is made based on the discriminant linear formulas gu and hr in step S354, the discriminant linear formulas gb and hl in step S356, and the discriminant linear formulas gu and hl in step S358. You.

By arranging the marks at the grid points on the display screen in this manner, the coordinates of the detected position can be specified even in a part of the captured image without capturing the entire displayed screen. . In addition, by devising the arrangement of the colors and shapes of these marks, it is possible to determine the vertical direction of the captured image captured by the position detection device main body, and to significantly expand the operation range of the position detection device main body. Further, there is an advantage that the performance load of the imaging lens can be reduced and the cost can be reduced. (B) Position calculation processing Since four position detection marks characterizing a rectangular shape are specified, a predetermined plane is determined based on the result of specifying the coordinates on the imaging surface based on the coordinate values of the four marks. The position calculation of the detected position Ps on the display image projected above can be performed.

The detected position P for these four marks
The coordinate calculation processing of s will be specifically described. FIG. 18 is a flowchart showing a specific procedure for calculating a detected position on a predetermined plane placed in a three-dimensional space.
9 is a detail of the operation of step S106 in the basic operation flowchart of FIG.

FIG. 19 shows an XYZ coordinate system in which the imaging surface of the position detecting device main body 100 is an XY coordinate system (referred to as an image coordinate system) and an axis extending vertically from the center of the image coordinate system is a Z axis. FIG. 3 shows a posture positional relationship between an image coordinate system and a X * -V coordinate system (referred to as a plane coordinate system) on a predetermined plane in a three-dimensional space having a coordinate system. Viewpoint O is the origin Om of the image coordinate system
From the focal length f. An angle ψ around the X axis, an angle γ around the Y axis, and an angle α or β around the Z axis in the XYZ coordinate system. Regarding the rotation around these angles, clockwise is defined as positive.

FIGS. 20 and 21 show a picked-up image q picked up from an arbitrary position by the operator with the image pickup direction of the image pickup means 100 provided on the position detecting device main body directed to a predetermined rectangular plane. . In the figure, the captured image has a detected position Ps, which is a coordinate position on a plane, matched with a reference position (origin Om of the imaging surface) set on the imaging surface. These four feature points q1, q2, q3, q4 correspond to Q1, Q2, Q3, Q4 in the plane coordinate system X * -Y * coordinate system of FIG.

The reference position indicating the detected position may be out of the range (rectangular shape) in which four feature points are formed. FIG. 21 shows an example in which the center of a captured image as a detected position is not included in a rectangular shape range formed by four feature points on a predetermined plane. (B1) Plane posture calculation process The plane posture calculation process, which is the first step for calculating the detected position, will be described with reference to the flowchart of FIG. 18, the block diagram of FIG. 3, and FIGS. explain.

First, in step S111, based on the coordinate positions of q1, q2, q3 and q4 already specified by the feature point specifying means of the feature point extracting means, the captured image q
Of the straight line I1. Calculate I2, I3 and I4. Next, in step S112, vanishing points T0 and S0 of the captured image data are calculated using these linear equations.
(Steps S111 and S112 correspond to the processing of the vanishing point calculation unit 5211 in the configuration block diagram of FIG. 3). When a plane having a rectangular shape is captured, a vanishing point always exists in the captured image. The vanishing point is a point where the parallel group converges. For example, a straight line q1 on the imaging surface corresponding to the straight line Q1 Q2
If q2 and the straight lines q3q4 and q1q4 corresponding to the straight line Q3Q4, and the right side Q1Q4 and q2q3 are completely parallel, the vanishing point will exist at infinity. When present at infinity, no perspective effect appears even if perspective projection is performed in that direction.

In the present embodiment, the planar shape placed in the three-dimensional space is a rectangular shape. Therefore, in the object coordinate system, 2
A set of parallels, on the X-axis side on the captured image of the image coordinate system,
There will be one vanishing point on each Y-axis side.
FIG. 20 shows the position of the vanishing point on the image data at the time of imaging at an arbitrary position. The vanishing point occurring on the X-axis side is S
0, the vanishing point occurring on the Y-axis side is T0. q1q2 and
The intersection of the extended straight line with q3q4 is the position of the vanishing point.

In step S113, after finding vanishing points S0 and T0, straight lines OmS0 and OmT0 connecting these vanishing points and the center Om of the captured image are calculated, and characteristic points qs1, qs2 and qt1 characterized by these straight lines. , Qt2. (Step S113 is a process executed by the vanishing line calculation unit 5112 in FIG. 3). Each vanishing axis S0Om, T0Om connecting the vanishing points S0, T0 and the imaging data center Om is used to determine two adjacent feature points. Intersection q that intersects the passing straight lines q1q2, q3q4, and q2q3, q1q4
t1 (Xt1, Yt1), qt2 (Xt2, Xt2), qs1 (X
s1, Ys1) and qs2 (Xs2, Ys2) are calculated.
In the description after qt1, qt2, qs1, and qs2 are calculated, S0Om, T0Om, qt1qt2, and qs1qs2 are all referred to as vanishing lines.

The vanishing straight lines qt1qt2 and qs1qs2 correspond to straight lines orthogonal to each other with respect to the detected position Ps on the plane, and serve as reference straight lines for calculating the detected position.
That is, each feature point qt in the image coordinate system (XY coordinate system)
1, qt2, qs1, and qs2 are plane coordinate systems (X * −Y *) in FIG.
Feature points T1, T2, S1, S2 on a predetermined plane in the coordinate system)
Corresponding to

In the vanishing point calculation processing in step S112, if it is determined that the vanishing point exists at infinity in the X-axis direction of the XY image coordinate system (XY coordinate system), the vanishing line is It becomes a parallel straight line. Next, step S114
Proceed to. In step S114, the coordinates are rotated by an angle β degrees about Om so that the X axis of the image coordinate system (XY coordinate system) coincides with the vanishing line OmS0 on the X axis side, and the coordinates are set to the X′-Y ′ coordinate system. Perform conversion processing. At this time, a coordinate conversion process may be performed by rotating the image coordinate system by an angle α degrees around Om so that the Y axis of the image coordinate system coincides with the vanishing line OmT0 on the Y-axis side to obtain an X ″ -Y ″ coordinate system. Either of the processes used in this embodiment is sufficient. (Step S114 is a process executed by the image coordinate system conversion unit 5214 in FIG. 3). FIG. 22 shows that the image coordinate system (XY coordinate system) is rotated by β degrees with the clockwise direction being positive, and X′-Y 'Coordinate system, X''-
FIG. 14 is a diagram for explaining image coordinate conversion processing for each of the Y ″ coordinate systems.

These image coordinate system rotation operations correspond to rotations about the Z axis in a three-dimensional space (XYZ coordinate system), and parameters of the posture position of a predetermined plane in the three-dimensional space are used. This is the operation to determine one. By matching the vanishing line qs1qs2 (or OmS0) on the X axis in this manner, the straight line Q1 on a predetermined plane placed in a three-dimensional space
Q2 and Q3Q4 have a positional relationship parallel to the X axis.

In the next step S115, based on the position coordinates in the X′-Y ′ coordinate system after the XY coordinate conversion on the obtained captured image, the characteristic points q1 of the image coordinate system X′-Y ′ system are obtained. , Q2, q3, q4 and feature points Q on a predetermined plane having a plane coordinate system with respect to the coordinate positions of qt1, qt2, qs1, qs2
1, Q2, Q3, Q4 and T1, T2, S1, S2 coordinate positions are associated. These associations are made by performing a perspective projection conversion process using a geometric technique. This perspective projection processing is performed in a three-dimensional space (X
-YZ coordinate system), which is a process of calculating a posture of a predetermined plane with respect to the imaging plane in the imaging plane, that is, an angle ψ around the Y axis and an angle γ around the X axis, which are two parameters for determining the plane posture. This is the process of calculating. The details of the perspective projection conversion process will be described later in section (b2). (Step S115
Is a process executed in the perspective projection conversion means 5123 in FIG. 3) The next step S116 is a plane coordinate system (X * -Y * coordinate system) based on the posture parameters of the predetermined plane calculated in step S115. The coordinate position of the detected position Ps is calculated. Details of the calculation of the detected position coordinates are (b3).
This will be described later in section (b4).

(B2) Perspective projection conversion processing Here, based on the result of specifying the coordinates of four feature points that characterize the rectangular shape in the image coordinate system (XY coordinate system) of the imaging surface, three-dimensional A perspective projection conversion process for calculating posture parameters (depression angle ψ, elevation angle γ) of a predetermined plane with respect to an imaging plane placed in space will be described.

First, two-dimensional perspective projection conversion will be briefly described with reference to FIG. In FIG. 23, the horizontal axis is Z
The axis and the vertical axis are the Y axis. O is an origin at the time of perspective projection conversion (hereinafter, referred to as a perspective viewpoint), 1 is an imaging surface having an XY coordinate system, and 2 is a two-dimensional plane. In the figure, the perspective point O is the origin, the horizontal axis is the Z axis, and the vertical axis is the Y axis of the imaging plane XY coordinate system. In other words, the X-axis of the imaging plane is perpendicular to the plane of the paper, and the center of the imaging plane is aligned with the Z-axis. The focal length of the lens optical system provided in this imaging means is f. The imaging plane 1 is located vertically on the Z-axis at a focal length f from the perspective point O. Actually, a lens optical system is placed at the front position of the imaging surface, and an inverted image of a two-dimensional plane is formed on the imaging surface. However, for the sake of simplicity, a CCD imaging system will be described here. It is configured to be arranged at the position of the rear focal point on the surface.

Further, it is assumed that the predetermined plane 2 having the X * -Y * plane coordinate system is in a posture inclined at an angle γ degrees with respect to the Y axis. Qi for each feature point in an image coordinate system having an XY coordinate system
(i = 1,2) is perspectively transformed by geometrical association with a corresponding feature point Qi (i = 1,2) on a predetermined plane having an X * -Y * coordinate system. The conversion formula is represented by Equation 1.

[0091]

(Equation 1)

Therefore, the feature points Qi (Y * i, Z * i) (i = 1, 2)
Are represented by the following equation (2).

[0093]

(Equation 2)

Next, the perspective projection conversion processing for calculating the position and orientation of a predetermined plane with respect to the imaging plane placed in the three-dimensional space will be specifically described. FIG. 24 shows a state in the three-dimensional space (X
FIG. 3 is a perspective view illustrating the position and orientation of a predetermined plane placed on a (YZ coordinate system). In the figure, a quarter rectangle of a predetermined plane is shown, and the position coordinates Q1 of a feature point on the predetermined plane on the imaging surface are shown.
The feature point coordinate points q1 (X1, Y1) and q2 (X2, Y2) corresponding to (X * 1, Y * 1) and Q2 (X * 2, Y * 2) are shown.

In the figure, T is the intersection of a straight line passing through the detected position Ps in the plane coordinate system (X * -Y * coordinate system) and parallel to each axis and a straight line passing through two adjacent feature points.
Position coordinates of three points, 1, T2 and S2, are shown. These illustrated feature points T1, T2, S2, and S1, not shown, correspond to feature points qt1, qt2, qs2, and qs1 characterized by vanishing lines on the captured image coordinates.

The coordinate points q3 (X3, Y3) and q4 (X4, Y4) corresponding to Q3 (X * 3, Y * 3) and Q4 (X * 4, Y * 4) are omitted. In the present embodiment, the origin Om (0,0, f) of the image coordinate system in the figure is the center position of a captured image, and this center position is a detected position on a predetermined plane to be imaged. The origin O (0,0,3) of the three-dimensional space XYZ coordinate system
0) is a perspective point at the time of the perspective projection conversion processing. f
Is the focal length.

The positional relationship between the predetermined plane and the imaging plane is expressed by X
Around the axis, an angle + ψ around the X axis and an angle + γ around the Y axis around the origin Om of the imaging surface. In each case, clockwise is defined as positive. This figure shows the result of a rotation operation around the Z axis (rotation of the XY coordinate system by + β degrees).

In this embodiment, based on the coordinate data of the feature points q1, qt1, and qs2 on the captured image (XY coordinate system), a predetermined plane (X * -Y * coordinate) corresponding to these feature points is used. The coordinate positions of the feature points Q1, T1, and S2 on the (system) were calculated by perspective projection transformation. FIG. 25 shows a predetermined plane placed in the three-dimensional space (XYZ coordinate system) shown in FIG.
FIG. 3 is a diagram orthogonally projected on a Z ′ coordinate projection plane (Y ′ = 0). The X'-Y'-Z 'coordinate system is obtained by converting the XYZ coordinate system into rotation coordinates. Only the straight line S1S2 exists on the X'-Z 'coordinate projection plane (Y' = 0) (shown by a thick line). The distance between the origin Om and the perspective point O in the X'-Z 'coordinate system is the position of the focal length f of the imaging lens. The results of expressing the coordinate position of each feature point in the plane coordinate system by the position coordinates in the X′-Y ′ coordinate system, which are associated with each feature point by perspective projection transformation, are expressed by Equations 3 and 4. You.

[0099]

(Equation 3)

[0100]

(Equation 4)

FIG. 26 is a diagram obtained by orthogonally projecting the predetermined plane shown in FIG. 24 onto the YZ'-Z 'coordinate projection plane (X' = O). Only the straight line T1T2 exists on the Y'-Z 'coordinate projection plane (X' = 0). In the figure, feature points qt1 and q1 on the captured image corresponding to feature points T1 and Q1 on a predetermined plane are shown.
Only the illustration is shown, and the feature points on the captured image corresponding to Q2, Q3, Q4 are omitted. The perspective projection conversion processing on the Y'-Z 'coordinate projection plane performs the same processing as the processing on the X'-Z' coordinate projection plane performed earlier, and qt1 and q1
Are calculated for the coordinate positions of the feature points T1, Q1 on the plane corresponding to.

Equation 5 shows the coordinate positions of T1 and Q1.

[0103]

(Equation 5)

The feature point T1 of the plane coordinate system shown in FIGS.
And the coordinate calculation process of Q1. As a result of performing perspective projection processing on two coordinate planes of the X'-Z 'coordinate projection plane and the Y'-Z' coordinate projection plane, characteristic points qt1 and characteristic points T1 and Q1 on the plane coordinates corresponding to q1 are obtained. The coordinate value is obtained.

FIG. 25 shows that T1 (X * t1, Z * t1 | x) and Q
1 (X * 1, Z * 1 | x), and from FIG. 26, T1 (Y * t1, Z * t1)
| Y) and Q1 (Y * 1, Z * 1 | y) are obtained. Three
X'Z 'projection plane (FIG. 25) orthogonally projecting a predetermined plane placed in a dimensional space (X'-Y'-Z' coordinate system), Y '
In the Z ′ projection plane (FIG. 26), the coordinate values regarding the Z ′ axis take the same value in each projection plane coordinate, and have the following relationship.

Z * 1 | x = Z * 1 | y Z * t1 | x = Z * t1 | y The following two relational expressions 6 and 7 can be obtained from the above conditional expressions.

[0107]

(Equation 6)

[0108]

(Equation 7)

As a result, the posture parameters of the predetermined plane placed in the three-dimensional space are the coordinates qi,
It was found that qti or qsi and the focal length of the imaging lens can be expressed by a simple relational expression. The relational expressions representing the position and orientation of the plane may be the following relational expressions 8 and 9 instead of the expressions 6 and 7.

The rotation direction of the angle ψ around the Y axis shown in Expressions 8 and 9 is opposite to Expressions 6 and 7.

[0111]

(Equation 8)

[0112]

(Equation 9)

The feature point q used in these relational expressions
i is expressed using one of the characteristic points qi (i = 1 to 4) obtained in the captured image and the characteristic points qti or qsi determined using the vanishing points calculated from these four characteristic points. ing. In the procedure of the present embodiment, the XY coordinate system is rotated by β degrees to convert the vanishing line qs1qs2 obtained from the image data obtained by the imaging means to the X'-Y 'coordinate system so as to coincide with the X axis. The case has been described. The other vanishing line qt1
The XY coordinate system is changed to X ''-so that qt2 coincides with the Y axis.
Even if the expression is converted to the Y ″ coordinate system, the same result is obtained although the mathematical expression is different.

Expressions in the middle are omitted, and only the result is given by Equations 10 and 1.
1 is shown.

[0115]

(Equation 10)

[0116]

[Equation 11]

In Equations 10 and 11, the position / orientation angle parameter ψ is represented by only two coordinate values q1 and qs2. In general, the coordinate position of a feature point in an image coordinate system is represented by the number of pixels, and C eq.
It goes without saying that the pixel size of the CD image sensor is required.

As described above, the angle calculation formula, which is the orientation parameter of the plane in the plane coordinate system, is expressed only by the coordinate data of the feature point calculated from the captured image and the focal length f, which is the parameter of the imaging means. It is a simple relational expression. The expression for calculating the orientation parameter of the plane is expressed by a simple expression without using a complicated coordinate transformation determinant as in the past, so that the arithmetic processing capability may be low. There are advantages, such as an economic advantage. If this is the case, the cost of the device will be reduced.

Further, when calculating the relative orientation of the plane, it is sufficient that the plane shape in the plane coordinate system is qualitatively rectangular, and the aspect ratio of the rectangle and the coordinates characterizing the rectangle are known. There is an advantage that position data relating to a plane, such as position data and data on the distance between the imaging plane and a predetermined plane, is not required.

(B3) Coordinate position calculation processing Based on the posture parameters of a predetermined plane in the three-dimensional space calculated by the perspective projection conversion processing, a plane coordinate system (X * −
The coordinates of the detected position on the predetermined plane in the Y * coordinate system are calculated by the coordinate calculation processing means 522.

The position to be detected on a predetermined plane in the plane coordinate system X * -Y * coordinate system is represented by the horizontal axis ratio mi = | S1Ps | / | S2P
s | and the vertical axis ratio ni = | T1Ps | / | T2Ps |. The equation for calculating the detected position Ps (X * i, Y * i) on the predetermined plane corresponding to the attitude parameters of Equations 6 and 7 is represented by the X-axis ratio m
And expressed in terms of the Y-axis ratio n, Any of these equations may be used.

Equation 12 is an equation using the axial ratios nx and mx.
This is a case where the XY image coordinate system is rotated by β degrees and converted into the X′-Y ′ coordinate system.

[0123]

(Equation 12)

Equation 13 is an equation using the axial ratios ny and my, and is obtained when the XY coordinate system is rotated by α degrees and converted to the X ″ -Y ″ coordinate system.

[0125]

(Equation 13)

As described above, the coordinates of the detected position can be obtained if either the plane orientation parameter ψ or γ can be calculated. (B4) Coordinate calculation of detected position Ps on all display images The position calculation described above specifies four marks (feature points) and specifies a detected position in plane coordinates defined by these four marks. I was able to.

Next, the four marks (feature points) for position detection specified on these imaging planes are placed on a predetermined plane (eg, screen plane, wall surface) on which a plurality of marks are arranged. If you can determine whether it corresponds to the mark,
The coordinates of the detected position in the screen coordinate system on the entire projection image can be easily calculated.

FIG. 27 shows the X * -Y * of the projected image of FIG.
FIG. 27B is a diagram illustrating the relationship between the coordinate system and the UV coordinate system of the original image of the personal computer in FIG. FIG. 27A shows a range in which a part of the screen image is captured by the imaging unit of the position detection device main body, that is, a captured image q. In this figure, for simplicity of description, the imaging surface of the position detection device main body is at the correct position (0 ° elevation, 0 ° depression) with respect to the screen image
The example was taken toward. Therefore, the X * -Y * coordinate system which is the coordinate system of the screen image and the coordinate system X-
The Y coordinate system is illustrated by overlapping the detected position Ps with the origin Om.

K3 (Q1), K6 (Q2), K5 (Q3), and K3 (Q1), K6 (Q2),
The detected position Ps (X *, Y *) in the screen coordinate system is determined based on the four mark coordinate values of 2 (Q4) in the UV coordinate system which is the cursor coordinate system of the original image. P
s (Ui, Vi) can be easily converted by the following equation.

[0130]

[Equation 14]

Here, the horizontal axis ratio m = | on the screen coordinate system
S1Ps | / | S2Ps |, vertical axis ratio n = | T1Ps | / | T2
Ps |. Also, the detected position is in the display screen area τ1 defined by the four marks,
The same processing can be performed in any of the other display screen regions τ2, τ3, τ4.

The method of determining which area of the entire display screen has been used by identifying the mark from the mark identification is not limited to the display image described in the present embodiment, but may include a plurality of features previously arranged at predetermined positions. It can be used for a wide range of applications in which a part on a plane having points is detected and the entire position is calculated.

[0133]

As described above, in the present embodiment, since the reference position of the image pickup surface is provided at the center position, the focal length of the image pickup lens and the four image position data characterizing the rectangular shape are obtained. With a simple relational expression using a small number of parameters, the position and orientation of a rectangular plane in a three-dimensional space and the position to be detected can be calculated. For this reason, position detection with high position detection accuracy can be easily realized by very fast arithmetic processing. Moreover, since the apparatus has a simple configuration, position detection can be easily performed regardless of the location.

Further, according to the embodiment of the present invention, the number of marks characterizing a rectangular shape can be minimized without capturing all the marks provided in the reference image on the display screen in the three-dimensional space. If a part of the display screen area defined by the above can be imaged, the detected position can be easily calculated from the image data of those characteristic points. Therefore, by determining the size of the minimum unit lattice shape that characterizes the rectangular shape according to the size of the display screen, the operation range of the position detection device of the present invention is significantly expanded and the burden on the imaging lens performance is reduced. .

Further, the position detecting device of the present embodiment
Since the position can be detected without providing a sensor on the display screen in advance, the position of the image projected on the wall surface can be detected. Further, the position detection device of the present embodiment is a new position detection method that can detect a position on a display screen from any position, and thus can be expected to expand applications in various fields.

[Brief description of the drawings]

FIG. 1 is a conceptual configuration diagram of a position detection device according to the present embodiment.

FIG. 2 is a configuration block diagram of a position detection device according to the present embodiment.

FIG. 3 is a block diagram illustrating a configuration of a plane posture calculation operation unit of the position detection device according to the present embodiment.

FIG. 4 is a configuration perspective view of a position detection device main body according to the present embodiment.

FIG. 5 is a first optical system of the position detection device according to the present embodiment.

FIG. 6 shows a second optical system of the position detection device according to the present embodiment.

FIG. 7 is a block diagram of a personal computer that receives a signal from the position detection device according to the present embodiment.

FIG. 8 is a flowchart illustrating a basic operation of the position detection device according to the present embodiment.

9A and 9B are views for explaining a position detection reference image in which a plurality of marks are arranged, and FIG. 9A is a first reference image; FIG.
(B) is the second reference image

FIG. 10 is a flowchart illustrating the operation of a feature point extracting unit of the position detection device according to the present embodiment.

FIG. 11 shows a first example of a mark array on a captured image.

FIG. 12 shows a second example of a mark array on a captured image.

FIG. 13 is a flowchart illustrating a basic operation of a mark specifying unit of the position detection device according to the present embodiment.

FIG. 14 is a flowchart illustrating a process of specifying a plurality of detected R marks by the position detection device according to the embodiment;

FIG. 15 is a flowchart illustrating a process of specifying a plurality of B marks detected by the position detecting device according to the embodiment;

FIG. 16 is a flowchart illustrating a process of specifying a plurality of detected E marks by the position detection device according to the present embodiment.

FIG. 17 is a flowchart for determining whether a detected position is outside a display screen.

FIG. 18 is a flowchart illustrating a basic operation for calculating a detected position on a predetermined plane (screen plane).

FIG. 19 is a diagram illustrating a relationship between a captured image coordinate system and a plane coordinate system.

FIG. 20 is a first example of a captured image in which characteristic points (marks) characterizing a rectangular shape are captured.

FIG. 21 is a second example of a captured image in which characteristic points (marks) characterizing a rectangular shape are captured.

FIG. 22 is a diagram illustrating a coordinate system on a captured image plane.

FIG. 23 is a view for explaining two-dimensional perspective projection.

FIG. 24 is a perspective view illustrating three-dimensional perspective projection.

FIG. 25 is an orthographic view of the screen in FIG. 24 on an X′-Z ′ coordinate plane.

FIG. 26 is an orthographic view of the screen in FIG. 24 on the YZ′-Z coordinate plane.

FIGS. 27A and 27B are diagrams for explaining the relationship between a coordinate system of a captured image on a screen image and a coordinate system of an original image of a personal computer;

[Description of Signs] 1 imaging means 2 A / D conversion means 3 frame memory 4 control means 5 image processing means 6A light irradiation means 6B collimation means 7 first button (shutter button) 8 second button (light irradiation button) 9 Output signal processing means 10 Power supply 11 Third button (left click button) 12 Fourth button (right click button) 51 Feature point extraction means 52 Perspective projection conversion means 53 Position calculation means 100 Position detection device body 110 according to the present embodiment 110 Screen plane (predetermined plane) 111 Image projected on screen surface 120 Personal computer 130 Projector

Claims (19)

[Claims] 1. A position detecting device for detecting a position to be detected on a predetermined plane placed in a three-dimensional space, comprising: a plurality of marks displayed at predetermined positions on the predetermined plane; Align the detected position on the plane with the center of the captured image,
Imaging means for imaging the plurality of marks; mark extraction means for extracting the mark imaged by the imaging means; and a mark less than the displayed mark among the plurality of marks extracted from the mark extraction means. A position detecting device comprising: a mark specifying means for specifying; and a position and orientation calculating means for calculating a detected position on the predetermined plane based on the position of the mark specified by the mark specifying means. 2. The position detecting device according to claim 1, wherein the arrangement of the plurality of marks is arranged in a grid pattern. 3. The position detecting device according to claim 1, wherein said plurality of marks use at least four kinds of colors or geometric shapes. 4. The position detecting apparatus according to claim 1, wherein the mark specification by said mark specification means is performed by specifying a center of a captured image as an origin. 5. The position detecting device according to claim 1, wherein a vertical direction and a horizontal direction with respect to a predetermined plane of an image pickup surface of the image pickup unit are determined based on the plurality of marks specified by the mark specifying unit. 6. The position detecting device according to claim 1, wherein said predetermined plane is a display means for displaying an image from an external device. 7. The position detecting device according to claim 6, wherein the external device is a projector. 8. The position detecting apparatus according to claim 1, wherein said mark extracting means performs mark extraction based on a difference process between images taken in time series by said image taking means. 9. The position detecting apparatus according to claim 1, wherein said plane position / posture calculating means is a perspective projection converting means. 10. A position detecting device for detecting a detected position on a display screen on which an image is displayed, a display means for displaying a reference image provided in advance on the image for detecting the detected position, and Image pickup means for picking up the image for detecting the detected position at the center of the picked-up image and including the reference image, and image processing for detecting the detected position based on the reference image picked up by the image pickup means A position detecting device comprising means. 11. A position detecting device for detecting a detected position on a display screen on which an image is displayed, wherein the first screen is an image for detecting the detected position and a position corresponding to the first screen. Display means for displaying a second screen, which is a reference image, and an image pickup device for picking up an image including the reference image displayed on the second screen by adjusting a detected position of the first screen to the center of the image. Means for detecting a position to be detected on the first screen based on a reference image of the second screen imaged by the imaging means. 12. The position detecting device according to claim 10, wherein the reference image has marks arranged at grid points. 13. The position detecting device according to claim 12, wherein the mark provided on the reference image is configured based on three primary colors of R, G, and B colors. 14. The position detecting device according to claim 11, wherein the second screen displays two types of screens having different luminance states when the second screen is picked up by the image pickup means. 15. The position detecting device according to claim 11, wherein the second screen is superimposed on the first screen. 16. The display switching operation of the second screen is the first screen switching operation.
12. The position detection apparatus according to claim 11, wherein the position detection apparatus is executed by a signal transmitted when a detected position on the screen of the image pickup apparatus coincides with a center position of a captured image of the imaging unit. 17. A position detecting method for detecting a detected position on a predetermined plane placed in a three-dimensional space, comprising: a display step of displaying a plurality of marks at predetermined positions on the predetermined plane in advance; An alignment step of aligning a detected position on the predetermined plane with a center of a captured image; an imaging step of capturing an image including at least four of the displayed marks; and the captured mark An extraction step of extracting a mark, an identification step of identifying a mark for calculating the position and orientation of the predetermined plane from among the plurality of marks extracted in the extraction step, and a mark position identified by the identification step Calculating a detected position on the predetermined plane based on the calculated position. 18. The position detecting method according to claim 17, wherein the predetermined plane is an image display screen. 19. The position detecting method according to claim 17, wherein said extracting step performs mark extraction by a difference process between a plurality of images taken in time series by said image pickup means.