WO2005096129A1

WO2005096129A1 - Method and device for detecting directed position of image pickup device and program for detecting directed position of image pickup device

Info

Publication number: WO2005096129A1
Application number: PCT/JP2005/006235
Authority: WO
Inventors: Syuhei Jyonishi
Original assignee: Tamura Corporation
Priority date: 2004-03-31
Filing date: 2005-03-31
Publication date: 2005-10-13
Also published as: JPWO2005096129A1

Abstract

To obtain position information of marks at four corners of a display even under a condition where not all of the four marks are photographed, by permitting mark information to include information for estimating positions of other marks. At the four corners of a display (1), marks (0-3) including index segments indicating arrangement directions of other three marks are provided, and the mark images are picked up by an image pickup device (2). A position detecting part (5) is provided with a mark detecting part (51) for extracting a mark from photographed images, a mark type discriminating part (52) for discriminating at which position each photographed mark is provided on the display, an index segment selecting part (53) for discriminating a segment included in each photographed mark, a calculating part (54) for calculating a position of a mark with unpicked up image, from an intersecting point of an extension of the index segment, and a relative coordinate detecting part (55) for calculating a relative position of the display and an image pickup device, based on position information of the photographed mark on an image flat plane and that of the mark not photographed on the image flat plane.

Description

Specification

Method and apparatus for detecting pointing position of imaging device, and program for detecting pointing position of imaging device

Technical field

[0001] The present invention relates to a pointing position of an imaging device for detecting relative coordinates on the display at an intersection of an optical axis (or a pointing direction) of the imaging device and a display plane when a display is shot by the imaging device. The present invention relates to a detection method and an apparatus, particularly, using an imaging device built in a gun or a pointing device for a shooting game.

Also, the present invention relates to a method and apparatus for detecting which position on the display the optical axis of the image pickup device indicates by photographing a plurality of marks prepared on the display side.

Further, the present invention relates to a program for detecting a designated position of an imaging device for realizing the position detecting method and apparatus on a computer.

Background art

[0002] Conventionally, in a shooting game using a CRT or a liquid crystal display, an aiming position of a firing device such as a gun which is aimed at a target is detected, or a predetermined position on an image projected on a screen type display is detected. A device using an imaging device is known as a device for detecting a position on a display indicated by a pointing device when the pointing device is used to indicate a position (see Patent Documents 1 to 3). ).

[0003] This type of device captures marks provided at four corners of a display as a subject, and determines the relative positions of the captured four corners on an image plane according to the distance, angle, and the like between the display and the imaging device. Utilizing the change in position, the direction of the optical axis of the imaging device with respect to the display, that is, the aiming position of the gun and the pointing position of the pointing device on the display are detected.

Patent Document 1: JP-A-8-71252

Patent Document 2: JP-A-11-305935

Patent Document 3: Japanese Patent Application Laid-Open No. 2001-148025 Disclosure of the invention

Problems to be solved by the invention

[0004] The pointing position detecting device of this type of imaging device uses a position on the image plane of the four corner marks photographed by the imaging device to determine the relative position on the display plane indicated by the optical axis of the imaging device. Since the position is detected, accurate position detection taking into account projection distortion always requires the position information of the specified four corner marks on the display. Note that the positions of the marks do not necessarily have to be at the four corners of the display as in Patent Documents 1-3, but it is indispensable to provide them at least at four positions in terms of detection accuracy.

[0005] In such a conventional technique, when the aiming or pointing position of the imaging device is set in the peripheral area of the display, the marks provided on the opposite side are out of the shooting range, and the four marks are placed on the image plane. And the positioning of the imaging device becomes impossible. Therefore, in order to capture these four marks within the image plane range, the distance between the display and the imaging device needs to be sufficiently large to show the entire display.

[0006] However, increasing the distance between the imaging device and the display leads to an increase in the size of a game machine or the like using the image pickup device. In particular, an arcade game machine or the like that requires a small installation space is large. It was inconvenient. When a large display is used, the distance between the imaging device and the display becomes larger, and the user operates a gun or pointer with a built-in imaging device from a remote location, thereby aiming at the desired position on the display. This makes it difficult to check the displayed contents.

[0007] The present invention has been proposed to solve the above-mentioned problems of the prior art, and an object of the present invention is to include information for estimating the position of another mark in information of a mark, and Then, by inferring the position of the mark, the position information of the mark at the four corners of the display can be obtained even in the situation where the mark is not photographed and the image is taken, and based on the four position information, the imaging device is used. It is an object of the present invention to provide a position detecting method and device capable of accurately performing the specified position detection.

[0008] Another object of the present invention is to provide an image processing apparatus in which all the marks at the four corners of the display are not photographed. Position detection that enables accurate detection of the pointing position of the imaging device even when the aiming position and the pointing position are directed to the peripheral area of the display by enabling the position confirmation of the corner mark It is to provide a method and an apparatus.

Yet another object of the present invention is to reduce the distance between the imaging device and the display by enabling the pointing position of the imaging device to be detected without photographing all the marks at the four corners of the display. Accordingly, it is an object of the present invention to provide a position detecting method and apparatus capable of miniaturizing the entire apparatus and improving operability.

Means for solving the problem

[0010] The present invention relates to a method for detecting an indicated position of an imaging device using four marks, in which marks are arranged at four corners of a display, and the other three marks are arranged at each mark. A linear index line segment indicating the direction is provided, and at the time of detection of the pointing position of the imaging device, if at least four marks are not captured by the imaging device, at least two of the captured marks are displayed at any position on the display. Processing for detecting the direction in which a mark that has not been captured exists, and determining the index line segment included in each captured mark, and determining the index line segment. A process is performed to extend the direction of the mark where the image is captured, obtain the intersection of the extended lines of each mark force, and calculate the position of the mark where the image is captured at the intersection. And this operation Calculating the positions of the four marks provided on the display on the image plane in the imaging device based on the positions of the marks that have not been photographed and the positions of the plurality of photographed marks. A process of detecting a coordinate position indicated by the optical axis of the imaging device on the display plane based on position information of the four marks on the image plane.

In another aspect of the present invention, the above method can be considered as a pointing position detection device for an imaging device or a program for detecting a pointing position of an imaging device.

According to such an embodiment, each mark provided at each of the four corners of the display includes information indicating the direction of another mark, that is, an index line segment, so that all the marks can be displayed on the image plane of the imaging apparatus. Even if is not photographed, if at least two marks have been photographed, the index line segment included in the photographed mark is detected, and this By extending the line segment, the position of the unphotographed mark on the image plane can be calculated. As a result, based on the position information on the image plane of the photographed mark and the unphotographed mark calculated by the calculation, the light of the imaging device on the display plane is displayed in the same manner as when all the marks are photographed. It is possible to detect the coordinate position indicated by the axis.

[0013] Another embodiment of the present invention relates to a method for detecting a pointing position of an imaging device using two marks, and when a display is photographed by the imaging device, the light of the imaging device on the display is displayed. In a method for detecting relative coordinates of an intersection between an axis and a display plane, marks are arranged at two of four corners of a display, and each mark is provided with a mark. A linear index line segment indicating the arrangement direction is provided, and when the pointing position of the imaging device is detected, the positions of the two marks captured on the image plane captured by the imaging device and the two marks are determined. A process of determining an index line segment included in the display, and extending the index line segment included in the two marks in the direction of a corner of the display provided with a mark. In both cases, the intersection of the extended line segment is calculated, and this intersection is provided with a mark to determine the position of the corner of the display on the image plane. A process of calculating the positions of the four corners of the display on the image plane based on the positions of the two marks photographed, and a process of calculating the positions of the four corners of the display on the image plane. Further, the method includes a process of detecting a coordinate position indicated by the optical axis of the imaging device on the display plane.

[0014] In another aspect of the present invention, the above method can be regarded as a pointing position detection device of an imaging device or a program for detecting a pointing position of an imaging device.

According to such an embodiment, two marks are provided on the display side, and each mark includes information indicating the direction of the corner where the display mark is not provided, that is, the index line segment. Then, by detecting the index line segment included in the two marks photographed on the image plane of the imaging device, and extending the index line segment, the image of the corner where the display mark is not provided is obtained. The position on the plane can be calculated. As a result, on the image plane of the corner of the display calculated by the calculated mark and the calculation Based on the position information, it is possible to detect the coordinate position indicated by the optical axis of the imaging device on the display plane in the same manner as when all the marks provided at the four corners of the display have been photographed.

The invention's effect

According to the present invention, when four marks are set at the four corners of the display, if any two of the four marks provided with three index lines can be imaged, the information of the index lines is obtained. In this case, the position of the mark can be predicted, and the operation range of the imaging device can be expanded as compared with the case where four marks must be taken.

[0017] When the marks are provided in at least two places among the four corners of the display, at least two marks provided with the index line segments are imaged, and the index line segments included in these two marks are captured. The position of other corners where no display mark exists can be predicted, and the coordinates indicated by the optical axis of the imaging device on the display plane are determined based on the position of these marks and the predicted position of the display corner. The position can be detected, and the number of marks to be set can be reduced as compared with the case where four marks are provided.

Brief Description of Drawings

FIG. 1 is a block diagram showing a configuration of Embodiment 1 of the present invention.

FIG. 2 is a flowchart showing the operation of the first embodiment of the present invention.

FIG. 3 is an explanatory diagram showing a principle of estimating a mark position according to the present invention.

FIG. 4 is a front view showing an example of a mark used in the present invention.

FIG. 5 is an enlarged front view of the mark in FIG.

FIG. 6 is a front view showing another example of a mark used in the present invention.

FIG. 7 is a front view showing the estimated position of the mark when the mark in FIG. 4 is used.

FIG. 8 is a front view showing deformation of a captured image due to projective distortion.

FIG. 9 is a front view showing deformation of a mark due to projective distortion.

FIG. 10 is a front view showing still another example of a mark used in the present invention.

FIG. 11 is a front view when the marks in FIG. 10 are arranged at the four corners of the display.

FIG. 12 is a front view showing a method for determining a mark from a captured image according to a size in the first embodiment. FIG. 13 is a diagram for explaining a direction category used in performing contour tracing in determining the mark in FIG. 4;

FIG. 14 is a view for explaining a method of detecting a triangle in determining the mark in FIG. 4.

[FIG. 15] A diagram showing a method for detecting one remaining unimaged mark when three marks are detected.

FIG. 16 is a front view of a display showing a combination of two marks detected when the marks are detected.

[FIG. 17] A diagram showing a method for detecting two remaining unimaged marks when two marks are detected.

FIG. 18 is a diagram showing a method of identifying a vertex of each mark when two marks are detected.

FIG. 19 is a diagram showing a method of selecting an index line segment used for estimating the position of an unimaged mark when two marks are detected.

FIG. 20 is a diagram showing a combination of picked-up marks when identifying four marks consisting of a picked-up mark and a non-shotted mark.

FIG. 21 is a view for explaining an effect when two marks provided at the lower part of the display are imaged in the present invention.

FIG. 22 is a view for explaining an effect when two marks provided on the left side of the display are imaged in the present invention.

FIG. 23 is a diagram showing a method of extracting the mark of FIG. 10 from a captured image when the mark of FIG. 10 is used.

FIG. 24 is a diagram showing another method of extracting the mark of FIG. 10 from a captured image when the mark of FIG. 10 is used.

Explanation of symbols

1… Display

2 ... Imaging device

3- AZD change

4 Frame memory

5… Position detector 6 ... Display control unit

50… Finoleta processing 咅

51Mark detector

52Mark type detection section

53Index line segment selection section

54 Not imaged mark position calculation unit

55Relative coordinate detector

BEST MODE FOR CARRYING OUT THE INVENTION

[0020] As in the present invention, when the relative positional relationship between the display and the image plane photographed by the imaging device is grasped as the subject using the projective transformation, the object and the image plane are displayed. It is known that the linearity in is a phase invariant, and the straight line is transformed into a straight line in the projective transformation. Therefore, if the intersection of the two straight lines A and B is C, the straight lines of the straight lines A and B after the projective transformation are A 'and B', and the position of the intersection C after the projective transformation is C ', then C, Is the intersection of A 'and B'. Now, if line segment information that includes the position of the mark on the opposite side of the intersection point on the extension of the line segment is included in each mark, if two marks on the subject (display) are imaged, the other The two mark positions can be inferred as intersections on the extension of the line segment.

[0021] The present invention utilizes these features,

(1) Expansion of the operation range of the imaging device when four marks are installed.

(2) Reduce the number of installed marks.

of! , Which is possible.

[0022] In this case, since each mark provided on the display has a shape including straight line information, a recognition process corresponding to that is necessary, and a process of estimating the coordinate position of the mark is added as a process. Required. The following shows the possible shapes of marks, their recognition methods, and the method of estimating the intersection after recognition. In the following description, the straight line information added to the mark for simplicity is hereinafter referred to as “index line segment”.

The accuracy of the index line segment is determined by the number of pixels constituting the index line segment (the number of pixels of the edge). In other words, if the index line segment is composed of a small number, The position of the intersection of is uncertain. However, it is considered that the state in which the four marks cannot be imaged occurs when the object and the imaging device as the position recognition unit are somewhat close to each other.

Now, as shown in FIG. 3, marks provided at the four corners of the display are referred to as marks 0, 1, 2, and 3, respectively. At this time, by including information such that the marks 1, 2, and 3 exist on the extension line in the mark 0, the direction in which the remaining marks exist even if only the mark 0 is imaged can be determined. To do. Similarly, the configuration of the marks 1, 2, and 3 includes the information of the line segment where the mark other than the mark exists on the extension line. However, since it is necessary to discriminate the mark from other figures, it is necessary to devise the shape of the mark.

Therefore, in the present embodiment, as shown in FIG. 4, the mark has one circular portion (filled portion) inside and is surrounded by a triangular frame (mark 0, 1). , And the shape (marks 2 and 3) surrounded by a square frame with no painted part inside. In addition, in order to facilitate extraction of the mark in the image, as shown in Fig. 6, the periphery of the mark is made of a low-luminance material, and the index line segments 1 to 3 and the circular portion that constitute the mark itself are formed. It is composed of a high brightness area.

With this configuration, the mark graphic portion in the captured image has a high contrast, and can be easily distinguished from other graphics by the density change information. These can be realized by a surface light source by transillumination, a retroreflector, and the like.

[0026] The frame forming the mark is made up of three straight lines, and each straight line serves as line segment information for predicting the position of the mark as described above. In addition, this can itself be a material for discrimination from other figures. In FIG. 5, index line segments 1 to 3 are line segments indicating the directions in which the lower left mark, lower right mark, and upper right mark exist, respectively. In the information of the index line segment, it is not necessary to actually insert the shape of the line segment, as long as the information can be derived from the shape of the mark.

For example, as shown in FIG. 6A, an index line segment can be created by connecting the vertices of a triangle. in this case,

(a) A set of four points that exist within a certain range.

(b) The convex hull figure is a triangle having points inside. (c) High brightness compared to other figures.

And the like, it is possible to distinguish the mark from the other figure taken. In addition, as shown in FIG. 6B, a frame may be provided on the outside to distinguish it from other figures.

By the way, when a triangular frame-shaped mark is used as described above, three straight lines included in the mark do not pass through one point on the mark, and thus have the following problem. FIG. 7 is a diagram showing a line segment of the above-mentioned mark and an installation position of another mark. As can be seen from Fig. 7, in this mark shape, the straight line obtained by the line segment of the mark does not pass through the intersection formed by the other two straight lines, and the positional forces of the specified four points on the subject to be obtained are shifted. . Therefore, when predicting the mark position from the vertical line segment, a straight line having the same inclination as the vertical line segment and passing through the intersection of the other two straight lines must be considered.

[0029] Moreover, strictly speaking, as shown in FIG. 8, when the imaging device is arranged at a predetermined angle with respect to the surface of the display and photographed, the image obtained by the imaging device is subject to projection distortion. become. Then, in the image subjected to the projection distortion as described above, as shown in FIG. 9, the image of the mark itself is also affected by the projection distortion. Therefore, the direction of the straight line obtained by the line segment of the mark and the vertical direction of the subject are different, and a line segment passing through the intersection of the other two straight lines with the same inclination as the vertical line segment is assumed as shown in Fig. 7. Cannot accurately estimate the position of other marks. However, regardless of whether it is a gun controller in a shooting game or a pointer for presentation, there is some tolerance in the focus and pointing position. By performing the correction that translates the straight line by the position of, the positions of the four marks on the display can be approximately predicted without any practical problem.

FIG. 10 shows an example of a mark in which the problem of the right-angled triangular mark as described above has been solved, in which information of three straight line segments included in the mark intersect at one point. Things. According to the mark having such a shape, the intersection of the three straight lines becomes a specified point set on the display as shown in FIG. 11, and the error caused by the projection distortion is eliminated. The mark of this shape can be obtained by changing the relative positions of the three straight lines included in each mark, or by combining a ring-shaped mark on the outer circumference of the radial mark on one side of the display. Determine where each mark is located on the display it can.

Example 1

1. Configuration of Example

Hereinafter, an embodiment of the present invention will be specifically described with reference to the drawings.

As shown in FIG. 1, the position detection device of the present embodiment includes a display 1 having an image display surface to be imaged, and an imaging device 2 for imaging the image display surface of the display 1. As the imaging device 2, a video camera or a digital camera equipped with a CCD imaging device is used.

At the four corners of the display 1, four marks 0, 1, 2, and 3 having different shapes are provided. Figure 4 shows an example of each of the marks 0, 1, 2, and 3.Each mark has a right-angled triangular shape as a whole, and each mark has three lines that indicate the direction in which the other marks are arranged. (Each side of a right-angled triangle) is included. Each of these marks has a different shape so that the location of the mark on the display 1 can be determined. In the present embodiment, the circles 0 and 1 on the left side of the display have circular filled portions. Is provided, and marks 2 and 3 on the right side can be discriminated from the fact that there is no filled portion. In addition, marks 0 and 3 on the upper part of the display 1 have horizontal straight lines located on the lower side of the triangle, and marks 1 and 2 on the lower side have horizontal straight lines located on the upper side of the triangle.

These marks can be attached to a frame portion of the display 1 or the like and provided by means such as printing or the like, or can be displayed by displaying them as an image on the display screen of the display 1. Further, in order to facilitate the extraction of the mark by the imaging device 2, it is preferable that the periphery of the mark is formed of a low-luminance material and the mark itself is formed of a high-luminance region. With this configuration, the mark figure portion in the image captured by the imaging device 2 has high contrast, and can be easily distinguished from other figures by the density change information. These can be realized by a surface light source by transmission illumination, a retroreflector, or the like.

[0034] The imaging device 2 is connected to an AZD variant 3 that uses the image data photographed thereby as digital image data, and outputs the output of the AZD variant 3 to the frame memory 4. The frame memory 4 temporarily stores the A / D-converted digital image data for each address corresponding to each pixel on the imaging plane of the CCD imaging device of the imaging device 2. It is. The frame memory 4 analyzes the image data temporarily stored in the frame memory 4 to detect the coordinates on the display plane indicated by the optical axis of the imaging device 2 with respect to the image display surface of the display 1. The position detection unit 5 is provided.

The position detection unit 5 includes a filter processing unit 50 that eliminates a figure that is unlikely to correspond to a figure force mark that is captured by the imaging device 2 and exists on the image plane, and a filter processing unit 50 that A mark detecting unit 51 is also provided for determining the image of the mark also for the passing force of the figure. Further, a mark type discriminating unit 52 for discriminating at which position of the mark force display determined by the mark detecting unit 51 is provided, and three index line segments included in each picked-up mark An index line segment selection unit 53 is provided for selecting an index line segment that is imaged from inside and extends in the direction of the mark. Further, the index line segments in the plurality of marks selected by the index line segment selection unit 53 are imaged, and are extended in the direction in which the marks exist, and the index line segments extended from the respective marks are taken. An unphotographed mark position calculation unit 54 that calculates an intersection and calculates the position on the image plane of a mark that has not been imaged on this intersection, and the position information on the image plane of the photographed mark and the unphotographed mark on the image plane. And a relative coordinate detecting unit 55 that calculates a coordinate position indicated by the optical axis of the imaging device on the display plane.

[0036] The output side of the position detecting unit 5 is connected to a display control device 6 provided in a game machine or a display device, and the display control device 6 is connected to the imaging device calculated by the relative coordinate detecting unit 55. Position information relative to the display surface is output. In this case, the display control device 6 calculates, for example, coordinates on the display indicated by the center of the imaging device based on the input position information of the imaging device, and displays an aiming point of the shooting game on the coordinates. Or display the pointing point of the pointing device.

[0037] 2. Details of operation of each unit

Next, the operation of each section in the present embodiment having the above-described configuration will be described in more detail.

(1) Filter processing unit 50

The filter processing unit 50 sets the mark figure to have high contrast as described above. Is used to distinguish from other figures based on the density change information in the captured image, and detects the shading information of the figure to be determined, and determines a figure with a high contrast exceeding a certain threshold as a mark candidate. I do. In this case, the use of an infrared transmission filter in advance in the imaging device 2 limits the number of figures to be discriminated to some extent as compared with the image in the visible light region, thereby more effectively performing filter processing. It can also be configured to do so.

As described above, in the present embodiment, the number of graphics to be determined is limited by a sharpening process that emphasizes the contrast between a mark and another portion, a noise removal process that uses an infrared filter, or the like. These processes are also effective in increasing the efficiency of the subsequent processes and reducing the load on the arithmetic unit.

(2) Mark Detector 51

The mark detection unit 51 determines whether or not an image corresponding to each mark exists in one frame of image data captured by the imaging device, and determines a position (coordinate) on the captured image plane. Is what you get. That is, since images other than the marks are variously included in the image captured by the imaging device, an image corresponding to the mark is extracted from the images.

In the processing of the mark detection unit 51, a mark is extracted from a captured figure by sequentially executing the following determination processing.

(1) Connected components within the range of the specified rectangle.

(2) Since each mark is a figure surrounded by three straight lines, the pixels constituting the captured image are connected, and only the frame-like figure is extracted from the connected image.

(3) Three linear forces.

Hereinafter, these processes will be sequentially described.

[0040] (2-1) Judgment processing based on dimensions and number of pixels

It is determined whether or not the graphic to be determined selected by the above filtering process is included in an area within a specified rectangle (rectangular in the vertical and horizontal directions of the image plane). That is, this determination is based on the size of the figure (the area of the figure and the width and height of the rectangle circumscribing the area of the figure) which are symmetrical to the determination. Implement at least one of

(A) Judgment by Total Number of Pixels in Region

In this process, since the mark includes three straight lines and the force has a certain magnitude, the ratio of the number of pixels in a certain area is determined. In other words, a figure with a large number of pixels and almost painted over, or a figure with too few pixels, on the other hand, cannot distinguish three straight lines even if they are mark candidates. , Can be excluded from mark candidates.

(B) Judgment is performed based on the width and height of a rectangle having horizontal and vertical sides in an image plane circumscribing the area of the graphic to be determined. That is, in the example of FIG. 12, only the area within the rectangle defined by the vertical rectangle width min and the vertical rectangle width max, and the horizontal rectangle width min and the horizontal rectangle width max are symmetric. Judge as a figure. In FIG. 12, since figures A, B, C, and D satisfy the above conditions, they are judged as mark candidates, and figures E and F are figures that are smaller or larger than the area. Judge that it is not a mark.

(2-2) Line Detection by Contour Tracking

After the above-described determination processing based on the size and the number of pixels, it is determined as to how much a linear element is included in the edge portion of the frame with respect to the target graphic as follows.

In FIG. 13, reference numeral A denotes an inspection start position (current position) at which a predetermined one-point force of the symmetrical figure is also selected. With the current position A as the center, eight directions, up, down, left, right, and diagonally, are classified into four direction categories as shown in the table of FIG. In other words, clockwise around the current position A,

2, 3, 4 directions: Direction category A

4, 5, 6 directions: Direction category B

6, 7, 0 direction… Direction category C

0, 1, 2 directions… Direction category D

And When the contour of the figure to be determined is tracked from the current position A, every time a direction change occurs, the location of the previous direction code change is XI, Yl, and the location of the current direction code change is Χ2, Υ2 And By the way, the following is known as the “definition of digital straight line”.

[Theorem]

A necessary and sufficient condition for a simple arc α to be a digital line segment on a continuous image is that α has a code characteristic. The figure at this time is called a digital segment or a digital straight line.

[0045] [Properties]

The necessary condition for a simple arc to be a digital straight line is that when the simple arc is represented by a direction code, the following (1) to (3) are all satisfied. However, the direction code represents the direction from one pixel to the next pixel in 45 ° units, and is called a chain code or Freeman code.

(1) There are no more than two types of direction codes, and the forces differ by 45 degrees from each other.

(2) At least one of these codes can only occur in runs of length 1! / ,.

(3) The other chord creates a run of at most two different lengths. And the lengths of these two runs differ only by one.

These points are described in detail in Non-Patent Document 1.

Non-Patent Document 1: Tokuwaki Junichiro Digital Image Processing for Image Understanding (Π) Published by Shokodo

Here, when the above condition is not satisfied in the contour tracking, a time when the above condition is not satisfied is referred to as a vertex prediction point, and the number of the points is used to determine how much a linear component is included in the edge of the figure. I do. For example, in the image shown in Fig. 14 (A), there are three long straight lines and short line segments at each vertex. Since the condition of (3) is not satisfied, it is not counted as a digital straight line, and only three long segments are detected as straight lines in this figure.

[0048] (2-3) Triangle detection

However, if only the determinations in (1) to (3) above are made up of individual small sides, such as a circle, those sides are not counted, so the following conditions must be added. In other words, as shown in Fig. 14 (Β), a concave portion formed by combining short segments is formed, and Since the short line segment forming the concave portion does not satisfy the above conditions (1) to (3), it is not counted as a straight line, and three remaining U-shaped straight lines other than the concave portion are detected, and the edge is detected. Is recognized as a figure of 3. However, even if three straight lines are detected, such a figure including a concave portion is not a triangular mark to be detected by the present invention, and such a figure is excluded as a mark candidate. There is a need. Therefore, in the present embodiment, it is determined whether or not a force is a mark candidate by adding the following condition.

[0049] [Additional conditions]

In contour tracing, a cycle is performed only once while always taking the same direction change category. That is, during contour tracing, each time a direction change occurs, the direction of change is determined. , The direction change category A—direction change category B—direction change category C one-way change category D one-way change category A or direction change category A—direction change category D—direction change force Direction change category B—As in direction change category A, the same direction change category is cycled only once and returns to the original tracking start position. At this time, the start category is not specified, and any direction change category force tracking may be started.

In this manner, as shown in FIG. 14 (B), a figure including a concave portion includes a different direction change category in the middle of a cycle. Can be determined as not a mark.

An algorithm for performing a straight line extraction process on a figure to be determined according to the definition of the digital straight line and the additional conditions as described above is as follows.

First, in performing contour tracing of the figure to be judged, the contour tracing start point is set as the starting vertex predicted point, and the position where the above condition (definition of the digital straight line) is not satisfied is set as the ending vertex predicted point. Is determined.

(1) The distance between the predicted start vertex point and the predicted end vertex point is defined as the size of the extracted side, and the following determination ′ processing is performed. Here, the determination distance of the side size is the sum of the absolute values of the difference in the horizontal direction and the difference in the vertical direction (distance between city blocks). (a) If the size of the side is a certain predetermined size, the positions (coordinates) of the predicted start vertex point and the predicted end vertex point are stored, the number of sides is incremented, and the predicted end vertex point is calculated. An update process is performed to set the start vertex prediction point of the next side.

(b) If the size of the side is not a certain specified size, only the updating process is performed with the predicted end vertex point as the predicted start vertex point of the next side.

[0053] (2) Category change judgment

If the change cycle is not the change cycle of the additional condition, it is determined that the mark is not a mark, and the processing on the graphic as a determination target is stopped.

(3) Determination of Number of Sides

At the end of the contour tracking, the number of the incremented sides is inspected for the figure satisfying the above conditions (1) and (2). If the number of sides is 3, the figure is marked. Is determined. In this case, each side force of the figure corresponds to an index line segment to be obtained. That is, the obtained side is a straight line passing through two points, the predicted start vertex point and the predicted end vertex point. On the other hand, if the number of sides of the figure is not 3, it is determined that the figure is not a mark.

[0055] (2-4) Calculation of vertex of frame

As described above, three intersections (vertices of the frame) are calculated from the extracted three straight lines as the figure medium force determined as the mark. That is, as shown in FIG. 14 (A), the edge is counted as a side by the processing of (b) in (1) above, and a short line and a straight line are recognized (see FIG. 14 (A)). The start vertex predicted point and end vertex predicted point of the three straight lines detected in are not necessarily located at the vertices of the frame. Therefore, in the mark discrimination processing, the intersection of two straight lines of each combination is obtained from the three straight lines of the figure determined as the mark, and a calculation process is performed to set those intersections as the vertices of the triangle formed by the frame. .

[0056] (2-5) Determination of Number of Marks

Hereinafter, such processing is performed on all the captured figures, and it is determined how many marks are present on the imaging plane. In this case, when the number of figures determined to be marks is 2 to 4, the force to be processed according to the number of determined marks is 1 or less. And the designated position cannot be determined, so the processing returns to the original image input processing again, and the determination processing of the captured medium-strength mark is performed. repeat.

When the number of graphics determined to be a mark is 5 or more (more than the number of installed marks), it is possible to narrow down the mark candidate graphics by further strictly determining the mark.

. For example, the degree of certainty of the linearity of the frame forming the mark can be used as a reference, or the shape and the number of pixels of an image included in the inside of the frame can be used as a reference for narrowing down. Specifically, in the case of a mark having a circular portion for left / right discrimination inside the frame, the presence of a circular figure in the frame is used as a criterion as a mark, or the number of pixels constituting the circular portion is fixed. If the figure is within the range, the figure is determined as a mark.If the mark does not have a circular part in the frame, the mark indicates that there is no other figure or a certain number of pixels in the frame. This is used as a criterion.

(3) Mark Type Discrimination Unit 52

The mark type discriminating section 52 discriminates whether the mark is provided on the left or right side of the display. For the figure determined as a mark by the processing of the mark detecting section 51, the mark type is determined. It detects whether or not there is a painted portion inside and determines whether or not it is a mark provided on the side of the display that is shifted on the basis of the presence or absence of the filled portion. This processing can be performed by detecting that a certain number or more of pixels exist in a region surrounded by three index line segments.

(4) Index Line Segment Selection Unit 53

(4 1) Mark position prediction: Basic concept

As described above, for each mark, the position of the vertex (the coordinates of the vertex of each mark) is calculated, and after the position of each mark on the imaging plane is determined, any index component included in the captured mark is determined. According to the direction in which is extended, an image is taken, and the position of the mark is estimated.

[0060] (A) Four shot marks

In the case of a figure power determined to be a mark, it is assumed that all marks have been imaged, and the optical axis of the imaging device, which is the center of the image plane, is determined based on the coordinates of these four marks on the imaging plane. Calculates where is located in the coordinate system of the display plane. This processing is publicly disclosed as shown in Patent Document 1 and the like. It is a technology of knowledge.

(B) Three shot marks

If there are three figures determined to be marks, the position of the remaining one mark is calculated using the line segments included in these three figures. In this embodiment, if at least two marks are recognized, the position of another mark can be calculated if three marks are detected as in this case. Accuracy can be improved by finding the intersection of three line segments.

[0062] The process for finding this intersection is performed as follows. First, as shown in Fig. 15, the coordinates on the image plane are represented by X and y, the straight line included in each mark is represented by y = ax + b, and the three detected marks are marked 0, 1, and 3. Assume 2. Examining the slopes and Y intercepts of each of the three straight lines of these marks 0 to 2 reveals that two of the three straight lines have the same Y intercept with the same slope as the straight lines of the other marks. Have. In other words, since the two captured images each include a line segment passing on the same straight line, the inclination and Y intercept of the three straight lines of each mark are examined. Two of the straight lines have the same slope and Y intercept as the straight lines of the other marks. This means that if three marks are taken, for each mark, one straight line does not necessarily pass on the same line as the other marks. Then, the fourth unmarked mark (mark 4) is located on the intersection of one straight line that does not pass on the same straight line as the straight line of the other mark of each mark.

[0063] (C) Two photographed marks

When four marks are provided on the display side, the pattern in which two marks are imaged is roughly classified into the following six patterns as shown in FIG.

(1) When only the upper mark is imaged (only the upper left mark and upper right mark are imaged)

(2) When only the lower mark is imaged (only the lower left mark and lower right mark are imaged)

(3) When only the right mark is imaged (only the upper right mark and lower right mark are imaged)

(4) When only the left mark is imaged (only the upper left mark and lower left mark are imaged)

(5) When a diagonal mark is captured ... Part 1 (Only upper left mark and lower right mark are captured)

(6) When the diagonal mark is captured ... Part 2 (Only the lower left mark and upper right mark are captured) Regarding (1) to (6), in any case, for example, as shown in FIG. 17 showing the state of (4) in which two marks on the left side have been imaged, a straight line The intersection of two straight lines excluding the straight line having the same direction among the components makes it possible to predict the positions of two marks that are not imaged. In other words, as shown in the graph on the right side of Fig. 17, the neutral force of the intersection of four straight lines other than the straight line considered to be the same among the straight lines of mark 0 and mark 1 indicates the positions of the remaining two marks. Predict. In this case, first, among the intersections of the four straight lines, the intersections of the straight lines derived from the same mark represent the vertices of the mark, and are excluded because they have nothing to do with the remaining two marks.

(4-2) Prediction of Mark Position: Specific Algorithm

(4 2—1) Mark vertex identification

Next, of the four intersections caused by the combination of the two straight lines of each mark, two intersections (for example, P in Fig. 17) that occur at locations other than the original positions of mark 2 and mark 3 are eliminated. It is necessary to distinguish between a straight line passing through the estimated mark position and a straight line that does not pass. Therefore, in the present embodiment, the straight line is distinguished by identifying a vertex having a property different from that of the other vertices among the vertices of each mark.

FIG. 18 shows a method of identifying vertices. The left side of the figure shows a figure identified as a mark, and this figure has three vertices a ′, b ′, and c ′. There is. Similarly, although not shown, it is assumed that another figure determined as a mark also has three vertices d ', e', and f '. In this case, since the vertices of the respective marks are not yet identified, the signs of the vertices are marked with “′” to distinguish them from the signs of the vertices after identification.

In this state, when examining the two marks, two of the vertices of each mark are on the same straight line L passing through the two vertices of the other mark. In the example of FIG. 18, the vertices a ′, b ′, d ′, and e ′ of the two marks are located on the same straight line L, as shown on the right side in the figure. Therefore, vertices c 'and f' not located on the same straight line L are defined as vertices c and f of each mark, respectively, and the remaining two vertices are arbitrarily defined as a, b and d, e. As a result, for each mark, each vertex can be identified based on the vertices c and f having different properties.

[0068] (4—2—2) Align the directions of the vectors

After the vertices are determined for each mark as described above, they are located on the same straight line. The direction of the vector determined by the straight line of each mark, that is, the straight line ab of the mark 01 and the straight line de of the mark 02 is aligned. That is, since the vertices a and b and d and e are arbitrarily determined in each mark, the directions of the vectors indicated by the straight lines ab and de on the same straight line are not always the same. Therefore, the directions of the vectors are matched based on the following rules.

(1) If ab = k-de k> 0 (inner product: positive value), do nothing.

(2) If ab = k-dek 0 0 (inner product: negative value), exchange vertices a and b.

(Here, be sure to adjust ab and de so that they are in the same direction.)

(4 2— 3) Determination of Estimated Mark Position

Next, when the types of the two marks being imaged are different, depending on whether the upper or lower mark is imaged and the mark at the diagonal position or the mark at the diagonal position is imaged, Since the conditions for extracting the combination of straight lines are different, it is necessary to make a determination. Since there are three patterns in this imaging pattern as shown in FIG. 19, the position of the vertex of each mark is determined for each pattern.

[0070] (a) Imaging pattern 1 (when only right and left marks are imaged)

In this pattern, as described above, with the vertices a, b, c and d, e, f of each mark determined so that the directions of the two captured images are the same, the cross product: If the sign of the ab X ac code is oc and the sign of the outer product de X df is β, then α = β (same sign). Therefore, the intersection of the straight line _ca and the straight line fd of each imaged mark is the position of the first mark that is not imaged, and the intersection of the straight line cb and the straight line fe is imaged. Become.

(B) Imaging pattern 2 (only upper and lower marks are imaged)

Also in this pattern, with the vertices a, b, c and d, e, f of each mark determined so that the directions of the two captured images become the same, the outer product: ab X ac code is set to α If the sign of the outer product de X df is | 8, _α = β (same sign) as in the imaging pattern 1 described above. Then, the intersection of the straight line ca and the straight line fd of each imaged mark is the position of the first mark not imaged, and the intersection of the straight line cb and the straight line fe is the position of the second image not imaged.

(C) Imaging pattern 3 (when a diagonally located mark is imaged)

In this pattern 3, each mark was taken so that the directions of the two captured images would be the same. When the vertices a, b, c and d, e, f of are defined, the outer product: ab X ac code is α, and the outer product de X df is | 8, then α (β (other sign) Become. Then, the intersection of the straight line ca and the straight line fe of each imaged mark is the position of the first mark that is not imaged, and the intersection of the straight line cb and the straight line fd is imaged, and is the position of the second mark. .

However, in the present embodiment, when such an imaging pattern 3 is detected (half β), the imaging process is performed again as an imaging error. In other words, this imaging pattern 3 occurs when the imaging device approaches and approaches the subject, and the force also rotates around the optical axis. When aiming in a shooting game or using a pointer in a presentation device, it is unnatural and almost impossible for the imaging device to be in such a state. Therefore, in the present embodiment, when such an imaging pattern 3 is detected, the user is prompted to perform another imaging as an error.

[0074] In such an imaging pattern 3, when calculating the positions of two marks that have not been imaged without causing an error, the upper and lower marks of each of the captured marks are made clear. It is necessary to change the shape of the mark. In other words, note that it is necessary to prepare three types of marks as mark shapes. The reason is that when performing the following vertex determination processing for each mark position, there is also a force that makes it impossible to specify the positions of the four marks only with marks of two different shapes.

[0075] In addition, the determination of the combination of straight lines for calculating these intersections is based on four marks on the display side in order to increase the range that can be pointed by the imaging device rather than to reduce the number of marks on the display side. It is necessary to consider only when the mark is provided, and when reducing the number of marks to be provided on the display side (for example, when the number of marks is two), it is known where the two marks taken on the display are located. There is no need for it.

(5) Unimaged Mark Position Calculation Unit 54

(5-1) Calculation of coordinates of non-imaging mark

As described above, after the index line segment selection unit 53 determines which combination of the index line segments of the two picked-up marks should be used to calculate the position of the unimaged mark, In accordance with the combination of the index line segments selected by the non-imaging mark position calculation unit 54 of! By calculating the coordinates of the intersection of the extension lines, the positions (coordinates on the image plane) of the two unimaged marks are calculated.

(5-2) Vertex Determination Processing for Each Mark Position

As described above, after the positions (coordinates) of the two marks that have not been imaged are included in the imaged marks and determined as the intersections of the extended lines of the circle, the marks 0 to 3 is a force corresponding to any mark provided on the display, that is, four marks are identified (vertexes of a square formed by four marks). In the following description, symbols A, B, C, and D indicate the identification positions of marks 0 to 3 and the three intersections a, b, c,. It does not indicate ... Also, let A and B denote the captured marks and C and D denote the marks estimated by the intersection.

[0078] However, when the number of detected marks is three, since marks detected in pairs of the detected marks are known, the mark in which only one point is detected is estimated from the intersection of C and three straight lines. As D, the vertices of the four marks are determined by the same processing as when two marks described below are imaged.

[0079] As a premise of this vertex determination, the type of the mark is clear based on the imaged mark feature (presence or absence of an inner circular portion), and in the mark position prediction processing described above, It is clear which combination of marks has been imaged. Then, as shown in FIG. 20, marks including the inner circular portion are referred to as the left side, and marks not including the inner circular portion are referred to as the right side. Vertices are determined by dividing the detected mark into two points: a left mark, two right marks, and two different types of marks.

[0080] (1) When it is known that the detection mark is the two marks on the left side

(Guess mark: mark 2, 3)

Performs cross product of AB XAC.

If the sign of AB X AC is positive, upper left is A, lower left is B

If the sign of AB X AC is negative, upper left is B, lower left is A

Perform cross product of AC X AD

If the sign of AC XAD is positive, lower right is C, upper right is D If the sign of AC X AD is negative, the lower right is D and the upper right is C

[0081] (3) It is known that the detection mark is two points on the right side.

(Guess mark: mark 0, 1)

Perform cross product of AB XAC

If the sign of AB X AC is positive, lower right is A, upper right is B

If the sign of AB X AC is negative, B is lower right and A is upper right

Perform cross product of AC X AD

If the sign of AC XAD is positive, the upper left is C and the lower left is D

If the sign of AC XAD is negative, the upper left is D and the lower left is C

[0082] (2) / (4) It is known that one point (A) of the detection mark is on the left side and the other point (B) is on the right side.

Perform cross product of AB XAC

(a) If the sign of AB X AC is positive, lower right is B and lower left is A

Perform cross product of AC XAD

If the sign of AC XAD is positive, the upper left is D and the upper right is C

If the sign of AC XAD is negative, the upper left is C and the upper right is D

(b) If the sign of AB X AC is negative, A is in the upper left and B is in the upper right

Perform cross product of AC XAD

If the sign of AC XAD is positive, lower left is C and lower right is D

If the sign of AC XAD is negative, the lower left is D and the lower right is C

[0083] (5) / (6) It is known that one point (A) of the detection mark is on the left side and another point (B) is on the right side.

When two oblique marks are picked up in this way, it is impossible to specify each mark by the above-described method using the cross product, and the vertex cannot be detected. For this reason, as described above, in the present embodiment, in the case of a combination of marks such as the imaging pattern 3, a re-imaging process is required to prevent the occurrence of the imaging pattern (5) Z (6). ing. Of course, if it is possible to combine two types of marks that distinguish left and right, and to distinguish the upper and lower left and right marks in different shapes (to prepare a total of three types of marks), this (5) In the case of Z (6), four vertices can be identified.

(6) Relative coordinate detection unit 55 In the relative coordinate detecting unit 55, based on the coordinates of the four marks on the image plane and the coordinates of the marks provided at the four corners of the display on the display plane, the coordinates of the captured image plane are obtained. By performing perspective projection conversion processing between the system and the coordinate system of the display plane, the relative positional relationship between the two coordinate systems is calculated. When the relative positional relationship between the two coordinate systems is determined, for example, it is possible to calculate which position on the display surface of the display unit is indicated by the indicator at the center (on the optical axis) of the imaging device. By outputting the calculation result to the display control device 6, it is possible to display the point indicated by the center of the imaging device on the display.

[0085] Note that the specification of the designated position by the perspective projection conversion process after the coordinates of the four marks on the captured image plane are calculated is described in JP-A-8-71252 and JP-A-2001-148025. , And Japanese Patent Application No. 2002-300478, which is filed by the applicant of the present invention, is a known technique. Therefore, in the present invention, any method may be employed for this partial processing. .

[0086] 3. Operation of the entire device

Next, the operation of the position detection device of the present embodiment will be described with reference to the flowchart of FIG. The processing according to the present embodiment is roughly divided into (1) processing of determining a mark from pixels on the imaged image plane, and (2) determining the position of the mark when the image is captured by the determined mark. The detailed operation of each step described in the flowchart is already described in the section of the algorithm of the present embodiment, so the details are omitted, and the processing is performed in any order by the device. Will be described.

[0087] This flowchart shows image processing to be performed on a figure on an image plane captured by the imaging device. As a pre-stage, the imaging device needs to be able to transmit infrared rays in advance. Use a filter and make the structure as small as possible. By doing so, the number of figures can be limited to some extent compared to an image in the visible light region.

(1) Mark Discrimination Processing

(Step 01)

With the infrared transmission filter, the image formed on the image plane by the light in the infrared region Then, a target area is extracted by a sharpening process and a binarizing process. That is, in order to distinguish a mark from another figure, the mark itself uses a wavelength in the near-infrared region or the like, and means for suppressing the number of figures generated in advance by installing a filter in the imaging device. Take.

(Step 02)

In order to judge only the frame-shaped figure, that is, the figure including the hole inside, from the figures captured and displayed on the image plane, all the pixels extracted in step 01 are determined. Then, the connection processing is performed.

(Step 03)

It is determined whether the figure composed of the pixels subjected to the connection processing is (1) a frame-like figure, and (2) a figure having a painted part (circular part) inside. That is, in the processing of (1), it is determined that a figure other than a frame-shaped figure is not a mark, and is excluded from mark candidates. Also, in the process (2), it is determined whether or not the figure determined as a mark candidate is a force having a painted portion inside the frame. The determination of the painted portion is used to determine whether the figure determined to be a mark in the following step is on the left, right, or on the side of the display.

[0089] (Step 04)

It is checked that the mark candidate graphic is a connected component within the range of the specified rectangle. In other words, a figure to be a mark should fall within a certain range of dimensions, and a figure outside the range is determined to be not a mark.

(Step 05)

It is determined whether or not the figure which is a mark candidate in step 04 is a frame in which three linear forces are also formed!

(Step 06)

The mark detected in step 05 includes three line segments (index line segments). The end point of the line segment determined to be force is not necessarily the vertex of a triangle. Therefore, the vertex coordinates of the frame are also detected for the intersection force of the three detected straight lines.

[0090] As described above, by sequentially executing the above steps, the marker is extracted from the captured image. For example, the figure to be determined is determined as follows.

Left mark: A frame with a solid part inside and three straight lines

Right mark: A frame consisting of three straight lines with no painted part inside

In this case, the number of marks should be in the range of 0 to 4 less than the number of marks installed on the display side. However, in the mark discrimination processing of (1), when there are more target figures than the number of marks to be set, it is also possible to compare the certainty of the shape of the internal point or the linearity of the frame.

[0091] (2) Mark position judgment processing

After the figure to be a mark is determined as described above, the position of each imaged mark and the strength of each imaged mark are estimated according to the number of determined marks. The position of the mark is determined.

(Step 07)

It is determined whether or not the four marks have been detected. If the four marks have been determined, the process proceeds to the next step 13 of identifying the vertices of the four marks. On the other hand, if four marks have not been detected, the process proceeds to step 08 for determining whether or not the three marks have been detected. In this determination, by including that the same straight line is included in each mark, the same straight line is determined even if the mark is determined to be a mark candidate in the mark figure determination process of (2). Those not included can be determined as not the original mark. The determination of whether or not to include the same straight line is performed in the same manner when determining the number of marks in steps 08 and 10 described later, so that the accuracy in selecting the marks from the captured figures can be improved. Is improved.

[0092] (Step 08)

When three marks are detected, the process proceeds to the next step 09, and the position of one mark remaining from the intersection of the three straight lines is obtained. On the other hand, if three marks have not been detected, the process proceeds to the next step 10 to determine whether two marks have been detected.

(Step 09)

In this step, of the three mark index line segments, the index line segments that are not on the same straight line as the other mark index line segments are extracted for each mark force. Find the position of one remaining mark. In this case, the position of the remaining one mark may be estimated from the combination of the index line segments of the two marks that also selects the neutral force of the three marks. After the positions of the remaining one mark have been estimated, the process proceeds to the next step 13 based on the three images captured and the estimated position of the one mark, and identifies the vertices of the four marks. Perform

[0093] (Step 10)

In this step, it is determined whether there are two detected marks. If there are two marks, proceed to step 11 to determine the straight line combination.If there are not two marks, that is, if there are not more than one imaged mark, the estimation of other mark positions is performed. If it is impossible, the process returns to step 01 to perform the process for the next captured image via the infinite loop process of step 15. In the above (1) mark discriminating process, if there are figures determined to be more than the number of marks to be set in the mark discriminating process, in the mark discriminating process, instead of reducing the number of marks to four or less, When there are five or more marks, it is also possible to configure so as to return to step 01 in order to perform the determination processing for the next captured image without positioning the marks.

[0094] (Step 11)

If it is determined in step 10 that the number of the captured marks is two, in step 11, which of the index line segments included in the two marks is imaged is used. Then, whether to guess the position of the mark! /, U, and executes the selection process of the combination of the index line segments. In this process, as described above, two vertices of a mark that are not on the same straight line are selected, and the vertices of each mark are identified based on the selected vertices, which is used for intersection detection. An index line segment is selected.

(Step 12)

After it is determined in step 11 which index line segment of the two marks is to be used, the intersection of the extension lines of those index line segments is determined, and the intersection of the extension lines of the two mark segments that have not been imaged is determined. I guess.

[0095] (Step 13)

As a result of the processing in steps 07, 09, and 12 described above, the imaged mark and the inferred mark In step 13, these four marks are the forces corresponding to any of the four corners of the display, that is, the vertices of the squares forming the display screen of the display and the four marks are determined. Determine the correspondence.

[0096] (Step 14)

In this step 14, based on the coordinates of the four marks on the image plane and the coordinates of each mark provided at the four corners of the display on the display plane, V and the coordinate system of the captured image plane By performing a perspective projection conversion process with the coordinate system of the display plane, the coordinates of the optical axis of the imaging device, which is the center of the image plane, in the coordinate system of the display plane are calculated.

[0097] (Step 15)

After the optical axis of the imaging device and the coordinate position on the display plane are determined as described above, the next data that is input to the imaging device, that is, moves instantly, unless a stop command is input. By repeating the analysis of the image data input from the imaging device, the detection of the pointing position of the imaging device at each point in time is repeated. Further, as described above, even when five or more marks are detected, or when only one or less marks are detected, a new mark detection process is performed on the next input captured image data. The pointing position of the imaging device is detected.

[0098] 4. Effect of Example

As described above, according to the present embodiment, even when two marks are not formed on the image plane and the force is not photographed, the position of the remaining two marks is estimated, so that the optical axis of the imaging device on the display is displayed. It is possible to detect the relative coordinates of the point of intersection with the display plane. As a result, in the related art, the point on the display indicated by the imaging device is limited to a range where all the marks on the display are imaged, and it is impossible to indicate near the periphery of the display. According to the present embodiment, it is only necessary to capture two marks, so that it is possible to indicate a point near the periphery of the display.

[0099] For example, FIG. 21 shows a case in which the lower two marks 1, 2 of the four corner marks of the display are not imaged on the image plane W of the imaging device. Plane The center point o of w indicates a predetermined point on the display surface of the imaging device as the center point of the imaging device. According to this embodiment, the positions of the other marks 3 and 4 can be estimated based on the index line segments included in the two marks 1 and 2. The area surrounded by these four marks corresponds to the display plane V of the display, and the center point O of the image plane W is the designated point on the display plane V. Can be moved to the vicinity of the periphery.

[0100] Similarly, FIG. 22 shows that, on the image plane W of the imaging device, two marks 0, 1 on the left of the four corners of the display are forces that are not imaged. By estimating the positions of the other two marks 2 and 3 that are not imaged, the center point O of the image plane W can be moved to near the periphery of the display plane.

[0101] Further, in the related art, when the imaging device is brought close to the display, imaging of the four marks becomes impossible and the relative coordinates indicated by the imaging device cannot be detected. However, in the present invention, at least two marks are detected. Since it is sufficient that the image is taken, it is possible to bring the image pickup device closer to the display as compared with the related art. As a result, according to the present embodiment, it is possible to reduce the size of a game device including a display and an imaging device, and to realize a presentation device in which a user of the imaging device is close to a large display. Example 2

[0102] Embodiment 2 is a case where only two marks are provided on the display side, and is intended to reduce the number of marks to be installed. In this embodiment, if the two marks provided at any two of the four corners of the four corners of the display are imaged, the positions of the corners of the two strong points without the display marks can be predicted. Therefore, the number of marks to be installed can be reduced.

[0103] On the other hand, the two marks that have been set must be imaged, and the pointing range is narrower than in the first embodiment. In this embodiment, since it is a precondition that two set marks are always imaged, it is not necessary to add information indicating the direction of the set marks.

As a specific configuration, it is sufficient to assume that any two of the four marks provided at the four corners of the display in the first embodiment are provided on the display. Yes. For example, when the number of marks to be set is reduced as in the present embodiment, the number of patterns to be imaged is smaller than when two arbitrary images are taken due to the force of the four marks in the first embodiment. In the mark position prediction processing performed in the first embodiment, the combination of straight lines is not required.

Further, in the process of identifying the four marks (determining the vertices of the display area surrounded by the four marks), in the second embodiment, since the position of the marked image is clearly known, The same type of mark in the first embodiment is imaged, and only the processing when the mark is performed is performed. Therefore, in the second embodiment, in the flowchart of the first embodiment shown in FIG. 3, the processing when the mark power is detected in step 07 and the three marks in step 08 and 09 are detected. In this case, no processing is required.

Example 3

Each of the above embodiments uses a triangular frame as a mark. Force The shape of the mark of the present invention is not limited thereto. As shown in FIG. 10 described above, it is also possible to use a mark having a combined force of three index lines intersecting at one point. In such a case, a method of determining a frame by contour tracing cannot be adopted in determining a mark from a captured figure. Therefore, as shown in FIG. 12, when the size of the mark shown in the first embodiment falls within a certain area, the point intersection of three straight lines is used as the feature amount of the mark, in addition to the judging method.

That is, as shown in FIG. 23, a combination of two straight lines is selected from the three straight lines to be determined, the position of the intersection is calculated, and the position of the intersection is divided by the calculated three intersections. The length of the straight lines LI, L2, and L3 is an indicator of the intersection of the three straight lines. In this case, the maximum value and the minimum value of the X and Y coordinates of the three intersections are obtained. Determine the lengths LX and LY in the X and Y directions, and judge that the one with the smallest LX + LY is closest to the intersection.

[0108] In this case, depending on the size of the area surrounded by the three intersections, it is also conceivable to determine the degree to which three straight lines intersect at one point. Even if the intersections B and C are far away from the intersection A, the area becomes close to 0 when the intersection C approaches the intersection B. Therefore, it is better to use the above-described positional relationship of each point as an index. But, By combining this area-based index with another index (such as the angle of the two detected index line segments), it is also possible to determine the degree of one-point crossing. It does not exclude indicators.

Thus, by adopting the mark configuration of the third embodiment, it is possible to characterize “consisting of three straight lines (index line segments) and intersecting at one point”. Therefore, when an image is taken by extending the index line portion of the two marks, and when the position of the other mark is estimated, the mark position is more accurate than the triangular frame-shaped mark of the first embodiment. Has some predictable advantages. In particular, the point of intersection is more accurate because it is not affected by errors due to the length of the line segments that make up the frame as shown in Fig. 8 or by projective distortion as shown in Fig. 9 (A). This makes it possible to perform accurate position prediction, and has an advantage that the accuracy of detecting the indicated position of the imaging apparatus based on the imaging mark and the prediction mark is high.

Industrial applicability

The present invention can be applied not only to a sight device of a game machine, a pointer of a display for presentation, but also to an application such as a device for detecting a gaze direction of a user in a driving simulator by mounting an imaging device on the head of the user. Applicable.

Claims

The scope of the claims

[1] In a method for detecting the relative coordinates of the intersection of the optical axis of the imaging device and the display plane on the display when the display is imaged by the imaging device,

Marks are arranged at the four corners of the display, and each mark is provided with a linear index line indicating the direction in which the other three marks are arranged.

If four marks are not captured by the imaging device when the pointing position of the imaging device is detected, it is determined which position of at least two of the captured marks is provided on the display, and the imaging is performed. The process of detecting the direction in which the unmarked mark exists, the index line segment included in each imaged mark is determined, and the index line segment is imaged and extended in the direction in which the mark exists. A process of calculating an intersection of a line segment where the force of each mark is also extended, and calculating a position of a mark that is not imaged on the intersection; and calculating the calculated position of the mark. Calculating the positions of the four marks provided on the display on the image plane in the imaging device based on the positions of the plurality of marks thus obtained;

Detecting the coordinate position indicated by the optical axis of the imaging device on the display plane based on the information on the position of the four marks on the image plane. .

[2] In the method for detecting the relative coordinates of the intersection of the optical axis of the imaging device and the display plane on the display when the display is imaged by the imaging device,

Marks are placed at two places at the four corners of the display, and each mark is provided with a mark, V, and a linear index line segment that indicates the direction in which the corners of the display are placed. At the time of position detection, a process of discriminating a position of the two captured marks on an image plane captured by the imaging device and an index line segment included in the two marks;

A mark is provided for the index line segment included in the two marks, Processing in which the intersection of the extended line is calculated, and the intersection is set as the position on the image plane of the corner of the display where no mark is provided. Calculating the positions of the four corners of the display on the image plane based on the positions of the corners of the display and the positions of the two shot marks, and the positions of the four corners of the display on the image plane. A method for detecting a pointing position of an imaging device, comprising a process of detecting a coordinate position indicated by an optical axis of the imaging device on a display plane based on information.

[3] The imaging device according to claim 1, wherein the plurality of marks have a triangular frame shape having index straight lines along the top, bottom, left, right, and diagonal directions of the display. Pointing position detection method.

[4] The instruction according to claim 3, wherein the plurality of marks are configured by a frame-shaped mark having a blank inside, and a mark having a solid portion inside the frame. Position detection method.

[5] The contour of the figure captured by the imaging device is traced! ヽ, and by detecting a cycle of a change in the direction of a straight line included in the captured figure, a convex block figure is determined from the captured figure, 4. The pointing position detection method for an imaging device according to claim 3, wherein when the convex block-shaped figure includes three linear components, it is determined that the mark is a triangle mark.

[6] A process of determining an index line segment included in each of the imaged marks and extending the index line segment in the direction in which the mark is present is performed by the process of determining the index line segment. In a state where the directions of the vector of the index line existing on the same line are aligned, the vertices of the two picked-up marks which do not exist on the same straight line are determined. The other vertices of each mark are identified on the basis of the vertices, and based on the three vertices identified by, the image is formed, the mark or mark is provided, and the corner of the display is provided. 6. The pointing position detection method for an imaging device according to claim 3, further comprising a process of determining an index line segment extending in a direction in which the image data exists. [7] The plurality of marks are radially formed by intersecting, at a single point, index straight lines along the top, bottom, left, right, and diagonal directions of the display. The pointing position detection method of the imaging device according to the above.

[8] The process of determining the index line segment included in each of the imaged marks and extending the index line segment in the direction in which the mark is present is performed by the process of determining the index line segment. In the state where the vector directions of the index line segments existing on the same line are aligned, based on the cross product of the two index line segments that do not exist on the same straight line in the two captured images, the index line on the same line is determined. Then, based on the positional relationship between the two index line segments and the three index line segments identified, an image is taken based on the positional relationship between the three index line segments, and a mark or mark is provided. 8. The method according to claim 7, further comprising determining an index line extending in a direction in which a corner of the display exists.

[9] On the image plane of the four marks provided on the display, based on the calculated position of the photographed! 撮影 mark and the positions of the plurality of photographed marks, Processing power for calculating the position The position of the two marks that have been imaged is known, and the force is also such that the two marks are not on the diagonal of the display! Claim 1 characterized by including a process of identifying a position of a mark provided on a display based on a cross product of a vector connecting the imaged mark and the intersection force predicted mark. The pointing position detection method of the imaging device according to the above.

[10] An apparatus for detecting the relative coordinates of the intersection of the optical axis of the imaging device and the display plane on the display when the display is imaged by the imaging device,

Display and

A mark provided at each of the four corners of the display, wherein each of the marks includes a linear index segment indicating a direction in which the other three marks are arranged; and at least two of the marks. An imaging device for photographing,

A mark detection unit that extracts a mark from an image captured by the imaging device; and determines at which position on the display each of the captured marks is provided. A different mark type discriminating unit;

An index line segment selecting unit that determines a straight line segment indicating the direction of another mark included in each of the captured marks;

The index line segment in the plurality of marks detected by the index line segment selection unit has not been imaged! It is extended in the direction in which the mark exists, and the intersection of the line segment extended from each mark is obtained. An unphotographed mark position calculation unit that calculates the position on the image plane of a mark that is not imaged above;

A computing unit that computes a coordinate position indicated by the optical axis of the imaging device on the display plane based on positional information of the photographed mark and the unphotographed mark on the image plane on the image plane. The pointing position detection device of the imaging device to be described.

When a display is photographed by an imaging device, a device for detecting the relative coordinates of the intersection of the optical axis of the imaging device and the display plane on the display,

Display and

Marks provided at two places at the four corners of the display, and each of the marks includes a linear index line segment indicating the arrangement direction of the other four corners where the display mark is not provided. When,

An imaging device for photographing the two marks,

A mark detection unit that extracts a mark from an image captured by the imaging device; a direction line determination unit that determines an index line segment indicating a direction of another mark included in each of the captured marks;

When a mark is provided in the plurality of marks detected by the index line segment selection section, the mark is extended in the direction of the corner of the display, and the intersection of the line extended from each mark is obtained. A display corner position calculator for calculating the position on the image plane of another corner of the display where no mark is provided on the intersection;

Based on the calculated information on the position of the corner of the display on the image plane at the intersection of the shooting mark and the index line on the image plane, calculate the coordinate position indicated by the optical axis of the imaging device on the display plane. An imaging device comprising an arithmetic unit. Pointing position detection device.

[12] In a case where the display is photographed by the imaging device, in a program for detecting the relative coordinates of the intersection of the optical axis of the imaging device and the display plane on the display,

Against the computer

A process of determining at least two of the marks arranged at the four corners of the display from the image of the display captured by the imaging device;

A process of determining at which position on the display the at least two determined marks are located, and detecting a direction in which a mark that has not been imaged exists;

The image center force is used to determine an index line segment indicating the arrangement direction of the other marks at the four corners of the display included in each of the determined marks, and the index line segment is imaged in the direction in which the mark exists. The process of calculating the intersection of the line segments where the marks are extended and the respective mark forces are also extended, the image is taken on the intersection, and the position of the mark is calculated;

The positions of the four marks provided on the display on the image plane in the imaging device are calculated based on the calculated positions of the marks that have been photographed and the positions of the plurality of marks that have been photographed. Processing,

Detecting the coordinate position indicated by the optical axis of the imaging device on the display plane based on the position information of the four marks on the image plane. Program.

[13] When a display is photographed by an imaging device, a program for detecting a relative coordinate of an intersection between an optical axis of the imaging device and a display plane on the display includes:

Against the computer

A process of determining two marks provided at two places at four corners of the display from an image of the display taken by the imaging device;

A linear index line segment indicating the arrangement direction of the corner of the display where no mark included in the two marks determined in the image is provided is determined, and the index line segment is determined. When the mark is provided, it is extended in the direction of the corner of the display, the intersection of the extended line segment is calculated, and this intersection is provided with the mark, and the mark is provided on the image plane at the corner of the display. Processing to be a position in,

A process of calculating the positions of the four corners of the display on the image plane based on the calculated positions of the corners of the display and the positions of the two captured marks; and an image plane of the four corners of the display. A program for detecting a designated position of an image pickup apparatus, which executes a process of detecting a coordinate position indicated by an optical axis of the image pickup apparatus on a display plane based on the position information above.