JP5195041B2 - Pointing device, object recognition device, and program - Google Patents

Description

  The present invention relates to a pointing device or the like for pointing an object in a three-dimensional space.

  As a means for measuring the position of a predetermined object in a three-dimensional space, there is a method of photographing the object with two cameras and calculating the position coordinates of the object by triangulation using the distance between the cameras as a base line. Often used. In addition, there is a technique for detecting a light amount of a light emitting element attached to a remote controller and detecting a posture angle of the remote controller.

  For example, in Patent Document 1, a region including a measurement point indicated by a light spot formed by a laser pointer is photographed by a plurality of cameras, and this image signal is a camera identification signal generated by a camera identification signal generator. Is added to the host computer, and the host computer detects a camera identification signal from the input image and identifies the camera from which the image was taken. A three-dimensional position measurement system is disclosed that calculates a three-dimensional position of a point to be measured by performing triangulation calculation on a pair of images based on data such as a light spot position in the image and a position of a photographing source camera. .

  Further, Patent Document 2 discloses that a device direction / direction confirmation signal is transmitted from one of a plurality of infrared light emitting elements attached in different directions, and the presence / absence of a response signal from the device is confirmed. A bidirectional remote control system is disclosed in which the position and direction of an apparatus is determined and stored by sequentially performing processing for all the infrared light emitting elements.

  Further, in Patent Document 3, the light emitting elements are sequentially displaced, light emission signals are supplied to the light emitting elements at the respective displacement positions, and light signals for position detection are emitted and output, and the light receiving elements for position detection of the light receiving device receive the light. By sequentially detecting the received light reception signal and appropriately calculating the detected light reception signal, the displacement state of the reference axis displacement angle θs is detected. A remote control device that controls the movement of the pointer using the displacement state of the reference axis displacement angle θs as an instruction signal for the pointer is disclosed.

  Furthermore, Patent Document 4 discloses the attitude angle detection of the remote control transmitter that guides the reference radiated light emitted from the reference receiver directly to the light receiving surface of the four-division photodiode of the remote control transmitter without using the condenser lens. A system is disclosed.

Japanese Patent Laid-Open No. 11-23262 JP 2000-4199 A JP 2006-155346 A JP-A-8-210825

In such a pointing device, depending on how the user who operates the pointing device is used, the target may not be accurately identified, and the target indicated by the pointing device can be measured with high accuracy without depending on the user's usage. It is hoped that it can be done.
An object of the present invention is to provide a pointing device or the like that is less dependent on how a user operates to specify an object.

  In the invention according to claim 1, when a plurality of reference point sets composed of light emitting points and the light emitting points are turned on for each reference point set, and the light emitting points of one reference point set are turned on, the one reference point set Control means for turning off the light emitting points other than the point set, and the reference point set includes three or more first reference points on a plane in which the mutual positional relationship is specified, and the first reference point From the positional relationship between the image of the first reference point and the image of the second reference point of the reference point set photographed by another device. The pointing device is characterized in that an object is specified.

The invention according to claim 2 is the pointing device according to claim 1, wherein the second reference point is not arranged on the plane on which the first reference point is arranged.
The invention according to claim 3 is the pointing device according to claim 1, wherein the arrangement of the first reference point and the second reference point is different for each set of reference points.
The invention according to claim 4 is the pointing device according to claim 1, further comprising visible light irradiation means for irradiating visible light to the same location indicated by the pointing device.

According to a fifth aspect of the present invention, there is provided a photographing unit for photographing a pointing device in which a plurality of first reference points including three or more points on a plane and a plurality of second reference points including one point are arranged, and the first A storage unit that stores a mutual positional relationship of reference points, a positional relationship of the second reference point with the first reference point, and a three-dimensional position of an object, and a mutual position of the first reference point from the storage unit The relationship and the positional relationship between the second reference point and the first reference point are read out, and the image of the first reference point and the image of the second reference point photographed by the photographing unit is which set of images. Based on the positional relationship between the image of the first reference point and the image of the second reference point captured by the image capturing unit , the three-dimensional position of the object is read from the determination unit and the storage unit. The point image and the second reference point image are identified, and the identified image And a computing unit for identifying the object instructing of the pointing device by the image of the image and the second reference point of the first reference point to determine the three-dimensional position, three-axis angle and pointing direction of the pointing device This is an object recognition apparatus characterized by the above.

In the invention according to claim 6 , the determination unit is any set of images of the first reference point and the second reference point due to a difference in arrangement of the first reference point and the second reference point. It is discriminate | determining, It is a target object recognition apparatus of Claim 5 characterized by the above-mentioned.
According to a seventh aspect of the present invention, when a plurality of sets of the first reference point image and the second reference point image are taken, selection of an appropriate set for specifying the object indicated by the pointing device The object recognition apparatus according to claim 5, further comprising a selection unit that performs the operation.

According to an eighth aspect of the present invention, a positional relationship between a first reference point composed of three or more points on a plane and a second reference point composed of one point is arranged on a computer. A function for determining which set of images the image of the first reference point and the image of the second reference point read out from the memory and imaged by the photographing unit, and the three-dimensional position of the object are read out from the memory The image of the first reference point and the image of the second reference point are identified and identified from the positional relationship between the image of the first reference point and the image of the second reference point captured by the photographing unit. A function for specifying an object indicated by the pointing device by obtaining a three-dimensional position, a three-axis angle, and a pointing direction of the pointing device from the image of the first reference point and the image of the second reference point. This is a program to be realized.

According to the first aspect of the present invention, it is possible to make it less susceptible to the orientation of the pointing device and the three-dimensional position when photographing the light emitting point, as compared with the case where this configuration is not adopted.
According to invention of Claim 2, compared with the case where this structure is not employ | adopted, the three-dimensional position and triaxial angle of a pointing device can be measured with high precision.
According to the invention of claim 3, it is possible to more easily determine which set of images the image of the first reference point and the image of the second reference point are compared to the case where this configuration is not adopted. it can.
According to the fourth aspect of the present invention, the pointing device can be operated more easily than when the present configuration is not adopted.
According to the fifth aspect of the present invention, the pointing device has a simpler configuration than the case where the present configuration is not employed, and the pointing device uses the first reference point image and the second reference point image captured by the photographing unit. It is possible to obtain an object recognition apparatus that can specify an object to be indicated with high accuracy. At this time, by specifying the plane on which the first reference point is arranged and its orientation from the image of the first reference point and the image of the second reference point, the three-dimensional position and three points of the pointing device on which the plane is set are specified. Axis angle can be measured. In addition, the object indicated by the pointing device can be specified by the positional relationship between the second reference point and the first reference point.
According to the sixth aspect of the present invention, it is possible to more easily determine which set of images the image of the first reference point and the image of the second reference point are compared to the case where this configuration is not adopted. it can.
According to the seventh aspect of the present invention, it is possible to specify the target object indicated by the pointing device with higher accuracy than when the present configuration is not adopted.
According to the eighth aspect of the present invention, compared with the case where the present configuration is not adopted, the object pointed to by the pointing device is increased from the first reference point image and the second reference point image captured by the photographing unit. Functions that can be specified with accuracy can be realized by a computer. At this time, by specifying the plane on which the first reference point is arranged and its orientation from the image of the first reference point and the image of the second reference point, the three-dimensional position and three points of the pointing device on which the plane is set are specified. Axis angle can be measured. In addition, the object indicated by the pointing device can be specified by the positional relationship between the second reference point and the first reference point.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
<System configuration>
FIG. 1 is a diagram illustrating an overall configuration of an object recognition system including an object recognition device, an instruction device, and an object to which the present embodiment is applied.
The target object recognition system shown in FIG. 1 calculates a pointing device 100 that is a subject to be imaged, a three-dimensional position, a three-axis angle, and a pointing direction of the pointing device 100, and the target device that the pointing device 100 points from these information. The object recognition device 200 that identifies 300 is configured. Then, the object recognition apparatus 200 includes a camera 20 as an example of a photographing unit that photographs the pointing device 100, and a three-dimensional position, a three-axis angle, and a pointing direction of the pointing device 100 based on an image captured by the camera 20. And an arithmetic unit 30 that calculates

2A to 2B are diagrams illustrating the configuration of the pointing device 100 to which the present embodiment is applied.
The pointing device 100 has a rectangular parallelepiped shape, and includes four sets of reference points each including a first reference point and a second reference point on each of its four side surfaces.
Here, FIG. 2A is a view of the pointing device 100 as viewed from one side. As shown in FIG. 2A, on one side surface of the pointing device 100, first reference points 11a to 14a used for calculating a three-dimensional position and a three-axis angle of the pointing device 100, and predetermined information And a second reference point 15a used for adding. The first reference points 11a to 14a and the second reference point 15a have a predetermined positional relationship with each other, and are configured by light emitting points such as LEDs (Light Emitting Diodes).

  The other three side surfaces include the first reference points 11b to 14b and the second reference point 15b, the first reference points 11c to 14c and the second reference point 15c, the first reference points 11d to 14d and the second reference point. A set of reference points each including a point 15d is provided.

  In addition, the pointing device 100 includes a light emission control device 16 as a control unit that controls lighting of the LED. The light emission control device 16 turns on the four reference point sets described above for each reference point set. The order of lighting is arbitrary, and the order may be regular or irregular. However, when the light emitting point of any one reference point set is turned on, it is preferable that the light emitting points other than this one reference point set are turned off. In this way, the set of reference points photographed by the camera 20 is likely to be one set, and the calculation described in detail later becomes easier. Moreover, there may be a time zone in which no reference point set is lit. In particular, it is preferable to provide a time zone in which no light is emitted between the lighting of each reference point set, because it becomes easy to avoid the influence of afterglow.

  Here, as a reference point set, a reference point set including the first reference points 11a to 14a and the second reference point 15a is taken as an example, and the arrangement of the reference points will be described below. However, the first reference points 11b to 14b and the second reference point 15b, which are other reference point sets, the first reference points 11c to 14c and the second reference point 15c, the first reference points 11d to 14d and the second reference point 15d. The same can be said when selecting.

The first reference points 11 a to 14 a are provided on the surface of the pointing device 100. On the other hand, the second reference point 15 a is provided at the tip of a leg attached to the pointing device 100.
The quadrangular shape formed by the first reference points 11a to 14a can be, for example, a parallelogram. In the example shown in FIGS. 2A to 2B, a square shape which is one form of the parallelogram is used, and one side thereof is 51 mm (not shown).
Further, the second reference point 15a needs to satisfy a certain condition with respect to the arrangement because of the necessity of identifying the first reference points 11a to 14a and the second reference point 15a, which will be described in detail later. For example, it can be considered that the first reference points 11a to 14a and the second reference point 15a are arranged at positions farthest from the center of gravity of five points. Here, a length of 7 mm (not shown) is provided at a location 46 mm (not shown) away from the intersection of the diagonal lines of the parallelogram in the direction of the midpoint of the line segment connecting the first reference point 12 a and the first reference point 13 a. A leg was attached, and a second reference point 15a was provided at the tip of the leg.

  The length of the leg on which the second reference point 15a is provided is arbitrary, but the image of the second reference point 15a (camera 20 (see FIG. 1)) even when the pointing device 100 is tilted with respect to the imaging surface of the camera 20. The image of the point photographed by the first reference point 11a to 14a and the second reference point 15a can be held at a position farthest from the center of gravity of each point formed by the images of the first reference point 11a to 14a and the second reference point 15a. It becomes. Therefore, it may be determined according to how much the pointing device 100 can be tilted with respect to the imaging surface of the camera 20 in actual use.

  Further, here, the second reference point 15a is not arranged on the plane on which the first reference points 11a to 14a are arranged by providing a leg and arranging it at the tip thereof, but is not particularly limited thereto. You may arrange | position on the same plane as the 1st reference points 11a-14a. However, since the first reference points 11a to 14a and the second reference point 15a, which will be described in detail later, can be identified with higher accuracy, the second reference point 15a is provided with the first reference points 11a to 14a as described above. It is preferable that they are not arranged on a flat surface.

  Predetermined information is added to the second reference point 15a according to the use of the device that is the pointing device 100, and an arbitrary role is given. For example, the meaning of performing an operation such as turning on / off the power supply of the object 300 indicated by the pointing device 100 can be expressed by blinking the second reference point 15a.

  In FIG. 1 and FIGS. 2A to 2B, a pointing device 100 having a rectangular parallelepiped shape provided with first reference points 11a to 14a and a second reference point 15a is described. However, in this embodiment, since the three-dimensional position and the three-axis angle of the pointing device 100 are calculated based on the images of the first reference points 11a to 14a photographed by the camera 20, the first reference points 11a to 11a are calculated. As long as 14a and the second reference point 15a have the above positional relationship, the shape of the pointing device 100 itself is not limited. Further, in the above example, the second reference point 15a has only to be arranged at a certain height with respect to the plane including the first reference points 11a to 14a, so that the leg as shown in FIG. It is not necessary to provide and arrange. For example, if the surface of the pointing device 100 has a shape protruding in a convex shape with respect to the plane including the first reference points 11a to 14a, the second reference point 15a is arranged directly on the surface of the pointing device 100. Also good.

FIG. 3 is a diagram illustrating a configuration example of the camera 20.
The camera 20 includes an optical system 21 that converges light emitted from the first reference points 11a to 14a and the second reference point 15a, and an image sensor 22 that is an imaging unit that detects light converged by the optical system 21. Prepare.

  The optical system 21 is configured by combining a single lens or a plurality of lenses. For example, a twin lens is used in which two hemispherical lenses are used and their spherical surfaces are combined face to face. Various unnecessary aberrations are appropriately removed by the combination of lenses and the coating applied to the lens surface.

  The image sensor 22 is configured by arranging image pickup elements such as a charge coupled device (CCD) and a complementary metal oxide semiconductor (CMOS). The surface of the image sensor 22 becomes the imaging surface of the camera 20. By installing a filter according to the emission wavelength of the LED used for the first reference points 11a to 14a and the second reference point 15a on the front surface of the image sensor 22, unnecessary light is eliminated, and the first reference points 11a to 11a are used. Only 14a and the second reference point 15a are photographed. Specifically, for example, when the emission wavelength of the LED used for the first reference points 11a to 14a and the second reference point 15a is 900 nm, a filter that transmits only a wavelength longer than 850 nm is installed on the front surface of the image sensor 22. To do.

FIG. 4 is a diagram illustrating the configuration of the arithmetic device 30.
As shown in FIG. 4, the arithmetic unit 30 determines the positional relationship between the first reference points (see FIG. 2), the second reference point (see FIG. 2) with the first reference point, and the object. The storage unit 31 that stores the three-dimensional position of 300 (see FIG. 1), the input image taken by the camera 20 is taken in, and which set of images is the image of the first reference point and the image of the second reference point When a plurality of sets of the first reference point image and the second reference point image are photographed, the determination unit 32 for determining the object is suitable for specifying the object 300 indicated by the pointing device 100 (see FIG. 1). The selection unit 33 that selects a set, and the three-dimensional position of the target object 300 are read from the storage unit 31, and the target object 300 indicated by the pointing device 100 is specified from the first reference point image and the second reference point image. And a calculation unit 34 for outputting a recognition result.

FIG. 5 is a flowchart for explaining the operation of the arithmetic unit 30.
Hereinafter, an outline of the operation of the arithmetic device 30 will be described with reference to FIGS. 4 and 5.
The determination unit 32 acquires an input image taken by the camera 20 (step 101). Then, the positional relationship between the first reference points and the positional relationship between the second reference point and the first reference point are read from the storage unit 31 (step 102), and the image of the photographed first reference point and the second reference point are read out. It is determined which set of images the image of the point is (step 103).
Next, when a plurality of sets of images of the first reference point and the second reference point are photographed, the selection unit 33 selects an appropriate set for specifying the object 300 indicated by the pointing device 100. Perform (step 104).
Next, the calculation unit 34 reads the three-dimensional position of the object 300 from the storage unit 31 (step 105), and the object 300 indicated by the pointing device 100 from the image of the first reference point and the image of the second reference point. Specify (step 106). This calculation identifies the image of the first reference point and the image of the second reference point from the positional relationship between the image of the first reference point and the image of the second reference point, and the image of the identified first reference point This is performed by obtaining the three-dimensional position, the three-axis angle, and the pointing direction of the pointing device 100 from the image of the second reference point, whereby the target object 300 pointed to by the pointing device 100 can be specified.

  Note that these functions are realized by cooperation of software and hardware resources. That is, the arithmetic device 30 is, for example, a personal computer, and a CPU (not shown) inside the personal computer or the like stores programs for realizing the functions of the storage unit 31, the determination unit 32, the selection unit 33, and the calculation unit 34, for example, a hard disk. These functions are realized in the arithmetic unit 30 by reading the data from the external storage device such as the main memory.

<Distinction of set in image>
Next, a method for discriminating which set of images the first reference point image and the second reference point image that have been taken will be described.
In order to make this determination, for example, in the present embodiment, it is conceivable to arrange the first reference point and the second reference point that are four sets different from each other. In this way, by arranging differently for each reference point set, the determination unit 32 can determine which set of images of the first reference point and the second reference point depending on the arrangement of the first reference point and the second reference point. It is possible to determine whether the image is an image. As another method, the number of first reference points may be different for each reference point. Further, even when all the arrangements are the same, for example, by distinguishing the LED to be blinked for each group and specifying the position of the LED, the determination unit 32 determines which group it belongs to It is also possible to do.

  Here, it is preferable that two or more sets of captured images of the first reference point and the second reference point are acquired, and an appropriate set is selected by the selection unit 33 described above. This selection is, for example, a set that can be calculated with higher accuracy for the three-dimensional position, the three-axis angle, and the pointing direction of the pointing device 100, or the image of the photographed first reference point and the second reference point. What is necessary is just to perform based on the fact that the total light quantity of the image is large. By doing in this way, the target object 300 to which the pointing device 100 points can be specified with higher accuracy.

<Identification of first discrimination point and second discrimination point in image>
Next, a method for identifying each image of the first reference point and the second reference point from the image taken by the camera 20 will be described.
FIG. 6 is an example of the pointing device 100 including three first reference points and one second reference point.
In the pointing device 100 shown in FIG. 6, the three first reference points 11a to 13a are arranged on the upper surface of the pointing device 100 so as to form a triangle (a1a2a3). The triangle (a1a2a3) does not need to have a special feature, but in the case of the triangle (a1a2a3) shown in FIG. 6, it is an isosceles triangle in which the lengths of the sides a2a3 and a1a3 are equal.
The second reference point 15a is arranged at a position farthest from the center of gravity of the four points of the first reference points 11a to 13a and the second reference point 15a. Further, the second reference point 15a is arranged at a position (for example, the front side of the paper surface in FIG. 6) that is higher than the plane including the first reference points 11a to 13a by a certain height. However, since FIG. 6 is a view seen from a direction perpendicular to the plane including the first reference points 11a to 13a, the height of the second reference point 15a is not shown.

  When arranged as described above, the geometric feature in the positional relationship between the first reference points 11a to 13a and the second reference point 15a is the positional relationship between the images of the reference points on the image captured by the camera 20. Is also saved. Therefore, it is only necessary to specify whether the image of each reference point is one of the first reference points 11a to 13a and the second reference point 15a based on this geometric feature. Specifically, the second reference point 15a can be identified by utilizing the positional relationship that the second reference point 15a is located farthest from the center of gravity of the four points.

  As described above, the second reference point 15a is not at the center of gravity of the triangle (a1a2a3) formed by the first reference points 11a to 13a, but at a certain height from the center of gravity. Therefore, strictly speaking, the geometric feature of the position of the second reference point 15a with respect to the first reference points 11a to 13a is not stored at the position of the corresponding point image. However, if the distance between the plane including the first reference points 11a to 13a and the second reference point 15a is sufficiently short, even if the triaxial angle of the pointing device 100 changes, the position of the second reference point 15a is It is not so far from the center of gravity of the triangle (a1a2a3). Therefore, the point image that is farthest from the center of gravity of the triangle (a1a2a3) may be used as the image of the second reference point 15a.

On the other hand, FIG. 7 is an example of the pointing device 100 including four first reference points and one second reference point.
The pointing device 100 shown in FIG. 6 described above is an example having three first reference points and one second reference point, but the pointing device 100 shown in FIG. Increased by one to 4 points.
The pointing device 100 shown in FIG. 7 is arranged such that the first reference points 11a to 14a have a parallelogram shape (a1a2a3a4). In this case, the side a1a2 and the side a3a4 are parallel, and the side a2a3 and a4a1 are parallel.
The second reference point 15a is arranged at a position that is a certain height higher than the plane including the first reference points 11a to 14a.
When arranged as described above, the positional relationship that the sides a1a2 and a3a4 of the first reference points 11a to 14a are parallel and the sides a2a3 and a4a1 are parallel is on the image photographed by the camera 20. It is also stored in the positional relationship of the image of each point. Therefore, it is only necessary to specify whether the image of each reference point is one of the first reference points 11a to 14a and the second reference point 15a based on this geometric feature.

  As described above, the case where the first reference point is 3 points and the case where the first reference point is 4 points has been described. However, even when the first reference point is 5 points or more, the first reference point and the second reference point are based on the above-described geometric features. By arranging, it is possible to identify the first reference point and the second reference point.

<Calculation of three-dimensional position, three-axis angle, and pointing direction of pointing device 100>
Next, a method for calculating the three-dimensional position, the three-axis angle, and the pointing direction of the pointing device 100 based on the positions of the first reference point and the second reference point will be described. When the first reference point is three points Since the calculation method is different for the case of 4 points or more, it will be described separately.

FIG. 8 is a diagram for explaining a method of calculating the three-dimensional position and the three-axis angle of the pointing device 100 when the first reference point is three points.
First, as shown in FIG. 8, the first reference point 11a from the optical center O of the camera 20 based on the point images of the first reference points 11a to 13a on the image taken by the camera 20 (on the image sensor 22). The direction di (i = 1, 2, 3) to each of ˜13a is calculated. di is a standardized unit vector that is on a straight line connecting the point images of the first reference points 11a to 13a and the optical center O on the imaging surface of the camera 20 and starts from the optical center O. The direction to the first reference point 11a is a vector d1 (= [x1, y1, z1]), the direction to the first reference point 12a is a vector d2 (= [x2, y2, z2]), and the first reference point Let the direction to 13a be a vector d3 (= [x3, y3, z3]).

  With reference to the optical center O of the camera 20, the position vector of the first reference point 11a is a1, the position vector of the first reference point 12a is a2, and the position vector of the first reference point 13a is a3. Each of these position vectors ai overlaps the above-mentioned vector di. Therefore, if the coefficients representing the length of each position vector ai are t1, t2, and t3, each position vector ai can be expressed by the following equation.

  Since the shape of the triangle formed by the first reference points 11a to 13a is known in advance, the length of each side of the triangle (the length between the first reference points 11a to 13a) is set to a1a2 = L1 and a2a3 = L2. , A3a1 = L3, the following equation is obtained.

  Arranging this, the following formula is obtained,

  Furthermore, the following equation is obtained.

ただし、
Ａ１＝ｘ１ｘ２＋ｙ１ｙ２＋ｚ１ｚ２、
Ａ２＝ｘ２ｘ３＋ｙ２ｙ３＋ｚ２ｚ３、
Ａ３＝ｘ３ｘ１＋ｙ３ｙ１＋ｚ３ｚ１
である。

However,
A1 = x1x2 + y1y2 + z1z2,
A2 = x2x3 + y2y3 + z2z3,
A3 = x3x1 + y3y1 + z3z1
It is.

  In order to have a real number solution, real numbers t1, t2, and t3 satisfying the following equation are substituted into the above equation 4, and all t1, t2, and t3 that satisfy equation 4 are obtained.

Usually, there are a plurality of sets of t1, t2, and t3 in which Expression 5 is established. Therefore, a specific set of coefficients t1, t2, and t3 is selected from a plurality of sets (candidates) of t1, t2, and t3 based on the position of the second reference point 15a.
Specifically, first, a set of position vectors a1, a2, and a3 corresponding to all of the calculated sets of coefficients t1, t2, and t3 are calculated, respectively, and a triangular cubic composed of the sets of a1, a2, and a3 is calculated. Find the original position and triaxial angle. Based on the three-dimensional position and the triaxial angle of each triangle, the position of the second reference point 15a with respect to each triangle is calculated and set as a position vector a4. Next, a position (image position) on the image when the point of the position vector a4 is captured by the camera 20 is calculated, and the calculated value is compared with the image position of the second reference point 15a actually captured. Then, a triangle corresponding to the position vector a4 having the smallest difference from the actual image position is extracted, and a set of t1, t2, and t3 corresponding to this triangle is taken as a correct answer.

As described above, since the directions d1, d2, and d3 and the distances t1, t2, and t3 of the first reference points 11a to 13a are specified, the triangle formed by the first reference points 11a to 13a (and the instruction including this) The three-dimensional position and the three-axis angle of the device 100) are obtained. In addition, the pointing direction of the pointing device 100 is determined by the positional relationship between the first reference points 11a to 13a and the second reference point 15a.
Note that the calculation method of the three-dimensional position and the three-axis angle of the pointing device 100 using the first reference points 11a to 13a and the second reference point 15a is not limited to the above procedure. For example, in the above procedure, one solution is extracted based on the image position of the second reference point 15a after obtaining a plurality of sets of t1, t2, and t3. From the beginning, each of the first reference points 11a to 13a is extracted. The distances t1, t2, and t3 to the first reference points 11a to 13a and the second reference point 15a may be obtained in consideration of the relationship between the image position and the image position of the second reference point 15a.

Next, a method for calculating the three-dimensional position, the three-axis angle, and the pointing direction of the pointing device 100 when the first reference point is four will be described.
For the position of the first reference point 11a, the coordinates in the coordinate system with reference to the pointing device 100 are [X ₁ , Y ₁ , Z ₁ ] ^T. Further, the coordinates of the image of the point of the first reference point 11a in the coordinate system on the image photographed by the camera 20 (on the image sensor 22) (see FIG. 3) are set to [u ₁ , v ₁ ]. In the following description, a coordinate system with the camera position as the origin is referred to as a camera coordinate system, and the coordinates of the first reference point 11a in the camera coordinate system are referred to as [X _c1 , Y _c1 , Z _c1 ] ^T.
In this case, using the 3 × 4 matrix as the projection matrix P, the following relational expression is established.

The projection matrix P in Equation 6 has a constant multiple indefiniteness, which is absorbed by the coefficient h on the left side of Equation 6. Accordingly, P ₃₄ = 1 can be set. Further, since the first reference points 11a to 14a are on the same plane, the Z coordinate can be set to zero. Therefore, if Z ₁ = 0, the third column of the projection matrix corresponding to Z ₁ in Equation 6 can be omitted. Therefore, Equation 6 can be modified as follows. In this case, there are 8 unknown projection matrices.

  The following two constraint equations can be obtained from the equation (7).

The relationship expressed by Equation 8 is similarly established for the first reference points 12a to 14a. That is, the following relational expressions hold for the first reference points 12a to 14a. Here, the coordinates of the first reference points 12a to 14a in the coordinate system based on the pointing device 100 are respectively [X ₂ , Y ₂ , Z ₂ ] ^T , [X ₃ , Y ₃ , Z ₃ ] ^T. , [X ₄ , Y ₄ , Z ₄ ] ^T, and the coordinates of the images of the first reference points 12 a to 14 a in the coordinate system on the image taken by the camera 20 are [u ₂ , v ₂ ], [U ₃ , v ₃ ], [u ₄ , v ₄ ]. Further, similarly to the first reference point 11a, Z ₂ = Z ₃ = Z ₄ = 0.

  The following formula is obtained by rearranging formula 8 and formula 9.

数１０式において左辺第１項をＡとし、また左辺第２項は射影行列Ｐの３列目がない行列の要素を表していることからＰ３とし、右辺をＢとして書き表すと、以下の式となる。

In Equation 10, the first term on the left side is A, and the second term on the left side is P3 because it represents an element of the matrix without the third column of the projection matrix P. Become.

  Multiplying both sides of Equation 11 by the inverse matrix of A from the left,

Thus, P3 can be obtained.
The projection matrix P can be represented by the product of the camera inner matrix C and the camera outer matrix [r1 r2 r3 t].

  Here, r1, r2, and r3 are rotation vectors for converting the coordinate system on the pointing device 100 into a camera coordinate system, and t is a translation vector. These vectors are all represented as column vectors having three elements. Since P3 is a matrix without the third column of the projection matrix P,

  It becomes. Therefore,

  Thus, the column vectors r1, r2, and t can be calculated. Since r3 is orthogonal to r1 and r2,

It can be expressed as. Therefore, r3 can be calculated from equation (16). As described above, all the rotation vectors r1, r2, and r3 and the translation vector t can be calculated. Therefore, assuming R = [r1 r2 r3], the three-dimensional positions of the first reference points 11a to 14a can be expressed as follows using the camera coordinate system. Note here, the coordinates of the first reference point 12a~14a in the camera coordinate system, respectively ^{_{[X c2, Y c2, Z}} c2] T, [X c3, Y c3, Z c3] T, [X c4, Y c4, Z _c4 ] ^T.

Since the camera coordinates of the first reference points 11a to 14a can be specified from the relational expression (17), the three-dimensional position and the three-axis angle of the pointing device 100 are obtained. The pointing direction of the pointing device 100 is determined by the positional relationship between the first reference points 11a to 14a and the second reference point 15a.
In the above description, the case where the first reference point is four points has been described. However, the calculation can be performed using the same method when the first reference point is five points or more.

<Identification of object>
The object 300 (see FIG. 1) indicated by the pointing device 100 can be specified from the three-dimensional position, the three-axis angle, and the pointing direction of the pointing device 100 obtained as described above. That is, the three-dimensional positions of various objects 300 are stored in the storage unit 31 (see FIG. 4), and which object 300 is instructed is calculated by reading and comparing the three-dimensional positions. And can be identified.
In this case, for example, it is assumed that the instruction device 100 indicates a television screen in the room as the object 300. Here, it is possible to cause the television to perform a specific operation by sending information from the object recognition device 200 (see FIG. 1) to a separately provided control device. For example, power ON / OFF. In addition, other operations such as channel switching and volume adjustment can be performed by instructing other places with the instruction device 100. The location to be instructed is not limited to a television screen or the like. For example, a specific location on the wall surface of the television may be designated as a location for performing the above-described operation. At this time, a laser pointer or the like as a visible light irradiating unit that emits visible light is attached to the tip of the pointing device 100, and a laser that is visible light is irradiated to the same location as the location indicated by the pointing device 100. The user operating the pointing device 100 may be made aware of the location pointed to by the user. In this case, the user can specify the object 300 more accurately. Since a plurality of sets of the first reference point and the second reference point are installed, even if the user uses the pointing device 100 at various angles or three-dimensional positions, any set of the first reference point is set by the camera 20. In addition, the camera 20 can easily capture the second reference point. That is, the pointing device 100 that is less dependent on how the user is used can be realized, and the same object 300 can be recognized regardless of which set of images the camera 20 captures.

  In the present embodiment, the first reference point and the second reference point are light emitting points constituted by LEDs. However, the present invention is not limited to this. For example, a retroreflecting plate is provided instead of the LED as the first reference point and the second reference point, light is emitted from an illumination device provided near the camera 20, and the reflected light from the retroreflecting plate is photographed by the camera 20. You may do it.

It is a figure which shows the whole structure of the target object recognition system containing the target object recognition apparatus, instruction | indication apparatus, and target object to which this embodiment is applied. It is a figure which shows the structure of the instruction | indication apparatus to which this embodiment is applied. It is a figure which shows the structural example of a camera. It is a figure explaining the structure of the arithmetic unit. It is the flowchart explaining operation | movement of the arithmetic unit. It is an example of a pointing device provided with three first reference points and one second reference point. It is an example of a pointing device provided with four first reference points and one second reference point. It is a figure explaining the method to calculate the three-dimensional position and three-axis angle of a pointing device when the first reference point is three points.

Explanation of symbols

11a-14a, 11b-14b, 11c-14c, 11d-14d ... first reference point, 15a, 15b, 15c, 15d ... second reference point, 20 ... camera, 30 ... arithmetic unit, 31 ... storage unit, 32 ... Discriminating unit 33 ... selecting unit 34 ... calculating unit 100 ... indicating device 200 ... object recognizing device 300 ... object

Claims (8)

A plurality of reference point sets consisting of luminous points;
The light emitting points are turned on for each reference point set, and when the light emitting points of one reference point set are turned on, the light emitting points other than the one reference point set are turned off, and control means,
The set of reference points includes three or more first reference points on a plane in which the mutual positional relationship is specified and one second reference point in which the positional relationship between the first reference points is specified. A pointing device characterized in that an object is specified from a positional relationship between an image of the first reference point and an image of the second reference point of the reference point set photographed by another device.   The pointing device according to claim 1, wherein the second reference point is not arranged on the plane on which the first reference point is arranged.   The pointing device according to claim 1, wherein an arrangement of the first reference point and the second reference point is different for each reference point set.   The pointing device according to claim 1, further comprising visible light irradiating means for irradiating visible light to the same location indicated by the pointing device. A photographing unit that photographs a pointing device in which a plurality of first reference points including three or more points on a plane and a plurality of second reference points including one point are arranged;
A storage unit for storing the mutual positional relationship of the first reference points, the positional relationship of the second reference point with the first reference point, and the three-dimensional position of the object;
The positional relationship between the first reference points and the positional relationship of the second reference point with the first reference point is read from the storage unit, and an image of the first reference point and the second reference captured by the imaging unit. A discriminator for discriminating which set of images the image of the points is;
The three-dimensional position of the object is read from the storage unit, and the first reference point image and the second reference point image are obtained from the positional relationship between the first reference point image and the second reference point image captured by the imaging unit. The pointing device is identified by identifying the image of the two reference points and obtaining the three-dimensional position, the three-axis angle and the pointing direction of the pointing device from the identified image of the first reference point and the image of the second reference point. A computing unit for identifying the target indicated by ;
An object recognition apparatus comprising:   The discriminating unit discriminates which set of images the images of the first reference point and the second reference point are based on a difference in arrangement of the first reference point and the second reference point. The object recognition apparatus according to claim 5.   When a plurality of sets of the image of the first reference point and the image of the second reference point are photographed, the image processing apparatus further includes a selection unit that selects an appropriate group for specifying the target object indicated by the pointing device. The object recognition apparatus according to claim 5. On the computer,
The positional relationship between the first reference point consisting of three or more points on the plane arranged on the pointing device and the second reference point consisting of one point is read from the memory, and the image taken by the photographing unit A function of determining which set of images the image of the first reference point and the image of the second reference point are;
The three-dimensional position of the object is read from the memory, and the image of the first reference point and the second reference point are determined from the positional relationship between the image of the first reference point and the image of the second reference point taken by the photographing unit. A point image is identified, and the three-dimensional position, the three-axis angle and the pointing direction of the pointing device are obtained from the identified image of the first reference point and the image of the second reference point. A function to identify the target object ,
A program that realizes

Images