JP2009134677A6 - Gesture interface system, gesture input wand, application control method, camera calibration method, and control program - Google Patents

Abstract

A gesture interface system using a wand having features as a camera calibration object and a gesture interface device in a three-dimensional space, a gesture input wand, an application control method, a camera calibration method, and a control program are provided.
A passive wand tracked by one or more cameras is used as a user interface device. Wands are inexpensive and are a natural way for users to interact with devices such as large displays. Each wand is used to specify a maximum of 6 degrees of freedom and is useful for operating 3D applications. Wands are also used to identify 3D real world points, useful for multi-camera calibration and physical space 3D model building. The described method provides a more accurate technique for estimating the wand pose and can be implemented using a single camera. In addition, the wand posture is estimated using information that can be used in the one-dimensional line scan in the image that simplifies the processing.
[Selection] Figure 1

Description

  The present invention relates to computer interfaces, and more particularly to gesture interfaces. In addition, the present invention relates to camera calibration techniques. That is, the present invention relates to a gesture interface system, a gesture input wand, an application control method, a camera calibration method, and a control program.

  Two conventional gesture interfaces are commonly used. One requires a sensor located in hardware such as a glove that the user needs to wear. This solution is expensive and unpopular with the user. The other is limited to two-dimensional interaction between the user and the screen by touching the screen or determining the position from the sensor. The former provides the user with more freedom. Therefore, it is possible to support a wider range of basic dialogue vocabularies than can be supported by the latter.

  For example, if a wand (cylindrical gesture input device) is used as a user interface, a maximum of 6 degrees of freedom is supported instead of 2 degrees of freedom. Also, more degrees of freedom are supported if multiple wands are used. Even when supporting a vocabulary that is wide enough for dialogue with two degrees of freedom, the alternative interface using the wand described above is advantageous. This is because, for example, in a dialog with a large screen, the user needs to be close enough to touch the screen, and as a result, the entire screen becomes invisible.

  In some applications, a partial solution is employed in which the sensor is placed on some object. The object can be lifted or placed and is similar to an object that the user is accustomed to using in that setting. For example, the sensor can be added to various pointing devices (positioning devices). However, this solution has various problems, including expense, weight, fragile parts, and the need for power supply by cords or batteries.

  In systems based on computer vision, the last three of these problems are avoided. In such a system, essentially all of the cost of the system is spent on the camera, but the camera is cheaper every day. Moreover, systems based on computer vision can be modified to work with an unlimited variety of objects. The main limiting factor is which objects can be reliably tracked. For example, a computer vision-based system designed to track “hand gestures” is attractive, but the difficulty of tracking the hand poses a major problem in robustness .

  Thus, what is particularly needed is a simple, static, and inexpensive object that reliably enables a gesture interface through computer vision technology with full intelligence in the associated camera system.

  “VisionWand: interaction techniques for large displays using a passive wand tracked in 3D”, “Symposium on User Interface Software and Technology, Proceedings of the 16th annual ACM symposium on User interface software and technology (Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology, 173-182 (2003)) Xiang Cao, Ravin Balakrishnan's vision wand research not only makes such a system possible, but their interaction is effective, intuitive, and user It has high evaluation characteristics by It is.

  The authors use a simple wand with colored edges that can be easily tracked by the camera against a contrasting colored background. Various simple wand gestures are used to control the photo application on the large screen, allowing both direct interaction with the photo and menu selection. Described below are two examples that illustrate the benefits of a simpler gesture interface. Zoom in with the “push” gesture with the wand parallel to the screen and zoom out with the “pull” gesture. If the wand is kept parallel to the screen for a short time, the entire menu is presented and a rotation gesture is used to move between menu items. As the user gets used to the system, the user will learn to place the wand at the rotational position of the desired menu item. Such a shortened dialogue is possible because the above-mentioned gesture interface can use these degrees of freedom.

Xiang Cao, Ravin Balakrishnan., "VisionWand: interaction techniques for large displays using a passive wand tracked in 3D." Xiang Cao, Ravin Balakrishnan., "Symposium on User Interface Software and Technology, Proceedings of the 16th annual ACM symposium on User interface software and technology." Pp.173-172 (2003). YASUJI SEKO, et al., "Proposal of recordable pointer: Pointed position measurement by projecting interference concentric circle pattern with a pointing device." IEEE, 2006, Corporate Research Group, Fuji Xerox Co., Ltd.

  However, the prior art cannot provide improved wand detection and tracking techniques. That is, it allows the user to interact with the computer system interface even when the majority of the object's field of view, including both ends of the wand, is obstructed, thereby providing a separate interface for use by the gesture interface. It is not possible to provide improvement techniques that can allow the natural degree of freedom.

  In addition, it would be desirable to have a wand that allows much higher accuracy. For example, it would be desirable to have a wand that can be used as a precision positioning device suitable for controlling position in a three-dimensional modeling tool.

  The methodology of the present invention is directed to a method and system that substantially eliminates at least one of the above problems and other problems associated with conventional gesture interfaces.

  That is, an object of the present invention is to use a gesture interface system, a gesture input wand, an application control method, a camera calibration method, using a wand having features as a camera calibration object and a gesture interface device in a three-dimensional space, And providing a control program.

  According to one aspect of the inventive concept, at least one camera that is a wand operated by a user and that operates to create an image of the wand with at least one feature, and the image of the created wand And a processing module operative to determine a wand attitude based on at least one characteristic of the wand. Wand posture includes wand position, wand orientation, and wand twist. The wand attitude determined as described above is used to control the user application.

  In accordance with another aspect of the inventive concept, a wand for use with a gesture computer interface is provided. The wand of the present invention includes a twist code area, a color code area, and a precision feature area.

  According to another aspect of the inventive concept, a method is provided for controlling a computer application based on a wand attitude that is operated by a user and has at least one feature. The method of the present invention includes the steps of creating an image of a wand operated by a user using at least one camera and determining the posture of the wand based on at least one feature of the wand using the created image. And a step of performing. Wand posture includes wand position, wand orientation, and wand twist. The method of the present invention further involves controlling a computer application based on the determined wand attitude.

  According to another aspect of the inventive concept, controlling a computer application based on an image of a wand operated by a user created by at least one camera and obtained by at least one camera. And calibrating at least one camera based on the image of the wand.

  Some other aspects related to the present invention will be described below. In addition, some of the other aspects related to the present invention will be apparent from the following description, or can be understood from the embodiments of the present invention. Aspects of the invention may be realized and implemented by means of the “elements” and “combinations of various elements with various aspects” specifically set forth in the following detailed description and the appended claims.

  It is to be understood that both the foregoing and following description is for purposes of illustration and description only and is not intended to limit the claimed invention or the application of the invention in any way. is there.

  The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, are used to explain and illustrate the principles of the techniques of the invention.

  That is, the invention described in the claims of the present invention has the following characteristics.

<Gesture interface system>
The invention described in claim 1 is a gesture interface system for interacting with a computer by a gesture, comprising: a. At least one camera operated by a user and operable to create an image of a wand for gesture input having at least one feature in an appearance pattern; b. Receiving an image of the wand created from the camera and determining the position of the wand, including the position of the wand, the orientation of the wand, and the twist of the wand, based on the at least one feature of the wand; Processing means for operating and controlling a user application using the determined attitude of the wand.

  According to a second aspect of the present invention, in the first aspect of the present invention, the processing means operates to determine the posture of the wand based on at least a plurality of color regions of the wand. Features.

  According to a third aspect of the present invention, in the first aspect of the present invention, the processing means operates so as to determine the posture of the wand based on at least the twist code region of the wand. And

  According to a fourth aspect of the present invention, in the third aspect of the present invention, the twisted code region includes a spiral pattern.

  According to a fifth aspect of the present invention, in the first aspect of the present invention, the processing means operates so as to determine the posture of the wand based on at least a fine feature region.

  According to a sixth aspect of the present invention, in the fifth aspect of the present invention, the precision feature region includes a checkerboard pattern.

  A seventh aspect of the invention is characterized in that, in the sixth aspect of the invention, the checkerboard pattern includes a dot or a blurred blob in the center of the checkerboard grid.

  The invention described in claim 8 is characterized in that, in the invention described in claim 1, the wand further comprises a barcode area for encoding information.

  The invention described in claim 9 is the invention according to claim 8, wherein the processing means determines the identifier of the wand based on the information encoded in the barcode area of the wand. It is characterized by operation.

  The invention described in claim 10 is the second camera operated by the user and operated to create the image of the wand in the invention of claim 1, wherein the at least one camera and Is further provided with the second camera arranged at a different position.

  The invention described in claim 11 is the invention described in claim 1, wherein the processing means reconfigures the scan line of the wand, and the scan line of the wand and the at least one feature of the wand It operates to determine the posture of the wand by obtaining an intersection point.

  According to a twelfth aspect of the present invention, in the first aspect of the invention, one end of the wand is provided with a color code, and the processing means uses the color code when only a part of the wand is visible. It operates to determine the posture of the wand.

  The invention described in claim 13 is the invention described in claim 1, wherein the at least one camera is an infrared camera, and a surface of the wand includes an infrared reflective material or a retroreflective material.

  The invention described in claim 14 is the invention according to claim 13, wherein the infrared camera includes at least one infrared light emission source and at least one infrared filter.

  The invention described in claim 15 is the invention according to claim 1, wherein the processing means calibrates the at least one camera based on the image of the wand acquired by the at least one camera. It is characterized by operating.

<Wash for gesture input>
The invention described in claim 16 is a gesture input wand for use in a gesture interface comprising: a. A torsion cord region; b. A color code region; c. A precise feature region is provided as a feature on the appearance pattern.

  The invention described in claim 17 is characterized in that, in the invention described in claim 16, the color code region comprises a plurality of color bands distributed along the length of the wand.

  The invention described in claim 18 is the invention described in claim 17, characterized in that the plurality of color bands have different colors.

  The invention described in claim 19 is characterized in that, in the invention described in claim 16, the twisted code region has a spiral pattern.

  The invention described in claim 20 is characterized in that, in the invention described in claim 16, a plurality of spiral patterns distributed along the length of the wand are provided.

  The invention described in claim 21 is characterized in that, in the invention described in claim 16, the precision feature region includes a checkerboard pattern.

  According to a twenty-second aspect of the present invention, in the twenty-first aspect, the checkerboard pattern includes a dot or a blurred blob at the center of the checkerboard grid.

  The invention described in claim 23 is characterized in that, in the invention described in claim 16, a bar code area including encoded information is further provided.

  The invention described in claim 24 is characterized in that, in the invention described in claim 23, the encoded information includes the identifier of the wand.

  The invention described in claim 25 is the invention described in claim 16, wherein at least one end of the wand is provided with a color code.

  The invention described in claim 26 is the invention described in claim 16, characterized in that the surface of the wand comprises an infrared reflecting material or a retroreflective material.

<Application control method>
The invention described in claim 27 is an application control method for controlling a computer application based on a posture of a wand for gesture input that is operated by a user and has at least one feature in an appearance pattern, comprising: a. Creating an image of a wand operated by the user using at least one camera; b. Determining the position of the wand, including the position of the wand, the orientation of the wand, and the twist of the wand, based on the at least one feature of the wand using the created image; c. Controlling the computer application based on the determined attitude of the wand.

<Camera calibration method>
The invention described in claim 28 is a camera calibration method for calibrating a camera based on an image of a wand for gesture input operated by a user and having at least one feature in an appearance pattern, comprising: a. Creating an image of a wand operated by the user using at least one camera and controlling the computer application based on the created image; b. Calibrating the at least one camera based on the image of the wand acquired by the at least one camera.

<Control program>
The invention described in claim 29 is a control program for controlling a computer application based on a posture of a wand for gesture input which is operated by a user and has at least one feature on an appearance pattern. A. Obtaining an image of the wand created by at least one camera; b. Determining the position of the wand including the position of the wand, the orientation of the wand, and the twist of the wand based on the at least one feature of the wand using the created image; c. Controlling the computer application based on the determined attitude of the wand.

<Gesture interface system>
According to the invention according to claim 1 (gesture interface system), the wand having at least one feature on the appearance pattern can be used to facilitate detection and tracking of the wand, and precise positioning from the posture of the wand is possible. As described above, there is an effect that it is possible to realize a reliable gesture interface that enables interaction with a computer, that is, control of an application.

  According to the invention of claim 2, it is easy to roughly track the wand, and there is an effect that it can be used for initial setting of tracking.

  According to the third and fourth aspects of the invention, there is an effect that it is useful for determining the twist of the wand with respect to the axis of the wand.

  According to the fifth and sixth aspects of the invention, there is an effect that when the wand is near the camera, the wand posture can be determined more precisely. According to the invention of claim 7, there is an effect that the accuracy can be further improved.

  According to the invention of claim 8, there is an effect that encoded information can be included.

  According to the invention which concerns on Claim 9, there exists an effect that a wand can be identified with an identifier.

  According to the tenth aspect of the invention, the position and posture of the wand can be accurately obtained, and even when one camera loses sight of the position of the wand, the tracking of the wand can be recovered. is there.

  According to the eleventh aspect of the present invention, there is an effect that it is possible to obtain sufficient information for determining the posture of the wand by one-dimensional line scanning.

  According to the twelfth aspect of the present invention, there is an effect that the posture of the wand that is difficult to distinguish can be distinguished.

  According to the inventions according to claims 13 and 14, there is an effect that the obtained infrared image is easily segmented and has a very high signal-to-noise ratio.

  According to the invention which concerns on Claim 15, there exists an effect that the wand used for gesture input can be used also as an object for camera calibration.

<Wash for gesture input>
According to the sixteenth aspect of the present invention (gesture input wand), it is possible to easily detect and track a wand by using a wand having a twist code area, a color code area, and a fine feature area as features on the appearance pattern. At the same time, there is an effect that it is possible to realize a precise gesture interface that enables precise positioning from the wand posture and enables interaction with the computer, that is, control of the application.

  That is, it becomes easy to determine the twist of the wand by the twist code area, it becomes easy to roughly track the wand by the color code area, and the attitude of the wand can be determined more precisely by the precision feature area. .

  According to the seventeenth aspect of the invention, there is an effect that the distance from the wand end can be uniquely determined by line scanning. According to the invention of claim 18, even when the distance from the end of the wand is uniquely determined, there is an effect that it becomes easy to roughly track the wand.

  According to the nineteenth aspect of the invention, there is an effect that it is easy to determine the twist of the wand by line scanning.

  According to the twentieth aspect of the present invention, there is an effect that the posture of the wand can be determined even when only a small portion of the wand is visible.

  According to the twenty-first aspect of the present invention, there is an effect that the wand posture can be determined more precisely when the wand is near the camera. According to the twenty-second aspect of the invention, there is an effect that the accuracy can be further improved.

  According to the invention of claim 23, there is an effect that encoded information can be included.

  According to the invention of claim 24, there is an effect that the wand can be identified by the identifier.

  According to the invention of claim 25, there is an effect that the posture of the wand that is difficult to distinguish can be distinguished.

  According to the twenty-sixth aspect of the present invention, there is an effect that the obtained infrared image is easily segmented and has a very high signal-to-noise ratio.

<Application control method>
According to the invention (application control method) according to claim 27, it is possible to easily detect and track the wand by using the gesture input wand having at least one feature on the appearance pattern, and to accurately detect the wand posture. Therefore, there is an effect that it is possible to perform accurate positioning and to perform interaction with the computer, that is, control of the application with certainty.

<Camera calibration method>
According to the invention of claim 28 (camera calibration method), the gesture input wand having at least one feature on the appearance pattern is used to facilitate the detection and tracking of the wand and to accurately detect the wand posture. Effective positioning is possible, and the camera can be calibrated with high accuracy.

<Control program>
According to the invention (control program) of claim 29, it is possible to easily detect and track the wand using the gesture input wand having at least one feature on the appearance pattern, and to accurately detect the wand posture. Positioning is possible, and there is an effect that dialogue with a computer, that is, application control can be reliably performed.

  In the following detailed description, reference is made to the accompanying drawings. In the drawings, elements having the same function are indicated by the same numbers. The foregoing accompanying drawings illustrate specific embodiments and implementations consistent with the principles of the present invention, but are not intended to limit the invention and are illustrated by way of example. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and other implementations may be made without departing from the scope and spirit of the invention. It should be understood that configurations may also be utilized and that structural changes and / or substitutions of various elements may be made. The following detailed description is, therefore, not to be construed in a narrow sense. In addition, the various embodiments of the present invention described below can be implemented in the form of software running on a general purpose computer, can be implemented in the form of dedicated hardware, or implemented as a combination of software and hardware. be able to.

  The inventive techniques described herein improve the aforementioned Shan Khao and Rabin Balakrishnan study referred to herein in several feasible ways. In particular, the technique of the present invention greatly facilitates wand detection and tracking and allows interaction even when the majority of the object's field of view, including both ends of the wand, is obstructed. It then allows the determination of the torsion of the object, thereby supporting another natural degree of freedom for use by the gesture interface.

  In the study of Shan Khao and Rabin Balakrishnan, the main role of the wand was as a gesture device (gesture input device), not a precise pointing device (positioning device). This is because, in their research case, it was difficult to achieve accuracy. The wand of the present invention allows for much higher accuracy. Thereby, for example, it can be used as a precise pointing device suitable for controlling the position with a three-dimensional modeling tool.

<Camera calibration and application to 3D reconstruction>
A desirable first step in any computer vision system is camera calibration. A common way to calibrate a camera is to swing a checkerboard around the front of the camera. To calibrate multiple cameras at once, it can often be difficult to find a checkerboard position that multiple cameras can simultaneously identify if each camera is not similarly positioned. Also, the checkerboard is used only for calibration, not as part of the system. However, those who use objects that are routinely used as part of the system so that they are at hand whenever the camera needs to be calibrated, such as when the camera is moved, added, or replaced However, it should be more convenient than a checkerboard.

  "Wide Area Camera Calibration Using Virtual Calibration Objects", such as Xing Chen, James Davis, and Philipp Slusallek, IEEE Comp. Soc. Conf. On Computer Vision and Pattern Recognition (CVPR), calibration using a laser pointer as described in the June 2000 issue of the Computer Society Conference on IEEE Computer Vision and Pattern Recognition (CVPR) Is successful. Calibration using a laser pointer addresses the problem of using a readily available device that is useful beyond calibration, as well as the problem of being visible from multiple cameras. However, there is a limitation in that it is not so easy to identify and track a single point.

  According to one embodiment of the technique of the present invention, the same object that enables the gesture interface is also used as an object for camera calibration. This is because there is a lot in common between how an object can be identified and how easy it is to track, both in calibration and gesture recognition. In particular, the features on the wand that allow the wand to function as a gesture interface device as well as an effective calibration object will be described.

  It would be useful if a three-dimensional (3D) space could be reconstructed from the camera image and the image points and the world points could be associated. The wand can be used to point to a “real world point” that identifies the corresponding image point obtained with any camera that can photograph the wand. For example, consider the problem of installing a surveillance system that includes many cameras. Modern monitoring systems can include a constrained three-dimensional model of the environment. Part of this model can be determined by the available floor plan. However, in the monitoring system, neither the position of furniture nor the texture image necessary for the model can be used.

  After the camera is set up and installed, an “application” can be built in which the user uses the wand to specify the location of furniture, displays, pictures, etc. For example, just as a drawing program has a mode to draw a flat rectangle on a plane, a space modeling application uses a wand to define a rectangle in a modeled space, such as on a table or desk. You can have a mode. Once the space model is formed, the wand also provides a way to specify virtual changes, such as trying where furniture can be moved.

<Example of wand design>
The remainder of this description discusses techniques for designing and tracking the wand of the present invention. FIG. 1 shows an example of a wand according to an embodiment of the present invention. In FIG. 1, some useful aspects of the wand 100 are illustrated. Specifically, the illustrated wand 100 has colored regions 101 to 103 (blue region 101, green region 102, red region 103). These regions are easy to track roughly and are easy to use for initial tracking. The wand 100 also includes a region having a spiral marking 104. This helical marking 104 helps to determine the wand twist about the axis of the wand 100.

  The wand 100 may also include information encoded in the form of a band having a cylindrical bar code 105 or a band having a corner (checkerboard) 106. These can be used to more accurately determine the wand's posture when the wand is near the camera. In order to increase accuracy, the checkerboard pattern 106 may include a dot or a blurred blob (colored portion with a blurred outline) in the center of the checkerboard grid.

  In one embodiment of the present invention, the wand of the present invention is easily constructed by printing a design pattern and winding the pattern around a cylinder. A printable design pattern 201 is shown in FIG. By using this design pattern 201, the wand 100 having the features of the present invention can be easily configured. The wand (design pattern 201) shown in FIG. 2 includes a handle area 202, a bar code area 105, a twist code area 104, color blobs 101, 102 and 103 for rough tracking, a precision feature area 106, Is included. This pattern 201 is printed and wound on a cylinder to produce a wand 100 as shown in FIG.

<Details of technique>
FIG. 3 shows an overview of a processing unit 300 corresponding to processing means of an exemplary system according to an embodiment of the present invention. The first step in tracking the wand is a coarse level of tracking. This can be based on the different colored areas found on the wand (see step 303). In one embodiment, in the initial setting step 302, the color history is searched using the color model 306 using the operation history 305 of the captured image 301. Once these colored regions are detected, for example, a modified version of the CAM SHIFT (continuously adaptive mean shift) tracking algorithm provided as part of the OpenCV toolkit well known to those skilled in the art. Are tracked using a dynamic model 308 or the like. If the tracking program (tracker) fails or the tracking results from multiple cameras do not match, the initialization of the tracking program is performed again using the initialization step 302.

  After the rough position of the wand is obtained, in step 304, a scan line along the length of the wand is detected using the wand edge 307 to extract the wand features. Is done. This procedure is further illustrated in FIG. For example, a dynamic model 308 such as a Kalman filter performs smoothing and predicts the position of the wand for processing of the next frame. In step 310, the wand's 6-axis pose (6D pose) is estimated. Subsequently, in step 309, gesture processing is performed. Then, the user application 311 is controlled using the result of the gesture processing.

<Method of determining posture>
Next, an example of a method for determining the posture from three points along the wand will be described. It should be understood that the method described is exemplary only and other more reliable methods that use more points may be used. For the sake of brevity, this simpler method is described here on the basis of typical poses and general case assumptions, but this method is easy to handle for special cases. Can be extended.

The attitude of the wand 100 is characterized by its “position” and “orientation”. The position is given by the vector w = (w _x , w _y , w _z ) in the x, y, z coordinate system. This is obtained as the position of the “endpoint” of the wand. The orientation of the wand 100 is given by the normal vector n = ( _nx , _ny , _nz ) as shown in FIG. In addition, the “twist” of the wand is also given by an angle θ that specifies a rotation about an axis determined by n. In the wand line-scan model, the position of the other points is given by the distance from the end points along the normal 501 of each point, as shown in FIG.

FIG. 4 shows an example embodiment of the wand 100 and its projection 402 onto the image plane 401. Specifically, the wand 100 is projected on the image plane 401 by pinhole perspective transformation. A wand posture with 5 degrees of freedom is expressed by “position” w = (w _x , w _y , w _z ) and “direction” described by “normal vector n = ( _nx , _ny , _nz )”, Specified by. Another degree of freedom is also provided by rotation (twisting) of the wand around n (not shown). The “orientation” n of any wand has a vanishing point at the point where a ray with direction n from the origin intersects the image plane, regardless of position.

The attitude of the wand 100 is determined from image coordinates that are key points along the wand. For analysis, the world point coordinates are considered in a natural reference frame of the camera, but of course, if the camera pose is known, these are It can be converted to an arbitrary coordinate system. Assuming a pinhole camera model with the center of projection at the origin, the image points (x, y) of the real world points w _x , w _y , w _z are given by

In the expression, f is a focal length of the camera, that is, a distance between the image plane 401 and the projection point, and x _c and y _c designate a principal point of the image.

FIG. 5 shows an example of a wand 100 decoding technique. The image is decoded based on a line scan (line scan). Wands are detected in the image using edge analysis. Or it is detected by simple color tracking of different colors. Next, a scan line 501 is acquired along the length of the wand, and the image coordinates of the intersection between the scan line 501 and the start point of the color area are (x ₀ , x ₀ ), (x ₁ , y ₁ ), (X ₂ , y ₂ ). Also, the image coordinates of the intersection between the scan line 501 and the torsion code area are (x _a , y _a ). This information is sufficient to determine the wand attitude.

The result of any value of f and x _c , y _c is simply a transformation and scaling of the image coordinates, so in this discussion f = 1, x ₀ = 0, y ₀ = 0 Assume. The image coordinates of the projected image of the point whose distance is r from the wand end point is obtained as follows.

Please note the following points. That is, at r = 0, this is simply an image of the end point of the wand. Also, at the limit where r is infinite, this is at the vanishing point (a = _nx / _nz , b = _ny / _nz ) corresponding to any ray in the direction _nx , _ny , _nz. approach.

Consider three points along the wand line where the first point is at the end of the wand, the second point is at a distance r ₁ from the end, and the third point is at a distance r ₂ from the end. . When a wand is detected in the image, these points are determined by line scanning along the wand and the starting points of the different color regions are searched. The coordinates of these three points are represented by (x ₀ , y ₀ ), (x ₁ , y ₁ ), and (x ₂ , y ₂ ). Using the above _{equation, w x, w y, w} z and n _x, n _y, six equations six unknowns that n _z, i.e., the following three formulas, in y _1, y _2, y ₃ Three similar equations can be determined.

  However, it should be noted that since n is a normal vector, there are 5 degrees of freedom. Solving these equations, the wand's attitude can be determined until it reaches “twist”, ie, a rotation a around the long axis of the wand.

<Helix to determine twist>
In addition, the image coordinates x _a , y _{a of the} point where the line scan intersects the twist encoding spiral is also determined. From this image coordinates and wand orientation, distance r _a to the point from the wand end is determined by the following equation.

In this case, the length of the twist encoding band on the wand is “L _T ”, the twist along the length is one rotation, and the start position of the band is “r _T ”. Then, the twist angle θ is as follows.

<Summary of the method for calculating the posture of the wand of the present invention>
The method of determining the posture of the wand of the present invention is as follows. The image points (x ₀ , y ₀ ), (x ₁ , y ₁ ), (x ₂ , y ₂ ) are input and the following formula is used to position point w _x, w _y, calculates a w _z, wand orientation vector n _x, n _y, and n _z, a.

<Other designs that enable posture determination>
In the following, a number of design features different from the above-described embodiment that are useful for determining the wand attitude will be described.

(Removal of ambiguity in posture)
Some poses are distinguishable in the strict case, but may not be distinguishable in the presence of relatively little noise. For example, it is actually difficult to distinguish between two postures, that is, a posture that is substantially parallel to the camera but inclined +5 degrees, and a posture that is substantially parallel to the camera but inclined -5 degrees. In one embodiment of the invention, the wand is designed to have a flat end that is orthogonal to the scan line. Both ends can be colored differently. Since only the end pointing to the camera is detected, the two postures can be distinguished depending on which color is detected.

(Determine posture even when most of the wand is difficult to see)
In one embodiment of the present invention, a band-like pattern that uniquely determines the distance from the end of the wand is used with a spiral pattern that is wrapped around the wand multiple times so that only a small portion of the wand is visible. Even the posture can be determined.

  FIG. 6 shows an example of an embodiment of a wand distributed feature pattern 600. Information can be redundantly encoded on the wand in many possible ways. The wand shown in this example has a plurality of torsion coding bands 601. A blue band 602 is disposed between the plurality of twist coding bands. The relative placement of each blue band 602 indicates the relative position of the blue band 602 along the wand.

(Decoding multiple feature points)
One embodiment of the technique of the present invention includes processing features that can be determined along a one-dimensional line scan. However, when the wand is sufficiently close to the camera, the width of the wand increases and points that are not on a single line can be used for posture estimation. The OpenCV package, known to those skilled in the art, provides an embodiment of an algorithm that determines the pose from a set of image points corresponding to real world points. In one embodiment of the invention, this algorithm is used to determine the wand pose using a single camera.

(Decoding based on multiple cameras)
Using multiple cameras has several advantages. One advantage is that when a wand's pose cannot be successfully decoded, typically because one camera is mostly aimed at that wand, the wand's pose is properly determined by another camera. Can be done. Also, the position of the wand can be determined much more accurately using two cameras, generally using trigonometry. Even if one of the cameras loses wand position due to, for example, difficulty in seeing specular reflections or wands, the tracking program in the camera determines the posture determined by the other camera. Help restore wand tracking.

(Encoding Wand Information Using Cylindrical Bar Code)
Once the wand attitude is determined, an area containing encoded information such as a wand identifier may be decoded. The image line scan is returned to a line scan along the wand. That is, the influence of fluoroscopy is eliminated and the barcode is decoded. This can be done in various ways. However, a simple method comprises n band positions of white or black that encode 1 bit of information per coding band.

(Infrared irradiation and retroreflective mark)
Another embodiment of the system of the present invention uses an infrared camera surrounded by infrared (IR) light emitting diodes as shown in FIG. In this embodiment, the wand is composed of infrared reflecting paper or retroreflective material that strongly reflects back infrared light in the direction of the light source. When these infrared cameras are equipped with suitable infrared filters, the resulting image is very easy to segment.

  FIG. 7 shows an example of an embodiment of an infrared camera for infrared tracking. This figure shows a modified network camera 701 such as “AXIS206 (registered trademark)”. The network camera 701 includes an infrared light emitting diode 702 as a light source. An infrared filter (not shown) is provided under the lens, so that the image of the wand composed of the infrared reflective material has a very high signal-to-noise ratio (S / N). .

<Example of hardware platform>
FIG. 8 is a block diagram that illustrates an embodiment of a computer / server system 800 upon which an embodiment of the inventive methodology may be implemented. The system 800 includes a computer / server platform 801, peripheral devices 802, and network resources 803.

  The computer platform 801 may include a data bus 804 or another communication mechanism that communicates information between various portions of the computer platform 801 and a processor 805 connected to the bus 804. The processor 805 processes information and performs other computational processes and control tasks. Computer platform 801 also includes volatile storage 806, such as random access memory (RAM) or other dynamic storage devices, connected to bus 804. Volatile storage 806 stores various information as well as instructions executed by processor 805. Volatile storage 806 can also be used to store temporary variables and other intermediate information during instruction execution by processor 805.

  The computer platform 801 may further include a read only memory (ROM or EPROM) 807 or other static storage device connected to the bus 804. The read only memory 807 and the like store static information and instructions for the processor 805 such as a basic input / output system (BIOS) as well as various system configuration parameters. A permanent storage device 808, such as a magnetic disk, optical disk, solid state flash memory device, etc., is provided and connected to the bus 804 for storing information and instructions.

  The computer platform 801 may be connected to a display 809 such as a cathode ray tube (CRT), a plasma display, or a liquid crystal display (LCD) via a bus 804. A display 809 displays information to a system administrator or a user of the computer platform 801. An input device 810 including keys such as alphanumeric characters is connected to the processor 805 via the bus 804 to transmit information and command selection. Another type of user input device is a cursor control device 811 such as a mouse, trackball, cursor direction key, etc. that communicates direction information and command selections to the processor 805 and controls cursor movement on the display 809. This input device typically has two degrees of freedom in two axes, a first axis (such as x) and a second axis (such as y), which allows the device to specify a position in the plane.

  An external storage device 812 that provides additional or removable storage capacity to the computer platform 801 may be connected to the computer platform 801 via the bus 804. In one embodiment of the computer system 800, an external removable storage device 812 may be used to facilitate data exchange with other computer systems. In addition, at least one camera 830 is connected via a bus 804 to capture a wand image.

  The invention is related to the use of computer system 800 for implementing the techniques described herein. In one embodiment, the system of the present invention may be on a machine such as computer platform 801. According to one embodiment of the invention, the techniques described herein are responsive to processor 805 executing one or more sequences of one or more instructions included in volatile memory 806. And executed by the computer system 800.

  Such instructions may be read into volatile memory 806 from another computer readable medium, such as persistent storage 808. Execution of the instruction sequence contained in volatile memory 806 causes processor 805 to perform the process steps described herein. In alternative embodiments, the present invention may be implemented using wired connection circuitry instead of or in combination with software instructions. Thus, embodiments of the invention are not limited to any specific combination of hardware circuitry and software.

  The term “computer-readable medium” as used herein refers to any medium that participates in providing instructions to the processor 805 for execution. A computer-readable medium is only one example of a machine-readable medium that may carry instructions for performing any of the methods and / or techniques described herein. Such a medium may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media includes, for example, optical disks and magnetic disks, such as storage device 808. Volatile media includes dynamic memory, such as volatile storage 806.

  Common forms of computer readable media include, for example, floppy disk, flexible disk, hard disk, magnetic tape, or any other magnetic medium, CD-ROM, any other optical medium, punch card, Paper tape, physical media with any other hole pattern, RAM, PROM, EPROM, flash EPROM, flash drive, memory card, any other memory chip or cartridge, or any other media that can be read by a computer included.

  Various forms of computer readable media may be involved in carrying one or more sequences of one or more instructions for execution to processor 805. For example, the instructions may first be carried on a magnetic disk from a remote computer. Alternatively, the remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 800 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal.

  The infrared detector can receive data carried as an infrared signal and suitable circuitry can place the data on the data bus 804. Bus 804 carries the data to volatile storage 806, from which processor 805 retrieves and executes the instructions. The instructions received by volatile memory 806 may optionally be stored on persistent storage 808 either before or after execution by processor 805. The instructions may also be downloaded to the computer platform 801 via the Internet using various data communication protocols known in the art.

  The computer platform 801 also includes a communication interface such as a network interface card 813 connected to the data bus 804. The communication interface 813 provides two-way data communication that connects to a network link 814 that is connected to a local network 815. For example, the communication interface 813 can be an integrated services digital network (ISDN) card or modem that provides a data communication connection to a corresponding type of telephone line. As another example, the communication interface 813 may be a LAN interface card (LAN NIC) that provides a data communication connection to a compatible local area network (LAN). Also well-known wireless links such as 802.11a, 802.11b, 802.11g, Bluetooth can be used for network implementation. In any such implementation, communication interface 813 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

  Network link 814 typically provides data communication through one or more networks to other network resources. For example, network link 814 may provide a connection to host computer 816 or network storage / server 822 via local network 815. Additionally or alternatively, the network link 814 may connect to a wide area or global network 818, such as the Internet, via a gateway / firewall 817. Thus, the computer platform 801 can access network resources located anywhere on the Internet 818, such as a remote network storage / server 819. On the other hand, the computer platform 801 may also be accessed by clients located anywhere on the local area network 815 and / or the Internet 818. Network clients 820 and 821 may themselves be implemented based on a computer platform similar to platform 801.

  Both the local network 815 and the Internet 818 use electrical, electromagnetic or optical signals that carry digital data streams. Signals through various networks and signals over network link 814 and communication interface 813 that carry digital data to and from computer platform 801 are exemplary forms of carriers that carry information.

  The computer platform 801 can send messages and receive data including program codes via various networks, including the Internet 818 and the local area network 815, the network link 814 and the communication interface 813. In the Internet example, the system 801, when acting as a network server, on the client (s) 820 and / or 821 via the Internet 818, gateway / firewall 817, local area network 815 and communication interface 813. It is also possible to send code or data required for the application program to be executed.

  The received code may be executed by processor 805 when received and / or stored in persistent or volatile storage devices 808 and 806, or other non-volatile storage, for later execution, respectively. In this manner, computer system 801 may obtain application code in the form of a carrier wave.

  Finally, it should be understood that the processes and techniques described herein are not inherently related to any particular apparatus and may be implemented by any suitable combination of components. Further, various types of general purpose devices can be used in accordance with the techniques described herein. It can also be seen that it would be advantageous to construct a dedicated device for performing the method steps described herein. Although the invention has been described in connection with specific examples, they are in all respects illustrative and not restrictive. Those skilled in the art will appreciate that many different combinations of hardware, software, and firmware are suitable for practicing the present invention. For example, the software described above can be implemented in a wide variety of programming or scripting languages such as assembler, C / C ++, Perl, Shell, PHP, Java (registered trademark), and the like.

  Furthermore, other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. Various aspects and / or components of the foregoing embodiments may be used alone or in any combination in a computer storage system having data replication capabilities. The specification and examples are to be regarded as illustrative only, with the true scope and spirit of the invention being indicated by the appended claims.

It is a figure which shows an example of embodiment of a wand. It is a figure which shows an example of embodiment of a wand pattern. It is a figure which shows the outline | summary of an example of the system which concerns on one Embodiment of the system of this invention. It is a figure which shows an example of embodiment of a wand and its projection on the image surface. FIG. 6 illustrates a wand decoding technique. It is a figure which shows an example of embodiment provided with the distributed feature pattern. FIG. 6 illustrates an example of an embodiment of a camera modified for infrared (IR) tracking. FIG. 2 is a diagram illustrating an example of an embodiment of a computer platform on which the system of the present invention can be implemented.

Explanation of symbols

100 Wand 101, 102, 103 Color blob (colored area)
104 Spiral marking (twist cord area)
105 Cylindrical barcode (barcode area)
106 Checkerboard pattern (precision feature area)
201 Design Pattern 202 Handle Area 300 Processing Unit 301 Captured Image (Video Image)
305 Operation history 306 Color model 307 Edge 308 Dynamic model (Kalman tracker, etc.)
311 User application 401 Image surface 402 Projection 501 Scan line 600 Feature pattern 601 Coding band 602 Blue band 701 Network camera 702 Infrared light emitting diode 800 Computer system 801 Peripheral device 803 Network resource 804 Data bus 805 Processor 806 Volatile storage device 807 EPROM / firmware storage device 808 Persistent storage device 809 Display 810 Keyboard (input device)
811 Cursor control device (mouse / positioning device)
812 External storage device 813 Network interface card 814 Network link 815 Intranet (LAN)
816 Host 817 Gateway / Firewall 818 Internet 819, 822 Network server / Network storage 820, 821 Client 830 Camera

Claims (29)

A gesture interface system for interacting with a computer by a gesture,
a. At least one camera operated by a user and operable to create an image of a wand for gesture input having at least one feature in an appearance pattern;
b. Receiving an image of the wand created from the camera and determining the position of the wand, including the position of the wand, the orientation of the wand, and the twist of the wand, based on the at least one feature of the wand; Processing means that operates and controls a user application using the determined attitude of the wand;
Gesture interface system with   The gesture interface system according to claim 1, wherein the processing unit operates to determine the posture of the wand based on at least a plurality of color regions of the wand.   The gesture interface system according to claim 1, wherein the processing means operates to determine the posture of the wand based on at least a twist code region of the wand.   The gesture interface system according to claim 3, wherein the twist code region comprises a spiral pattern.   The gesture interface system according to claim 1, wherein the processing means operates to determine the posture of the wand based on at least a precision feature region.   The gesture interface system according to claim 5, wherein the precision feature region comprises a checkerboard pattern.   The gesture interface system according to claim 6, wherein the checkerboard pattern includes a dot or a blurred blob in the center of the checkerboard grid.   The gesture interface system according to claim 1, wherein the wand further includes a barcode area for encoding information.   9. The gesture interface system according to claim 8, wherein the processing means operates to determine an identifier of the wand based on the information encoded in the barcode area of the wand.   The second camera that is operated by the user and operates to create an image of the wand, further comprising the second camera arranged at a position different from the at least one camera. The gesture interface system described in 1.   The processing means operates to determine the posture of the wand by reconstructing the scan line of the wand and determining an intersection of the scan line of the wand and the at least one feature of the wand. The gesture interface system according to claim 1.   2. The wand according to claim 1, wherein one end of the wand is provided with a color code, and the processing means operates to determine the posture of the wand using the color code when only a part of the wand is visible. Gesture interface system.   The gesture interface system according to claim 1, wherein the at least one camera is an infrared camera, and a surface of the wand includes an infrared reflective material or a retroreflective material.   The gesture interface system according to claim 13, wherein the infrared camera comprises at least one infrared emission source and at least one infrared filter.   The gesture interface system according to claim 1, wherein the processing means is operative to calibrate the at least one camera based on the image of the wand acquired by the at least one camera. A gesture input wand for use with a gesture interface,
a. Torsion cord area;
b. A color code area;
c. Precision feature area,
A wand for gesture input that has a feature on the appearance pattern.   17. The gesture input wand of claim 16, wherein the color code area comprises a plurality of color bands distributed along the length of the wand.   The gesture input wand according to claim 17, wherein the plurality of color bands have different colors.   The gesture input wand of claim 16, wherein the twist code region comprises a spiral pattern.   The gesture input wand of claim 16, wherein the twisted code region comprises a plurality of helical patterns distributed along the length of the wand.   The gesture input wand of claim 16, wherein the precision feature region comprises a checkerboard pattern.   The gesture input wand of claim 21, wherein the checkerboard pattern comprises a dot or blurred blob in the center of the checkerboard grid.   The gesture input wand according to claim 16, further comprising a bar code area including encoding information.   The gesture input wand of claim 23, wherein the encoded information comprises an identifier of the wand.   17. The gesture input wand of claim 16, wherein at least one end of the wand comprises a color code.   17. The gesture input wand of claim 16, wherein the surface of the wand comprises an infrared reflective material or a retroreflective material. An application control method for controlling a computer application based on a posture of a wand for gesture input operated by a user and having at least one feature in an appearance pattern,
a. Creating an image of a wand operated by the user using at least one camera;
b. Determining the position of the wand, including the position of the wand, the orientation of the wand, and the twist of the wand, based on the at least one characteristic of the wand using the created image;
c. Controlling the computer application based on the determined attitude of the wand;
An application control method comprising: A camera calibration method for calibrating a camera based on an image of a wand for gesture input operated by a user and having at least one feature in an appearance pattern,
a. Creating an image of a wand operated by the user using at least one camera and controlling the computer application based on the created image;
b. Calibrating the at least one camera based on the image of the wand acquired by the at least one camera;
Including a camera calibration method. A control program for controlling a computer application based on a posture of a wand for gesture input that is operated by a user and has at least one feature on an appearance pattern,
By computer
a. Obtaining an image of the wand created by at least one camera;
b. Determining the position of the wand, including the position of the wand, the orientation of the wand, and the twist of the wand, based on the at least one feature of the wand using the created image;
c. Controlling the computer application based on the determined attitude of the wand;
Control program to execute

Images