JP4538610B2 - Information input / output system - Google Patents

Description

  The present invention relates to an information input / output system that provides a user interface environment that allows a user to operate a computer intuitively and easily using physical objects and their movements.

  In order to operate a computer, a WIMP (Windows (registered trademark) -Icons-Menus-Pointer) type interface has become widespread. However, as the display becomes larger, multi-functional, and multifunctional, Increases and misoperations are a problem. The tangible user interface (TUI, Non-Patent Document 1) is considered to be one of the post-WIMP type interfaces because an intuitive operation is possible by combining a real object and a digital expression.

  The inventors previously proposed a tangible table top interface (TTT) as a form of TUI in Non-Patent Document 2. This is because a large LCD with an ultrasonic surface acoustic wave (SAW) touch panel is used as a tabletop display, and a plurality of ultrasonic transmitters for three-dimensional measurement placed on the tabletop display are called “tags”. Used as a real object.

  The three-dimensional position of the tag is transmitted from the ultrasonic transmitter to a plurality of ultrasonic receivers installed above the display, and the distance between the tag and each receiver is determined using the ultrasonic flight time. It can be measured and measured using the principle of multi-sided surveying.

  Each tag is associated with a GUI object such as an operator window or a manual window. When the tag is moved, the GUI object also moves following the tag. TTT features such as affordance of tags, intuitive tag gestures (issue commands according to how tags are moved), and asymmetric two-handed operation using tags and touch panels allow users to use a direct and intuitive input / output system It is possible to provide to.

Other methods for measuring the position of a real object such as a tag include a method of measuring the position of an object using an RFID tag (Patent Document 1, Non-Patent Document 3), a marker attached to the object, and a camera. There is also a method (Non-patent document 4, Non-patent document 5) for measuring the movement of an object from a taken image.

  In the case of Patent Document 2 and Patent Document 3, in the rear projection display, a camera is installed on the back surface of the screen like the projector, and the real object on the display is recognized by the camera. As a recognition method, a method of adding an identification code to a real object, reading the code with a camera, irradiating infrared light from the back of the screen, and measuring the reflected light with the camera is considered.

Further, a measurement object with a built-in optical sensor is irradiated with a pattern (marker pattern) whose luminance changes temporally or spatially from a display device such as a projector or a liquid crystal display, based on the output of the optical sensor. Various studies have also been conducted on methods for measuring the position and rotation of real objects that incorporate a group of optical sensors. In Non-Patent Document 6, measurement of the position and rotation of an actual object in which the optical sensor group is incorporated is performed using an optical sensor group including five optical sensors and a marker pattern displayed on the display device. Yes.
JP 2002-132446 A JP 2005-165864 A JP 2005-317032 A H. Ishii and B. Ullmer. Tangible Bits: Towards seamless interfaces between people, bits andatoms.In Proc.ACM CHI'97, pages234-241, 1997 Takeshi Kurata, Takahiro Oyabu, Nobuchika Sakata, Masakatsu Kourogi, and Hideaki Kuzuoka, "Tangible Tabletop Interface for an Expert to Collaborate with Remote FieldWorkers", In Proc. 1st International Conference on Collaboration Technology (CollabTech2005) in Tokyo, Japan, pp.58- 63 (2005) J. Rekimoto, B. Ullmer, and H. Oba. DataTiles: A modular platform for mixed physical and graphic interactions. In Proc. ACM CHI2001, pages 269-276, 2001. Jun Rekimoto, "Matrix: A Realitime Object Identification and Registration Method for Augmented Reality", In Proc. APCHI'98, pp. 63-68, 1998 H. Kato and M. Billinghurst. Marker tracking and HMD calibration for a video-based augmentedreality conferencing system.In Proc.IWAR99, pp.85-94, 1999. Maki Sugimoto, Kaoru Kojima, Kyohiro Nakamura, Hideaki Arai, Masahiko Inami, “Position and Posture Control Using Index Images Displayed on an Image Presentation Device”, Transactions of the Virtual Reality Society of Japan vol.10 No.4, pp.485 -493, 2005

  In the case of the tag position measurement method using the ultrasonic transmitter / receiver for 3D measurement described in Non-Patent Document 2, the tag posture cannot be measured. It was. In addition, there are situations in which position measurement becomes difficult due to disturbances such as shielding caused by a user's hand entering between the transmitter and the receiver, and disturbance such as multipath due to reflection of ultrasonic waves.

  The methods shown in Non-Patent Document 4 and Non-Patent Document 5 also have a problem that measurement cannot be performed if there is an obstacle between the marker on the real object and the camera. It is inherently difficult to apply to a covering system.

In the case of the real object position acquisition method using the RFID tag described in Patent Document 1 or Non-Patent Document 3, it is realized by arranging sensors that read the RFID tag on the surface of the flat display at regular intervals. However, in this method, since the RFID tag is recognized only around the position where the sensor for reading the RFID tag is arranged, when the RFID tag sensor is regarded as a positioning means, only spatially discrete positioning can be performed. Become. That is, there is a problem in increasing the positioning resolution, and it is difficult to apply to an interface that requires moving the tag to an arbitrary position on the display.

  Further, as described in Non-Patent Document 6, the position / rotation of an actual object in which the optical sensor group is incorporated is measured using five optical sensor groups and a marker pattern displayed on the display device. In such a case, a flat display such as a liquid crystal or plasma is used as a display device instead of a projector, and the light sensor group is used so as to be in direct contact with the display screen of the display device. Can be avoided. Further, there is no need to install a special device such as an ultrasonic receiver or a camera above the display device, so that there is an advantage that the system configuration can be simplified.

  However, in this case, four optical disturbances due to the amount of parallel movement in the biaxial direction on the display screen of the display device, rotation around the axis orthogonal to the display screen, illumination other than the display device, etc. are obtained using five optical sensors. Calculated and redundant. As a result, there is a problem in that the minimum area of the surface on the real object in which the optical sensor group is incorporated becomes larger than necessary. Further, if the optical sensor group is separated from the display screen, measurement cannot be performed.

  In facilities such as science and technology exhibition halls and museums, there are often cases where display devices are ubiquitous for presenting information such as explanations and explanations of exhibits. The content of information presented on each display device should be changed according to the degree of understanding and direction of interest for each user. In practice, however, input interfaces such as touch panels, buttons, mice, and keyboards are used. In many cases, the user selects the desired information interactively. Therefore, the time cost until the information desired by the user is obtained is high, and it is necessary to become familiar with the operation technique of different input interfaces on the spot, and those input interfaces are unspecified number of users. Because it touches, it can never be said to be clean.

  The present invention has been made in view of the above circumstances, and in the present invention, an actual object necessary for realizing a general-purpose TUI (an actual object corresponding to a sensor tag in an embodiment of the present invention described later). A method for measuring the position and orientation of the body is provided using an optical sensor group including four optical sensors and a marker pattern displayed on a display device. Further, by using the acceleration sensor and the gyro sensor together, it is possible to continuously measure the position and orientation even in the situation where the optical sensor group does not exist on the marker pattern. Furthermore, it is possible to provide an information providing service according to the user without touching the display devices installed at a plurality of places where an unspecified number of users may be in contact with bare hands.

  Specifically, an object of the present invention is to provide an information input / output system capable of realizing a user interface environment in which a user can operate a computer intuitively and easily using a physical object and its movement. .

In order to achieve the above object, an information input / output system according to the present invention has, as a basic structure, one or a plurality of optical sensor groups each including four optical sensors and a wireless communication device. A display device for displaying a marker pattern on the display screen corresponding to the sensor tag when the user tag processing is instructed by touching or approaching the sensor tag to the display screen and instructing user interface processing. An information input / output system including an information processing system that receives information on a tag operation in which the sensor tag is moved through the stationary communication device and executes the user interface processing via the stationary communication device And the information processing system has a bottom surface of the sensor tag in contact with the display screen of the display device or In Chikashi was state, the brightness of the marker pattern displayed on the display screen by the optical sensors of the sensor tag is measured, the position of the display screen of the sensor tag based on the measurement values of the resulting brightness and Display Sensor tag position rotation calculation means for calculating rotation information about an axis orthogonal to the screen, and the type, position, orientation, or the type of marker pattern displayed on the display screen of the display device according to the result of the sensor tag position rotation calculation means Marker pattern display control means for changing the combination of the above and information processing means for performing user interface processing based on sensor tag position and rotation information by the sensor tag position rotation detection means.

  As one aspect of the present invention, in the information input / output system according to the present invention, the four optical sensors of the optical sensor group are arranged at positions where each optical sensor is a vertex of a square, and the marker pattern is In the square arrangement of the optical sensors, two optical sensors arranged on one diagonal of the square are referred to as an optical sensor pair A, and the remaining two optical sensors arranged on the other diagonal are referred to as an optical sensor pair B. In this case, one type of the marker pattern is divided into four regions so as to correspond to the respective photosensors, and the brightness of the two regions corresponding to the photosensor pair A corresponds to the same axial direction. The brightness of the two areas changes linearly in the opposite direction, and the brightness of the remaining two areas corresponding to the photosensor pair B is in the axial direction perpendicular to the axis where the brightness of the two areas corresponding to the photosensor pair A changes. Brightness of the second region is changed linearly in opposite directions I, configured as brightness change of the marker pattern is a marker pattern not rotationally symmetric.

  As another aspect, in the information input / output system according to the present invention, the sensor tag includes an acceleration sensor, a gyro sensor, or both in addition to the optical sensor group as a sensor. The information processing system further includes: A hybrid sensor tag position / orientation calculation unit is provided for calculating the position and orientation of the sensor tag using the measurement value obtained from the acceleration sensor and / or the gyro sensor and the measurement value obtained from the optical sensor group. .

Thus, the situation where the sensor tag is away from the display screen of the display device, the situation where the sensor tag is moving at high speed, the position of the sensor tag or the axis around the axis orthogonal to the display screen only with the sensor tag position rotation calculation means. situation which can not accurately calculate the rotation, even in such situations to be acquired rotation information about the axis other than the axis perpendicular to the display screen, the position and rotation information of the sensor tag is Ru obtained, the information processing means, the resulting Based on the position and rotation information of the sensor tag, it is possible to appropriately perform user interface processing.

  As another aspect, in the information input / output system according to the present invention, the sensor tag includes a photosensor group equivalent to the photosensor group on a surface other than the bottom surface, and the information processing system includes any surface of the sensor tag. Sensor tag surface recognizing means for recognizing whether or not is in contact with the display screen, and the sensor tag surface recognizing means switches a signal for signal processing to a signal from the optical sensor group in contact with the display screen. Configured.

  In addition, regarding the case where the display devices are ubiquitous, in the information input / output system according to the present invention, a plurality of the display devices and a plurality of the installed wireless communication devices are installed in pairs at a plurality of locations, respectively. The processing system further includes wireless communication including a tag ID for identifying the sensor tag between the wireless communication device of the sensor tag assigned to each user and the stationary wireless communication device paired with the display device. Detecting which display device the user's sensor tag is approaching, and controlling the tag tag of the approaching sensor tag, controlling the information processing means and the marker pattern display control means, Information corresponding to the user carrying the sensor tag indicated by the ID and the marker pattern are displayed close to the sensor tag. Display device installed at a plurality of locations where an unspecified number of users may come into contact based on information output from the sensor tag position rotation calculating means and the hybrid sensor tag position / orientation calculating means And a custom information dialogue means for providing an information providing service according to the user without touching it with bare hands.

  According to the information input / output system of the present invention configured as described above, when a technique for measuring the position and orientation of a sensor tag is provided using a marker pattern displayed on an optical sensor group and a display device, Although an optical sensor group consisting of five optical sensors is necessary, the present invention can be realized with an optical sensor group consisting of four optical sensors, and the effect of reducing the size and power consumption of the sensor tag can be obtained. . Further, by using the acceleration sensor and the gyro sensor together, the position and orientation can be continuously measured even in the situation where the optical sensor group does not exist on the marker pattern. When using the information provision service with display devices installed in multiple locations, it is possible to display information according to the user using the tag ID of the sensor tag, and furthermore, an unspecified number of users can contact The information providing service can be operated by using a sensor tag carried by a less user. This provides an information input / output system that can realize a user interface environment in which a user can operate a computer intuitively and easily using physical objects and their movements.

  Embodiments of the information input / output system of the present invention will be described below with reference to the drawings. FIG. 1 is a diagram for explaining the system configuration of an information input / output system according to the present invention. As a typical embodiment, a schematic system configuration centering on an input / output operation unit is shown. In FIG. 1, 1 is a user using a user interface environment, 10 is a sensor tag of a physical object operated by the user, 20 is a display device installed so that a display screen 21 is horizontal, and 30 is a display device 20. This is a stationary wireless communication device that is incorporated in the peripheral portion of the wireless communication device and performs wireless communication with the sensor tag 10. Of course, the installation type wireless communication device 30 does not need to be incorporated in the display device 20, and may be installed in a state of being close to the display device 20. The stationary wireless communication device 30 performs wireless communication with a wireless communication device (31: FIG. 4) incorporated in the sensor tag 10 to perform data communication. Further, as the display device 20, the one in which the touch panel 26 is incorporated is used, but when the position where the sensor tag 10 is placed is obtained by the sensor tag position rotation calculation unit 201 as described later, The touch panel 26 may not be incorporated. Reference numeral 50 denotes a window screen associated with the sensor tag 10.

  In FIG. 1, reference numeral 200 denotes an information processing system. The information processing system 200 includes, as processing modules for performing information processing according to the present invention, a sensor tag position rotation calculating unit 201, a marker pattern display control unit 202, an information processing unit. Means 203 are provided.

  The sensor tag position rotation calculation means 201 of the information processing system 200 is displayed on the display screen by the optical sensor group of the sensor tag 10 as described later, with the bottom surface of the sensor tag 10 in contact with or close to the display screen 21 of the display device 20. Data processing is performed to measure the brightness of the marker pattern and calculate the position of the sensor tag 10 on the display screen 21 and the rotation information about the axis orthogonal to the display screen based on the measured brightness value. In addition, the marker pattern display control unit 202 performs a process of changing the type, position, orientation, or combination of marker patterns displayed on the display screen 21 in accordance with the data processing result by the sensor tag position rotation calculation unit 201. The information processing unit 203 performs user interface processing in response to a user input / output operation instruction such as an operation on the window screen 50 based on the sensor tag position and rotation information by the sensor tag position rotation detection unit 201.

  In an embodiment of the information input / output system according to the present invention, as a basic configuration of the system, as will be described in detail later, one or a plurality of optical sensor groups each including four optical sensors are provided on the bottom surface and a wireless communication device is provided. An information processing system 200 that executes data processing related to user input / output operations including individual sensor tags 10, display devices 20, installed wireless communication devices 30 that communicate with the wireless communication devices of the sensor tags 10, and user interface processing. In addition, in the information processing system 200, the processing modules of the sensor tag position rotation calculation unit 201, the marker pattern display control unit 202, and the information processing unit 203 are configured as a system that performs the above-described data processing.

  2-4 is a figure explaining the structure of the display apparatus 20 and the sensor tag 10 in detail. FIG. 2 is an enlarged view of the display device 20, and FIG. 3 illustrates the display device 20 in a state where the sensor tag 10 placed on the display screen 21 of the display device 20 is removed. A marker pattern 40 and a window screen 50 displayed on the display screen 21 are shown. As can be understood from FIGS. 2 and 3, the marker pattern 40 displayed by the data processing of the marker pattern display control unit 202 is displayed below the sensor tag 10. The sensor tag 10 is placed on the marker pattern 40. Each sensor tag 10 is associated with a window screen 50, and user interface processing for the corresponding window screen 50 is executed by an operation instructing user interface processing such as movement operation and rotation operation of the sensor tag 10 by the user 1. The

  FIG. 4 is a diagram illustrating the structure of the sensor tag 10. As shown in FIG. 4, the sensor tag 10 here has a hexahedral cube structure, but the optical sensor group 12 of the four optical sensors 11 is provided on the bottom surface in contact with the marker pattern 10 on the display screen 21. Any shape may be used as long as the structure is the same.

  As shown in FIG. 4, the sensor tag 10 includes component parts such as an optical sensor group 12 including four optical sensors 11, a wireless communication device 31, an acceleration sensor 70, and a gyro sensor 71. A microcomputer (not shown) that functions as a control device for controlling the component parts is built in. The sensor tag 10 is displayed on the display screen 21 in a state where the bottom surface of the sensor tag 10 is in contact with or close to the display screen 21 of the display device 20 by the optical sensor group 12 including the four optical sensors 11 provided on the bottom surface. The measured brightness of the marker pattern 40 is measured, the wireless communication device 31 is controlled by the control of the microcomputer of the control device, and the measured data is transmitted to the information processing system 200 via the stationary wireless communication device 30. . Similarly, the acceleration sensor 70 and the gyro sensor 71 detect position information and rotation information in the sensor tag 10, and similarly control the wireless communication device 31 under the control of the microcomputer of the control device, so that the installation type wireless communication is performed. The measured data is transmitted to the information processing system 200 via the device 30.

  As will be described later as another embodiment, the position of the sensor tag 10 on the display screen 21 based on the measured brightness value obtained by the optical sensor group 12 using the data processing function of the microcomputer of the control device. The rotation information about the axis orthogonal to the display screen 21 may be calculated.

  In an information input / output system of another embodiment, as shown in FIG. 19, an information processing system 300 that performs data processing is provided. This information processing system 300 may be incorporated in the same housing as the display device 20 and the installation type wireless communication device 30, or may use a computer resource that can be accessed via a network. Of course, the sensor tag itself can also be used as a computer resource that can be accessed via a wireless network, and the sensor tag position rotation calculating means 100 may be provided in the microcomputer of the control device built in the sensor tag 10.

  In the information input / output system of another embodiment, as will be described later, using the information processing system 300, the sensor tag position rotation calculating means 100, the marker pattern display control means 101, the hybrid sensor tag position / orientation calculating means 102, the sensor tag ID. The functions of the acquisition unit 103, the custom information dialogue unit 104, and the sensor tag surface recognition unit 105 are realized.

  What is important here is that the optical sensor group 12 measures the brightness of the marker pattern 40 displayed on the display screen 21 in order to execute the data processing of the sensor tag position rotation calculation means 100, and the sensor tag position rotation calculation means 100. Then, as will be described later, the position of the sensor tag 10 in the display screen 21 and the rotation information about the axis orthogonal to the display screen 21 can be calculated, and the marker pattern display control means 101 is calculated using the calculated result. Thus, the position and orientation of the marker pattern 40 can be updated and changed.

  By such an information input / output system, the position of the window screen 50 displayed on the display screen 21 of the display device 20 can be changed without using a mouse or a keyboard, or data can be copied and pasted according to the positional relationship of a plurality of sensor tags. The user interface processing can be executed. In addition, by using the plurality of sensor tags 10 and the touch panel 26 in combination, an intuitive operation technique using both hands as shown in Non-Patent Document 2 can be realized.

  Next, the arrangement of the optical sensor group 12 and the marker pattern 40 will be described. In order to facilitate understanding, first, an example of Non-Patent Document 6 of the conventional method will be outlined.

  FIG. 5 is a diagram for explaining an example of a combination of a photosensor group 15 constituted by five photosensors and a marker pattern 41 according to a conventional method. In the conventional method, as shown in FIG. 5, the position / orientation of the sensor tag is measured using an optical sensor group 15 including five optical sensors and a marker pattern 41. Here, the optical sensor group 15 shown in FIG. 5 represents the arrangement of the optical sensors p1 to p5 on the bottom surface of the sensor tag. Further, only the marker pattern 41 is shown in FIG.

  These five optical sensors p1 to p5 are arranged in a cross shape at a constant interval d. In the marker pattern 41 here, as shown in FIG. 6, the brightness of the center circle area is an intermediate value between the maximum and minimum values of the brightness and darkness of the marker pattern, and other peripheral areas excluding the center circle area In the case of the marker pattern 41, the brightness changes linearly in the x-axis or y-axis direction, and the brightness change is rotationally symmetric about the center optical sensor p5. Yes.

  As shown in FIG. 5, the radius of the marker pattern is 1.5d. Each of the optical sensors p1 to p5 can move within a circle having a diameter d as shown in FIG. If it moves more than this, the optical sensors p1 to p4 protrude outside the marker pattern 41, so that measurement becomes impossible.

  In the case of this conventional method, when the optical sensor group 15 is translated on the marker pattern 41 from the position shown in FIG. 5, the translation amount is calculated from the output of the optical sensor group 15 by the following calculation.

ここで、

となる。外乱光による輝度の変化が一様ならば、ｌ_ｎ３−ｌ_ｎ２≒０，ｌ_ｎ４−ｌ_ｎ１≒０ なので、

として、平行移動による出力の変化分をとりだすことができる。 Specifically, here, attention is first focused on a change in the y-axis direction. The amount of parallel movement of the optical sensor group is ty. Since the luminances of the optical sensors p1 and p4 change in proportion to ty, changes corresponding to ty appear in their outputs l ₁ and l ₄ . _Let this output change amount be l _ty . Similarly, the outputs of the optical sensors p2 and p3 change by l _ty in accordance with the parallel movement amount tx in the x-axis direction. A change amount of each optical sensor output due to a change in luminance caused by disturbance light is defined as l _ni (i = 1,..., 5). Outputs l _{1 to} l ₅ of the optical sensors p _{1 to} p ₅ are expressed as follows.

here,

It becomes. If the change in luminance due to ambient light is uniform, l _n3 −l _{n2 ≈0} and l _n4 −l _{n1 ≈0} .

As shown in FIG.

にわたって変化するため、

として、光センサの出力から平行移動量ｔｘ，ｔｙを求めることができる。 Assuming that the distance between the photosensors is d, as shown in FIG.

Change over time,

As a result, the parallel movement amounts tx and ty can be obtained from the output of the optical sensor.

外乱が一様であるならば、各光センサｐ１〜ｐ５の出力から

として、回転による出力変化を取り出すことができる。最終的に、平行移動の計測と同様に考えると、

となり、光センサ群１５の回転量が求められる。 Next, calculation of the rotation amount by the conventional method will be described. When the optical sensor group 15 rotates θ (rad) from the state of FIG. 5, the output of each sensor changes by the same amount as when translated by rot (= d · sin θ). At this time, the outputs l _{1 to} l ₅ of the optical sensors p _{1 to} p ₅ are expressed by the following equations.

If the disturbance is uniform, from the output of each photosensor p1-p5

The output change due to rotation can be taken out as follows. Finally, considering the same as measuring parallel movement,

Thus, the rotation amount of the optical sensor group 15 is obtained.

  Further, as shown in FIG. 8, when the parallel movement and the rotation occur simultaneously, the parallel movement amount and the rotation amount can be stopped independently by the following calculation process.

このとき外乱が一様であるならば、

として、マーカパターン４１上にある光センサ群１５の平行移動量、回転量を独立に求めることができる。 First, outputs l _{1 to} l ₅ of the optical sensors p _{1 to} p ₅ are expressed as follows.

If the disturbance is uniform at this time,

As described above, the parallel movement amount and rotation amount of the optical sensor group 15 on the marker pattern 41 can be obtained independently.

  In the conventional method, four variables of tx, ty, θ, and disturbance n are calculated using the five optical sensors p1 to p5, and there should be redundancy originally. However, since the brightness change of the marker pattern 41 is rotationally symmetric, five optical sensors p1 to p5 are required.

  In the present invention, a combination of the optical sensor group 12 composed of four optical sensors and the marker pattern 40 devised for the pattern is used so that the number of optical sensors used can be reduced to four and equivalent measurement can be performed. 9 and 10 show examples of the arrangement of the optical sensors and the marker pattern according to the present invention. 11 and 12 show the marker pattern 42 and the marker pattern 43 shown in FIGS. 9 and 10, respectively. Note that the two dotted lines shown in FIG. 12 are for illustrating that the marker pattern 43 is divided into four regions, and the dotted lines themselves are not included in the marker pattern.

  In the marker pattern 42 shown in FIG. 9, since the measurement result of the optical sensor p3 directly represents the disturbance n, the measurement error greatly affects the position / posture measurement error. However, the marker pattern 43 as shown in FIG. By using, more stable measurement can be performed. The feature of this marker pattern 43 is that when the optical sensors p1 and p3 are the optical sensor pair A and the optical sensors p2 and p4 are the optical sensor pair B, the brightness of the two areas corresponding to the optical sensor pair A is along the y-axis direction. The brightness of the two areas changes linearly in the opposite direction, the brightness of the remaining two areas corresponding to the photosensor pair B changes linearly in the opposite direction along the x-axis direction, and the marker pattern The lightness of 43 is configured not to be rotationally symmetric.

  When the marker pattern 43 and the optical sensor group 16 are used, the following calculation is performed by the sensor tag position rotation calculation unit 202 (FIG. 1) or the sensor tag position rotation calculation unit 100 (FIG. 19), whereby the optical sensor group 16 (sensor tag ) Movement can be measured.

ここで外乱が一様であれば、

として出力変化分が算出される。

より、以下のように平行移動量と回転量を求めることができる。

FIG. 13 is a diagram for explaining the size comparison between the marker pattern 43 and the marker pattern 41 used in the conventional method. When each optical sensor of the optical sensor group 16 composed of four optical sensors is within the measurement range 60 shown on the right in FIG. 13, the output of each sensor is as follows.

If the disturbance is uniform,

As shown in FIG.

Thus, the parallel movement amount and the rotation amount can be obtained as follows.

  As shown in FIG. 13, in the case of the marker pattern 43 (the same applies to the marker pattern 42), the area of the marker pattern is set to 65% and the optical sensors p1 to p4 are compared with the marker pattern 41 used in the conventional method. The area of the squares 17 (FIG. 10) that are the vertices can be 50%. This means that the sensor tag incorporating the optical sensor group can be reduced in size.

  Next, a rectangular marker pattern specialized for calculating the parallel movement amount will be described. The circular marker pattern 42 and the marker pattern 43 described so far have a relatively narrow range in which the optical sensor group 16 can translate. Thus, in another embodiment of the present invention, the problem is solved by using a marker pattern that allows parallel translation at a high speed in combination with a circular marker pattern.

  FIG. 14 is a diagram for explaining a marker pattern that allows parallel translation at high speed. In FIG. 14, an example in which the position of the optical sensor group is calculated by the sensor tag position rotation calculating unit 201 while the circular marker pattern 43 and the rectangular marker pattern 44 are selected and displayed by the marker pattern display control unit 202. Is shown. By expanding and contracting so that the brightness changes linearly in the axial direction along the moving direction of the optical sensor group 16 (sensor tag), the measurement range in that direction can be expanded, although the measurement accuracy is sacrificed.

  However, since the amount of movement in the direction in which the lightness does not change cannot be measured with the rectangular marker pattern 44, it is necessary to match the lightness change direction in the marker pattern 44 with the movement direction of the optical sensor group 16. Therefore, as will be described later, the hybrid sensor tag position / orientation calculation means 102 is used. By the hybrid sensor tag position / orientation calculation means 102, the movement direction of the optical sensor group 16 is the transition direction estimated from the acceleration sensor 70 incorporated in the sensor tag 10 including the optical sensor group 16, or the situation where the optical sensor group 16 is moving at a low speed. The marker pattern display control means 202 determines the type, size, position, orientation, and the like of the marker pattern, and the marker pattern corresponding to the display screen 21 is estimated from the movement direction calculated using the circular marker pattern 43 in FIG. Is displayed.

ただし、

である。 When the rectangular marker pattern 44 is used, the parallel movement amount t can be calculated using the width w of the rectangular marker pattern 44 as follows.

However,

It is.

  FIG. 15 shows an embodiment in which the position of the optical sensor group 16 is calculated using the time series marker pattern 45. The marker pattern display control means 202 performs a wide area search of the optical sensor group 16 by narrowing the search range while changing the position and size of the marker pattern and the direction of light and dark in time series as shown in FIG. Can do. Such a wide area search is effective when an initial search for a sensor tag or when the information input / output system loses sight of the sensor tag in the middle.

Next, an embodiment of calculation of the position and orientation of the sensor tag by the hybrid sensor tag position and orientation calculation means 102 using the optical sensor group and the acceleration sensor will be described. The relative change of the three-dimensional position is obtained by double integration of acceleration obtained by the triaxial acceleration sensor 70 built in the sensor tag 10. However, since errors accumulate, high accuracy cannot be obtained particularly when measuring long-term movement at low speed. On the other hand, in the case of the sensor tag position rotation calculation means 100 using the optical sensor group 12 and the marker pattern 43 described above, the measurement accuracy at low speed is high. Thus, by using both sensors together, as shown in FIG.
(A) Medium / low speed movement,
(B) high speed movement,
(C) three-dimensional movement,
Thus, a wide range of typical operation patterns of the sensor tag 10 can be handled.

  FIG. 16 is a diagram for explaining the correspondence in several typical operation patterns of the sensor tag 10 of (a) medium-low speed movement, (b) high-speed movement, and (c) three-dimensional movement. For example, in the situation as shown in FIG. 16A, the position and rotation of the sensor tag are measured using only the circular marker pattern 43. In the situation like the movement operation (b1) of the sensor tag in FIG. 16 (b), it is determined that the sensor tag 10 is moving at a relatively high speed based on the previous measurement result. Therefore, based on the predicted value from each sensor. A wide range of position measurements is performed using a rectangular marker pattern 44 whose position, orientation, and size are determined. In this case, if it is determined that the robot is moving at a low speed, the circular marker pattern 43 can be used again as in the sensor tag moving operation (b2) in FIG.

  Further, in the situation as shown in FIG. 16C, the accelerometer 70 is used for position prediction of the sensor tag 10 and measurement of the position and orientation in the air. As shown in FIG. 16C, when the sensor tag is grounded again on the display screen 21 after three-dimensionally moving, when searching for an initial position, or when the sensor tag 10 is out of the marker pattern. Performs a wide area search of the sensor tag 10 using the time-series rectangular marker pattern 45. Thereby, a sensor tag can be searched when an information input / output system loses sight of a sensor tag on the way.

  As described above, the sensor tag 10 can detect the position and orientation of the sensor tag by incorporating the gyro sensor 71 in the sensor tag 10. In this case, all three-axis rotations may be measured by a gyro, but by calculating inclination information (Roll, Pitch) from acceleration obtained by an accelerometer, other than rotation around the axis along the direction of gravity (Yaw) Therefore, only the Yaw rotation may be measured with a gyro.

  FIG. 17 shows another embodiment of the sensor tag. In this case, the sensor tag 110 shown in FIG. 17 includes the optical sensor group 13 and the optical sensor group 14 on different surfaces in addition to the optical sensor group 12 on the bottom surface.

  As described with reference to FIG. 4, the acceleration sensor 70 and the gyro sensor 71 may be incorporated in the sensor tag 110. In this case, as shown in FIG. 19, the sensor tag surface recognition means 105 provided in the information processing system 300 uses, for example, the measurement values of the optical sensor groups 12, 13, and 14 provided in the sensor tag 110, It is determined on which surface of the sensor tag 110 the optical sensor groups 12, 13, and 14 measure the marker pattern. Further, by measuring the motion of the sensor tag using the acceleration sensor 70 or the gyro sensor 71, the orientation of the sensor tag 110 can be acquired. Furthermore, the direction of the sensor tag may be acquired by using the optical sensor group and information such as an acceleration sensor and a gyro sensor together.

  The sensor tag surface recognition means 105 is mounted on the information processing system 300. As described above, the sensor tag 110 itself also has a built-in microcomputer as a control device for controlling the operation. Since it can be regarded as a part of the processing system, the sensor tag surface recognition means 105 may be mounted on the sensor tag 110 side or may be mounted on a computer resource on the display device 20 side.

  By adopting such an embodiment, it is possible to obtain information on which surface of the sensor tag 110 is in contact with the display screen 21 and how it moves, so that more various operation techniques can be provided to the user. Can do. For example, when the surface of the optical sensor group 12 is down, rotation of the sensor tag around an axis orthogonal to the display screen changes the volume of music, and when the surface of the optical sensor group 13 is down, user interface processing such as switching music is performed. The operation technique can be provided.

  FIG. 18 shows a case where a display device is ubiquitous like the display device 20 or the display device 22 in a facility such as a science and technology exhibition hall or a museum for presenting information such as explanations and explanations of exhibits. 1 shows an exemplary embodiment of an information input / output system according to the present invention.

  In this embodiment, there are two users, user 1 and user 2, each carrying a sensor tag 10, and in the information input / output system of the present invention, which user carries which sensor tag. Know in advance.

  For example, in the display device 20, the user 1 and the user 2 are approaching the display device 20 using each sensor tag 10. Each of the sensor tags 10 communicates with the stationary wireless communication device 30 paired with the display device 20, and the users 1 and 2 are approaching the display device 20 by transmitting the sensor tag ID from the sensor tag 10. Can be known by the system (sensor tag ID acquisition means 103). As a result, an information providing service suitable for each of the user 1 and the user 2 can be displayed on the display device 20, and each of the users 1 and 2 can provide the information providing service using the sensor tag possessed by the user. Can be operated (custom information interaction means 104).

  In the example of FIG. 18, the user 2 approaches the display device 22 after approaching the display device 20. Also here, the stationary wireless communication device 32 paired with the display device 22 and the wireless communication device 31 of the sensor tag 10 perform wireless communication of information including the sensor tag ID, and the sensor tag ID acquisition unit 103 causes each sensor tag 10 to By being identified, information corresponding to each user can be provided by the custom information interaction means 104.

  In other words, the sensor tag 10 carried by the user not only automatically selects the information service, but without touching the display devices installed in a plurality of places where an unspecified number of users may contact, The information service according to the user can be operated and used. Furthermore, when a photo sensor group is incorporated into a mobile phone, it can function as a sensor tag equipped with a wireless communication device, so a large display can be achieved by placing or contacting a mobile phone owned by a user on a display device. When an information service window specific to the user is displayed on the apparatus and the mobile phone is moved, the information service can be operated accordingly.

  Thus, by using the information input / output system according to the present invention, for example, a communication terminal using a TUI for remote cooperative work as described in Non-Patent Document 2 can be realized. In addition, in facilities where display devices are ubiquitous such as science and technology exhibition halls and museums, it is possible to apply to information presentation service systems such as explanations and explanations of exhibits.

It is a figure explaining the system configuration | structure of the information input / output system by this invention. 3 is a first diagram illustrating the structure of the display device 20 in detail. FIG. 4 is a second diagram illustrating the structure of the display device 20 in detail. FIG. It is a figure explaining the structure of the sensor tag 10 in detail. It is a figure explaining the example of the combination of the optical sensor group 15 comprised by five optical sensors by the conventional method, and the marker pattern 41. FIG. It is a figure which shows only the marker pattern 41. FIG. It is a figure explaining the range which can measure parallel movement by the combination of the optical sensor group 15 and the marker pattern 41. FIG. It is a figure explaining the independence of the parallel movement amount and rotation amount when parallel movement and rotation occur simultaneously. It is a figure which shows one example of arrangement | positioning and the marker pattern of the optical sensor group by this invention. It is a figure which shows another example of arrangement | positioning and the marker pattern of the optical sensor group by this invention. It is a figure which shows only the marker pattern 42 shown by FIG. It is a figure which shows only the marker pattern 43 shown by FIG. It is a figure which compares and demonstrates the size of the marker pattern 43 and the marker pattern 41 used by the conventional method. It is a figure explaining the example of the marker pattern which accept | permits parallel translation at high speed. It is a figure which shows one Example which calculates the position of the optical sensor group 16 using the time series marker pattern. It is a figure explaining the correspondence in some typical operation patterns of sensor tag 10 of (a) middle low speed movement, (b) high speed movement, and (c) three-dimensional movement. It is a figure which shows another Example of a sensor tag. Information input / output system according to the present invention in a case where a display device is ubiquitous like display devices 20 and 22 for presentation of information such as explanations and explanations of exhibits in facilities such as science and technology exhibition halls and museums FIG. 6 shows a schematic diagram illustrating an exemplary embodiment of the present invention. It is a block diagram which shows the example of another system configuration | structure of an information input / output system.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 User 2 User 10 Sensor tag 11 Photosensor 12 Photosensor group 13 Photosensor group 14 Photosensor group 15 Photosensor group 16 Photosensor group 17 Square 20 Display device 21 Display screen 22 Display device (vertical type)
30 Installation Type Wireless Communication Device 31 Wireless Communication Device 32 Installation Type Wireless Communication Device (Vertical Type)
40 Marker pattern 41 Marker pattern 42 Marker pattern 43 Marker pattern 44 Rectangular marker pattern 45 Time series marker pattern 50 Window screen 60 associated with sensor tag Measurement range 70 Acceleration sensor 71 Gyro sensor 100 Sensor tag position rotation calculation means 101 Marker pattern display control Means 102 Hybrid sensor tag position and orientation calculation means 103 Sensor tag ID acquisition means 104 Custom information dialogue means 105 Sensor tag face recognition means 110 Sensor tag 200 Information processing system 201 Sensor tag position rotation calculation means 202 Marker pattern display control means 203 Information processing means 300 Information processing system

Claims (5)

One or a plurality of sensor tags provided with a photosensor group including four photosensors on the bottom surface and a wireless communication device ; instructions for user interface processing by moving the sensor tag in contact with or approaching the display screen A display device that displays a marker pattern on a display screen corresponding to the sensor tag, a stationary wireless communication device that communicates with the wireless communication device, and an operation of moving the sensor tag via the stationary communication device An information input / output system comprising an information processing system that accepts tag operation information by executing the user interface process,
In the information processing system,
With the bottom surface of the sensor tag in contact with or close to the display screen of the display device, the brightness of the marker pattern displayed on the display screen is measured by the optical sensor group of the sensor tag, and the measured brightness value is obtained. a sensor tag position rotation calculating means for calculating the rotation information about the axis orthogonal to the position and table示画surface of the display screen of the sensor tag Te,
Marker pattern display control means for changing the type, position, orientation, or combination of marker patterns to be displayed on the display screen of the display device according to the result of the sensor tag position rotation calculation means;
An information input / output system comprising: information processing means for performing user interface processing based on sensor tag position and rotation information by the sensor tag position rotation detection means. The information input / output system according to claim 1,
The four photosensors of the photosensor group are arranged at positions where each photosensor becomes each vertex of a square,
In the square arrangement of the photosensors, the marker pattern includes two photosensors arranged on one diagonal of the square as photosensor pair A, and the remaining two photosensors arranged on the other diagonal. In the case of the optical sensor pair B, one type of the marker pattern is divided into four areas so as to correspond to the respective optical sensors, and the brightness of the two areas corresponding to the optical sensor pair A has the same axial direction. , The brightness of the two areas linearly changes in the opposite direction, and the brightness of the remaining two areas corresponding to the photosensor pair B is orthogonal to the axis where the brightness of the two areas corresponding to the photosensor pair A changes. An information input / output system, wherein the brightness of the two regions changes linearly in the opposite direction along the axial direction, and the change in brightness of the marker pattern is not rotationally symmetric. The information input / output system according to claim 1,
The sensor tag includes an acceleration sensor, a gyro sensor, or both in addition to the optical sensor group as a sensor,
The information processing system further includes:
Characterized by comprising hybrid sensor tag position / orientation calculation means for calculating the position and orientation of a sensor tag using a measurement value obtained from the acceleration sensor and / or gyro sensor and a measurement value obtained from the optical sensor group. Information input / output system. The information input / output system according to claim 1,
The sensor tag includes an optical sensor group equivalent to the optical sensor group on a surface other than the bottom surface,
The information processing system comprises sensor tag surface recognition means for recognizing which surface of the sensor tag is in contact with the display screen,
The information tag input / output system, wherein the sensor tag surface recognition means switches a signal for signal processing to a signal from a group of optical sensors in contact with the display screen. The information input / output system according to claim 3,
The plurality of display devices and the plurality of stationary wireless communication devices are installed in a plurality of locations as a pair,
In addition to the information processing system,
The sensor tag of the user by wireless communication including a tag ID for identifying the sensor tag between the wireless communication device of the sensor tag assigned to each user and the stationary wireless communication device paired with the display device Sensor tag ID acquisition means for detecting which display device is approaching and acquiring the tag ID of the approaching sensor tag;
The information processing means and the marker pattern display control means are controlled to display the information corresponding to the user carrying the sensor tag indicated by the tag ID and the marker pattern on the display device close to the sensor tag, and the sensor tag position Based on the information output from the rotation calculation means and the hybrid sensor tag position / orientation calculation means, use without touching the display devices installed at multiple locations where an unspecified number of users may be in contact with bare hands. Custom information interaction means to provide information providing services according to the person,
An information input / output system comprising:

Images