JP4328282B2 - Numerical input method and apparatus, and numerical input program - Google Patents

Description

  The present invention relates to a numerical value input method and apparatus and a numerical value input program for calculating a positional relationship between a visual code having a predetermined shape and a camera, converting the positional relationship into a numerical value, and inputting the numerical value.

Conventionally, small portable information terminals such as PDAs and mobile phones have hardware limitations. In particular, mobile phones that are widely used today have a small liquid crystal display for display, and it is difficult to display a large amount of information simultaneously. In addition, the mobile phone has a minimum number of numeric keys and small buttons having various functions, and the user must use these small buttons.
Japanese Patent Application Laid-Open No. 08-129557

  In the above-mentioned Patent Document 1 and the like, a posture detection means is added to the terminal for the purpose of solving the problem that it is difficult to operate a button with one hand with a PDA and the construction of an automatic presentation system in which the user aligns the document with the direction in which the terminal is held. However, a technique for inputting or presenting information by moving a terminal is disclosed.

  However, usually, when setting a plurality of numerical values on a small portable information terminal, most information terminals input respective numerical values by complicated button operations while switching the numerical setting target on the screen. The operation tends to be complicated. For this reason, it is difficult for a small portable information terminal to realize an application in which a plurality of contents are linked.

  For example, an application in which a small mobile information terminal is used to designate a facility on the map while displaying a map, and further designates a time and reserves the facility is used for screen switching and complicated numerical input. Since the operation must be performed many times, there is a problem that the operation is particularly complicated.

  The present invention has been made in view of the above circumstances. The visual code assigned to the paper surface is detected by image processing, the positional relationship between the terminal and the visual code is calculated, further converted into a numerical value, and the numerical value is transmitted to the terminal. There is an advantage that there is no need to add a device for posture detection because it uses a digital camera that is input or used by many existing terminals, and the purpose of the above conventional technology is to detect posture and position. It is another object of the present invention to provide a different numerical input method and apparatus and a numerical input program.

In order to achieve the above object, the present invention provides a numerical input method using a visual code having a predetermined shape and an imaging means provided in a portable device , the imaging means based on a marker at a corner of the visual code. A step of detecting a visual code area in the video by the step of detecting an identification number of the visual code in the video by the imaging means based on the color barcode in the visual code area; The visual code is image-processed, and the positional relationship between the visual code and the imaging means is calculated based on the positional relationship defined by the identification number, and the positional relationship vector indicating the positional relationship is substituted into a predetermined function to obtain a numerical value. and converting the steps of the input values of the input object which is defined the number in the identification number, the lifting It is characterized in.

  Alternatively, the positional relationship vector is a parameter indicating the inclination of the visual code with respect to the shooting direction of the imaging unit, a parameter indicating the distance between the imaging unit and the visual code, or the position of the imaging unit to the position of the visual code. It is a step in which one or a combination of parameters indicating the direction is combined.

  Alternatively, the step of converting into a numerical value may substitute the positional relationship vector and the positional relationship vector calculated a predetermined time into a predetermined function and convert it into a numerical value.

  Alternatively, the calculated numerical value or positional relationship vector is displayed as a character or a figure in a display step.

Also, a numerical input device using a visual code having a predetermined shape and an imaging means provided in a portable device, and detecting a region of the visual code in the video by the imaging means based on a marker at a corner of the visual code Means for detecting an identification number of the visual code in the video by the imaging means based on the color barcode in the area of the visual code, image processing the visual code in the video by the imaging means, and the identification number the positional relationship in defined, means for calculating a positional relationship between the visual code image pickup means, and means for converting numbers into a positional relation vector indicating the positional relationship between a predetermined function, said identifying said numerical And means for setting an input value of an input target specified by a number .

  Alternatively, the positional relationship vector is a parameter indicating the inclination of the visual code with respect to the shooting direction of the imaging unit, a parameter indicating the distance between the imaging unit and the visual code, or the position of the imaging unit to the position of the visual code. One of the parameters indicating the direction or a combination of any of the parameters is characterized.

  Alternatively, the means for converting to a numerical value substitutes the positional relationship vector and the positional relationship vector calculated before a predetermined time into a predetermined function, and converts it into a numerical value.

  Alternatively, the calculated numerical value or positional relationship vector is displayed as a character or a figure by a display means.

  Further, the step in the numerical value input method is a program for causing a computer to execute.

  As described above, according to the present invention, a numerical value is input using the positional relationship with the visual code, and further, the operation is performed while browsing the information printed on the paper or the like to which the visual code is added. In the conventional mobile phone terminal, it is possible to easily implement an application in which difficult contents are linked.

  In addition, according to the present invention, the volume can be adjusted by actually turning the mobile phone terminal, and the user can intuitively input a continuous amount rather than the operation using a conventional button, and the facility can be displayed on the map. It is possible to easily construct the system for specifying and making a reservation.

Embodiments of the present invention will be described below with reference to the drawings. The system of FIG. 1 described below in the present embodiment includes a mobile phone terminal 1 which is a portable information terminal and a visual code 20 (FIG. 2) printed on a paper surface. In this system, the user inputs a numerical value to the mobile phone terminal 1 by imaging the visual code 20 with a camera which is an imaging means of the mobile phone terminal 1 and moving the mobile phone terminal 1 with a predetermined movement. is there.
[Configuration of information terminal]
Next, the cellular phone terminal 1 used in the present embodiment will be described. In FIG. 1, a mobile phone terminal 1 includes an internal memory 2 for storing data, a wireless communication means 3 using the Internet, an infrared communication means 4 for communicating with other electronic devices, a digital camera 5 as an imaging means, an input An operation button 6 as means, a CPU 7 for performing information processing such as image processing, and a liquid crystal monitor 8 as display means are provided.

  The CPU 7 includes functions of an identification number acquisition unit 7a, a visual marker detection unit 7b, a visual code information acquisition unit 7c, and a positional relationship calculation unit 7d in order to perform information processing such as image processing.

  In FIG. 1, in addition to the above-described configuration, an external database unit 9 is provided, and the external database unit 9 stores a visual code information database 11 including visual code information 10 and the like.

[Configuration of Visual Code 20]
The visual code 20 used in this embodiment will be described with reference to FIGS. The visual code 20 is composed of a visual marker 21 composed of double concentric circles having different sizes and colors (indicated by shading in the figure), and a color barcode 22.

  The visual code 20 is printed on paper and provided to the user. The visual code 20 is a square, a visual marker 21 is disposed at each vertex, and one or more color barcodes 22 are disposed between predetermined vertices.

  The visual code 20 can arrange characters and figures that assist the user such as icons. The configurations of the X, Y, and Z axes (51, 52, 53) of the visual code 20 used in the following embodiments are set as shown in FIGS. The following three-dimensional coordinate systems are all left-handed systems.

  The visual marker 21 is composed of two circles. The ratio of the radius of the large and small circles is 1: 2. The two circles are concentric circles and are arranged so that their centers coincide. The two circles are filled with colors having different hues.

  In the present embodiment, three colors of blue, red and yellow are used for filling. The visual marker 21 can be identified using the combination of colors used for the marker. In the present embodiment, this is used as the vertex number of the visual marker 21.

  The correspondence between color combinations and vertex numbers in the present embodiment is as shown in FIG. For example, in the case of the square visual code 20 used in the present embodiment, as shown in FIG. 3, the vertex numbers: 1 to: 4 is allocated.

  The color barcode 22 arranged on the visual marker 21 uses three colors of blue, red and yellow for filling. As shown in FIG. 5, the barcode components are set such that all adjacent blocks 40 have different colors (shown in shades in FIG. 5).

Below, each process of the detection of the visual code 20, calculation of a positional relationship, and conversion to a numerical value is demonstrated concretely.
[Detection of visual code 20]
Next, in describing the detection method of the visual code 20 from the camera image and the method of extracting the identification number used in the present embodiment, a process of detecting the marker 21 from the camera image of the visual marker 21 will be described first.

  First, the CPU 7 replaces the values of pixels having hue values other than the fixed three colors with black from the camera image 60 shown in FIG. Next, the camera image is examined from the upper left corner point to the right.

  At this time, according to FIG. 4, the vertex number of the detected area can also be detected at the same time. For example, when an area in which the color is changed to red, blue, and yellow is detected, the area is a part of the visual marker 21 configured by a circle with red on the outside and blue on the inside. 4 to “6” can be detected.

  When the area that is a part of the visual marker 21 is detected, the CPU 7 writes the detected vertex number in the area. When the investigation of one line of the camera image is completed, the line moves to the lower line and the same investigation is performed.

  The same processing is performed downward from the upper left corner point. A pixel that is detected in both the right direction and the downward direction and the vertex number that has been overwritten and coincides in both directions is assumed to be an area constituting the visual marker 21, and the center of gravity of the area is detected, and the visual The position and vertex number of the marker 21 in the camera image are detected (70 in FIG. 7).

  Finally, the CPU 7 can connect the centroids of the visual codes 20 with straight lines in the order of vertex numbers, and extract the area surrounded by the straight lines as the visual code area 81 (FIG. 7) in the camera video.

  The CPU 7 acquires the vertex coordinates of the vertex number: 1 and vertex number: 2 markers in the camera image obtained by the above process, and sees them on a straight line from the vertex of vertex number: 1 to the vertex of vertex number: 2. Detects color change for each block.

  As shown in FIG. 5, when the hue difference with the adjacent block 40 is 120 °, the bit is “1”, and when the hue difference is −120 °, the bit is “0”. Bit information is extracted from the color change of the block.

  When the same color is allowed to the adjacent block 40 and the barcode is configured to read information from a predetermined position of the barcode, when the barcode is distorted in the camera image by projective transformation, the position is also affected by the distortion. It is difficult to read and read.

However, if the blocks are arranged so that the colors are always different and the change in the color is read, it is possible to accurately read even a barcode that is distorted and projected by projective transformation.
[Calculation of positional relationship]
Next, a method of calculating the positional relationship using the shape of the visual code area 81 shown in FIG. 7 in the camera image obtained by the above processing and the vertex coordinates will be described with reference to FIG.
[Camera parameter matrix]
First, a camera parameter matrix that can be used for calculation of all positional relationships will be described. As shown in FIG. 8a, it is possible to calculate the geometric relationship between the points on the camera image and the coordinates of the real world page by using the visual code 20 on one plane.

  Using this geometric relationship, the visual code 20 (FIG. 8b) in the camera image can be normalized (FIG. 8a) to the original real world plane viewed from the front. Specifically, the four corner points of the visual code 20 in the image are associated with the four corner points of the real-world visual code, and a camera parameter matrix is calculated by the following equation (1).

By using this camera parameter matrix, it is possible to calculate three-dimensional coordinates in the real world from two-dimensional coordinates in the camera video. [Reference, Junichi Kyokumoto, Augmented Reality Construction Method Using Two-Dimensional Matrix Code, WISS '96]
Using the method described in the reference, as shown in FIG. 9, the normal vector of the visual code 20 in the camera coordinate system, the vector of two orthogonal diagonal lines of the visual code 20, and the following equation (2) The position vector of the center of gravity of the visual code 20 in the camera coordinate system can be detected. Using these four vectors, the posture and position of the camera with respect to the visual code 20 can be calculated, and the matrix C can be obtained.

[Inclination around the Z axis]
A method for calculating the inclination of the visual code 20 around the Z-axis with respect to the shooting direction of the camera will be described.
[Calculation of rotation angle using matrix C]
When passing through the center of the visual code 20 and rotating only around the Z axis, the matrix C is represented by the following equation (3). From this matrix C, the rotation angle φ indicating the inclination can be calculated.

[Calculation of rotation angle using vector from center of gravity to vertex]
Since the method using the camera parameter matrix described above is complicated, the calculation cost is high. Therefore, a method with a lower calculation cost will be described below.

  As shown in FIG. 10 a, a vector A connecting these two points is calculated from the coordinates of the vertex of vertex number 1 obtained from the camera image and the coordinates of the center of gravity of the visual code 20. When the visual code 20 faces the camera, the angle formed by the vector A with the Y axis in the image is 45 ° (110).

The angle formed by the vector A obtained from the camera image with respect to the Y axis is calculated, and by subtracting 45 ° therefrom, the rotation angle around the Z axis of the visual code can be obtained as shown in FIG. ).
[Tilt around X axis]
A method for calculating the inclination of the visual code 20 around the X axis with respect to the shooting direction of the camera will be described with reference to FIGS.
[Calculate slope by side ratio]
First, according to FIG. 11 a, the inclination is calculated using the ratio of the pair of opposite sides of the visual code 20. Although this method cannot detect a precise rotation angle, it can easily detect the rotation direction and the rough size of the rotation angle.

  A case where the inclination around the X axis is calculated will be described. The ratio of the vector length L1 from the vertex of vertex number: 1 to the vertex of vertex number: 2 and the length of vector L3 from the vertex of vertex number: 3 to the vertex of vertex number: 4 is calculated (FIG. 11a). 120).

In the camera image, it is rotated around the X axis by ψ (: −180 <ψ <180). L1> L3 when ψ> 0, and L1 <L3 when ψ <0. From this, by calculating the ratio of L1 and L3, it is possible to calculate the degree of rotation around the X axis. The value of ψ increases as the value of L1 / L3 increases, and the value of ψ decreases as the value of L1 / L3 decreases.
[Calculation of rotation angle using matrix C]
Similar to the inclination around the Z axis, the rotation angle around the X axis can be strictly calculated using the matrix C. When the rotation is only about the X axis, the matrix C is as shown in the following equation (4), and ψ is calculated as the rotation angle.

[Tilt around Y axis]
Next, a method for calculating the inclination of the visual code 20 around the Y axis with respect to the shooting direction of the camera will be described with reference to FIG.
[Calculate slope by side ratio]
Similar to the rotation angle calculation method around the X axis, the inclination is calculated using the ratio of a pair of opposite sides of the visual code 20. When calculating the rotation angle around the Y axis, the vector length L2 from the vertex of vertex number: 2 to the vertex of vertex number: 3 and the vector length from the vertex of vertex number: 4 to the vertex of vertex number: 1 The ratio of the length L4 is calculated (121 in FIG. 11b).

In the camera image, it is rotated about the Y axis by θ (: −180 <θ <180). When θ> 0, L2> L4, and when θ <0, L2 <L4. From this, by calculating the ratio of L2 and L4, it is possible to calculate the degree of rotation about the Y axis. The value of θ increases as the value of L2 / L4 increases, and the value of θ decreases as the value of L2 / L4 decreases.
[Calculation of rotation angle using matrix C]
In addition, the rotation angle around the Y axis can be strictly calculated using the matrix C. At this time, the matrix C in the case of rotating only around the Y axis is represented by the following equation (5), and θ is calculated as the rotation angle.

[Vector from camera position to visual code position]
A method for calculating a vector from the camera position to the position of the visual code 20 will be described.
[Method using matrix C]
E _t of matrix C is the vector from the position of the camera to the position of the visual code 20 (FIG. 9 to the number 2).
[distance]
A method for calculating the distance between the position of the camera and the position of the visual code 20 will be described.
[Method using matrix C]
By obtaining the absolute value of the vector e _t, it is possible to calculate the distance of the terminal and the visual code 20 (FIG. 9 to the number 2).
[Calculation of distance by area]
In FIG. 12, the area 134 of the visual code area in the camera video is proportional to the square of the distance between the terminal and the visual code 20 when the terminal is facing the visual code 20. The maximum value of the area 134 of the visual code area in a state where the visual code 20 can be detected, the distance at that time, the minimum value of the area, and the distance at that time are calculated in advance. Assuming that the distance changes in proportion to the area between the maximum value and the minimum value, a function for obtaining the distance from the area is created. The distance can be calculated from the area based on this function.
[Calculation of distance using circumference length]
The calculation of the area 134 of the visual code area requires a raster scan, and thus the calculation cost tends to increase. Therefore, the circumference length 130 that can be calculated from the sum of the distances between adjacent vertices is used for distance detection. The circumference 130 is proportional to the distance between the terminal and the visual code 20.

The maximum value of the circumference length of the visual code area in a state where the visual code 20 can be detected and the distance at that time, the minimum value of the circumference length and the distance at that time are calculated in advance. Assuming that the distance linearly changes between the maximum value and the minimum value, a function for obtaining the distance from the circumference is created. Based on this function, the distance can be calculated from the circumference.
[Distance calculation using diagonal lines]
In FIG. 12, the length 131 of the diagonal line is also proportional to the distance between the terminal and the visual code 20, similar to the circumferential length 134 of the visual code area. The distance between the terminal and the visual code 20 can also be calculated using the length 131 of the diagonal line.

The maximum value of the diagonal length of the visual code area in a state where the visual code 20 can be detected and the distance at that time, the minimum value of the diagonal length and the distance at that time are calculated in advance. Assuming that the distance linearly changes between the maximum value and the minimum value, a function for determining the distance from the length of the diagonal line is created. Based on this function, the distance can be calculated from the length of the diagonal line.
[Calculation of distance by circumscribed rectangle]
In FIG. 12, the area of the circumscribed rectangle 132 is the area of the region, and the length of the side of the circumscribed rectangle is proportional to the square of the distance between the terminal and the visual code 20, or the distance as well as the diagonal and circumference of the region. . The distance between the terminal and the visual code 20 can be calculated using the length and area of these sides.

  The circumscribing rectangle 132 is a diagonal of two points: a point composed of the smallest X and Y coordinates and a point composed of the largest X and Y coordinates in the visual code area in the camera video. A rectangle with dots.

  When the visual code 20 facing the digital camera is within the angle of view and the visual code 20 can be detected, the maximum side length or area of the visual code area and the distance, side length, or The minimum value of the area and the distance at that time are calculated in advance.

Assuming that the distance linearly changes between the maximum value and the minimum value, a function for obtaining the distance from the length or area of the side of the circumscribed rectangle 132 is created. Based on this function, the distance can be calculated from the length or area of the side of the circumscribed rectangle 132.
[Deviation between the position of the visual code on the XY plane and the position of the camera]
A method for calculating the component on the XY plane of the vector from the camera position to the position of the visual code 20 will be described.
[Method using matrix C]
Of the vector e _t (x, y) to calculate the components (FIG. 9, the number 2).
[Displacement between the center of gravity of the area and the center of the camera angle of view]
In FIG. 12, by calculating the difference vector between the center of the camera image and the center of gravity 133 of the visual code area in the image, the component on the XY plane of the vector from the camera position to the position of the visual code 20 is calculated. Can be calculated.
[Conversion to numbers]
Next, a method for converting the positional relationship into a numerical value will be described.
[One-dimensional numerical value from one-dimensional positional relation vector]
When the positional relationship vector consists of one parameter, the transformation can be expressed as y = f (x), where y is defined as a numerical value and x is defined as a positional relationship. Hereinafter, the conversion method performed by the function f (x) will be specifically described.
[Maximum / minimum mapping / linear function / opposite version]
When the numerical input target has a range, it is possible to create a linear function capable of calculating a numerical value from the positional relationship by associating the maximum value and the minimum value of the positional relationship with the maximum value and the minimum value of the input target. The maximum and minimum values of the positional relationship in a state where the visual code 20 can be detected are calculated, the range of the values is calculated in advance, and the linear function is used for the f (x). When the maximum value of the calculated distance is 100, the minimum value is 20, and the range of values to be numerically input is 0 to 127, f (x) = 127 ^* (x / 80-1 / 4). .

Using this linear function, the positional relationship can be converted into a numerical value. It is also possible to create and convert a linear function in which the maximum value of the positional relationship is associated with the minimum value of the numerical input target and the minimum value of the positional relationship is associated with the maximum value of the numerical input target.
[Maximum and minimum correspondence, n-order function, opposite version]
It is also possible to extend the f (x) to an nth order polynomial and convert the positional relationship into a numerical value.
[Logarithm / exponential function]
It is also possible to use a logarithmic function or an exponential function for the f (x). For example, when the distance is used as the positional relationship, a subtle change in the positional relationship can be easily converted as it is closer, and the positional relationship can be roughly converted into a numerical value as it is farther away.
[Discrete operation by Gaussian function]
It is also possible to use a Gaussian function for f (x). The Gaussian function is represented by y = [x] and returns an integer value not exceeding it. For example, [1.45] = 1 and [−2.1] = − 3. By using this for the conversion from the positional relationship to the numerical value, the continuous amount of the positional relationship can be easily converted to discrete menu selection or the like.

y = n ^* [x] (n is an arbitrary real number). By multiplying a Gaussian function by a real number as in this equation, the amount of positional relationship required when moving one menu item can be adjusted.
[Discrete operations using thresholds]
It is also possible to configure so that f (x) is processed according to a threshold value. Specifically, by configuring f (x) = 0: 0 <x <10 and f (x) = 1: x ≧ 10, the positional relationship can be converted into numerical values of “0” and “1”. it can.

Further, by using a plurality of threshold values, f (x) = 0: 0 <x <10, f (x) = 1: 10 ≦ x <20, and f (x) = 2: 20 ≦ x. , F (x) can be configured as a function having several stages of return values. Similar to the Gaussian function, it can be used to select an item from a menu or two choices.
[Periodic input by periodic function]
By using a periodic function such as sin and cos for the f (x), the positional relationship can be numerically converted periodically.
[Enter real number as is]
In order to use the terminal and the visual code 20 as a measure, or to use an actual distance or the like in an application, the calculated positional relationship can be used as a numerical value as it is. At this time, f (x) = x.
[Convert positional change]
The calculated x is stored as the position history information in the memory means provided in the terminal, and the difference between the calculated x and the positional relationship xp stored in the position history information for a certain period of time is expressed as f (x -Xp) can be converted into a numerical value corresponding to the amount of change in the positional relationship.
[Convert position ratio]
The calculated x is stored as position history information in the memory means provided in the terminal, and the ratio of the calculated x and the positional relationship xp before a certain time stored in the position history information is expressed as f (x / Xp) can also be converted into a numerical value corresponding to the positional relationship ratio.
[Change function]
It is also possible to configure by changing f (x) according to the positional relationship value and time. For example, when x is the distance, f (x) = 127 ^* (x / 80-1 / 4): 20 <x <60, f (x) = 64 ^* (x / 80 + 3/4): Assuming 60 ≦ x <100, it is possible to configure so that the amount of change in the numerical value is small at the point where the distance is 60.
[Multiple input]
When the positional relationship is composed of several parameters such as rotation and distance around the Z axis, if Y is defined as a numerical value and X is defined as a positional relationship vector, it can be expressed as Y = F (X), and a plurality of inputs It can also be configured to detect numerical values from
[Difference of multiple inputs]
The calculated X is stored as position history information in the memory means included in the terminal, and the difference between the calculated X and the positional relationship X ′ stored in the position history information for a predetermined time is calculated using the function F. XX ′) can be converted into a numerical value corresponding to the positional relationship ratio.
[Multi-input ratio]
The calculated X is stored as position history information in the memory means provided in the terminal, and the ratio of each component of the calculated X and the positional relationship X ′ stored in the position history information for a predetermined time is stored using the function F. It is also possible to convert to a numerical value corresponding to the ratio of each component of the positional relationship vector. For example, when X = (a, b, c) and X ′ = (a ′, b ′, c ′), Y = F ((a / a ′, b / b ′, c / c ′)) It can be.

It is also possible to substitute the absolute value ratio of the positional relationship vector instead of each component and convert it to a numerical value. In this case, it is also possible to convert to a numerical value by y = f (| x | / | x ′ |) using a function in the case where the positional relationship vector consists of one parameter.
[System example]
The system in this embodiment will be described below.
[Overview of operation flow]
An outline of the flow of processing is shown in the flowchart of FIG. The user images the paper on which the visual code 20 is printed with a digital camera (151). The captured camera video is sent to the CPU 7 and displayed on the monitor 8 (152). The visual code 20 is detected by the method (153). The CPU 7 acquires the coordinates and identification numbers of the four vertices of the visual code in the camera video (154).

  Next, the CPU 7 acquires visual code information from the external database unit (156) based on the identification number, and acquires, from the visual code information, positional relationship elements calculated from the camera video, numerical input target information, and the like (155). ).

  The CPU 7 calculates the positional relationship from the camera video (157) and converts it into a numerical value (158). The numerical value is input to the input target of the terminal (159), and an image indicating the result of the numerical value input to the camera image is superimposed and displayed on the monitor 8 (160).

Since these processes are performed in real time, the user can directly input the numerical value by moving the terminal while viewing the visual code 20 through the monitor 8. Finally, when the end of numerical value input is determined by the button (161), the numerical value is stored in the terminal memory and reflected. The above processing flow will be described in detail below.
[Image code display and camera image display]
When the user takes an image so that the visual code 20 falls within the angle of view of the digital camera, the coordinates of the four vertices of the visual code 20 on the image are detected from the camera image sent to the CPU 7 by the method described above. Next, the identification number of the visual code 20 is detected by the method of reading the color barcode 22 described above.
[Visual code information]
Next, the CPU 7 acquires the visual code information 170 shown in FIG. 14 from the external visual code database means 9 using the identification number. The visual code information 170 includes positional relationship information 171 for identifying elements of positional relationship calculated from the camera video, numerical input target information 172, and numerical input auxiliary information 173.
[Positional information]
The positional relationship information 171 is information for identifying elements of the positional relationship between the camera and the visual code 20 calculated from the visual code area in the camera video. Elements include the position, posture, and tilt of the visual code with respect to the camera. The positional relationship information holds an identifier, and an element is identified by the identifier. In the present embodiment, there are the following four identifiers shown in FIG.

  1. “Spinner” is an identifier indicating that the rotation angle and inclination around the Z-axis of the visual code 20 are used as positional relationship elements. In “spinner”, the user rotates the terminal around the Z axis and inputs a numerical value.

  2. “Tilt” is an identifier indicating that the rotation angle or inclination of the terminal around the Y or X axis is used as a positional relation element. In “tilt”, the user tilts the terminal around the Y or X axis and inputs a numerical value.

  3. “Sink” is an identifier indicating that the distance between the visual code 20 and the terminal is a positional relationship element. In “sink”, the user moves the terminal closer to or away from the visual code 20 and inputs a numerical value.

4). “Slider” is an identifier indicating that the position of the visual code 20 viewed from the terminal is a positional relationship element. In “slider”, the user moves the terminal parallel to the XY plane and inputs a numerical value.
[Numeric input target information]
The numerical input target information 172 is information for identifying a numerical input target. Numeric input targets such as the volume of the terminal, the scroll amount when browsing the text, and the time are identified by the information. Also, the method of numerical conversion from the positional relationship is determined depending on the object.
[Numeric input target auxiliary information]
The numerical input auxiliary information 173 is information for identifying information on the network of the content information when the visual code 20 needs to acquire the content information from an external network unit or the like.
[Read visual code information]
The CPU 7 reads the visual code information 170 received from the external database means 9 and identifies the positional relationship to be calculated, the method of converting the numerical value from the positional relationship, and the input target. Hereinafter, a specific configuration example of the visual code information will be given and processing contents will be described.
[Volume adjustment with spinner]
A case where the positional relationship information 171 is “spinner” and the input target is the volume of the terminal will be described. First, by any one of the positional relationship calculation methods, the visual code first enters the angle of view, and the inclination around the Z axis of the visual code 20 with respect to the shooting direction of the camera at the moment of detection is stored as an initial value. .

  While the user moves the terminal and the visual code 20 continues to be detected, a difference value between the calculated rotation angle and the initial value is calculated. Since it is rotation, the maximum value and the minimum value of the difference value are 0 ° to 360 °. Based on the conversion method using the linear function, the maximum and minimum values of the volume of the terminal are associated with the maximum and minimum values of the difference value, and the volume is calculated.

In FIG. 15, while the user is photographing the visual code 20 with the camera, a graphic 181 that is actually turning the volume as shown in FIG. When the user presses the numerical input end button, the set volume is stored on the terminal side. The user can operate the terminal as if turning the volume volume of the actual audio device.
[Text browsing with tilt]
The identifier of the positional relationship information 171 is “tilt”, the input target is the scroll amount, the text is acquired from the input target auxiliary information, and the text page associated with the visual code 20 is viewed while scrolling up and down. An example will be described.

  In FIG. 16, the CPU 7 acquires the address of the text page on the external network from the numerical input auxiliary information 172, and displays the text page superimposed on the camera image of the monitor 190. Next, a rough angle is calculated by a method of calculating the inclination of the visual code around the X axis with respect to the shooting direction of the camera.

  Next, the numerical value of the angle is calculated as an integer value by the conversion method using the Gaussian function. The amount of scrolling at a time is determined according to the size of the integer value, and the scroll direction is determined in the manner of upward 191 if positive and downward 192 if negative.

The CPU 7 scrolls and displays the superimposed text according to the scroll amount and direction. By using a Gaussian function as the conversion method, the user can adjust the scroll amount step by step.
[Browsing timetable with sink]
The identifier of the positional relationship information 171 is “sink”, the numerical input target is scrolling, an airplane timetable is acquired from the input target auxiliary information, and the timetable associated with the visual code 20 is scrolled up and down. An example of browsing will be described.

  In FIG. 17, the user moves the terminal closer to the visual code 20, 201, away from the terminal 200, and browses the airplane timetable associated with the visual code 20 while scrolling to select an airplane having a desired departure time.

  First, the CPU 7 obtains the address of the airplane timetable on the external network from the numerical input auxiliary information 173 and superimposes and displays the airplane timetable on the camera image of the monitor.

Next, the distance between the visual code 20 and the camera is calculated by the above method. The distance is converted into a scroll amount by the exponential function. Due to the nature of the exponential function, the amount of change in scroll amount increases as the distance increases. Although it is difficult to move the terminal greatly while always keeping the code at the angle of view with the camera away from the visual code 20, this difficulty can be reduced by using a numerical conversion method using an exponential function. When the user finds a desired flight schedule while moving the terminal, the flight can be reserved by determining the reservation with a button.
[Time setting by slider]
An example in which the identifier of the positional relationship information 171 is “slider” and the target of numerical input is time will be described. An example will be described in which the user moves the terminal parallel to the XY plane of the visual code 20, moves the slider on the paper, and designates the time.

  By the above method, the component on the XY plane of the vector from the position of the camera to the position of the visual code 20 is calculated. This X component is used. When the center of the camera image approaches the left end of the visual code, the difference in the X coordinate of the two-dimensional vector from the center of the camera image to the center of the visual code becomes a positive value. When the visual code approaches the right end, the difference between the X coordinates becomes a negative value. By mapping the approaching to a positive value to approaching an early time, and similarly mapping the approaching to a negative value to approaching a later time, the movement amount in the X-axis direction is converted into time.

In FIG. 18, the user brings the terminal close to the visual code 20 to obtain a sun and moon figure, a figure such as a slider bar, and a time (210), and puts the visual code 20 in the direction of the sun figure, that is, the left end. When the terminal is moved while taking an image, the designated time as in the slider bar approaches the early morning time (211), and when the terminal is moved while taking the visual code 20 in the direction of the shape of the moon, that is, the right end, Can specify the time to approach midnight (212). The user can set the time on the terminal by pressing a button when the desired time is specified.
[Variations / sinks / sliders / multiple inputs / multiple outputs]
A combination of the positional relationship elements is also possible. An example of adjusting the color by combining “slider and sink”, adjusting the brightness by the distance between the visual code 20 and the terminal, and assigning the hue and saturation to the position of the terminal with respect to the visual code 20 will be described.

  In FIG. 19, when a visual code 20 is detected, a hue ring 221 in which a hue having brightness corresponding to the distance is assigned to an angle and saturation is assigned to a distance from the center of the hue ring is superimposed and displayed on the monitor. The A cursor 223 is displayed at the center of the camera image, and the selected color indicated by the cursor is displayed at the upper left of the monitor (222).

  When the terminal moves on the XY plane of the visual code 20, the cursor moves relative to the camera image, and the color whose hue and saturation are changed is displayed as the selected color. Further, when the user moves the terminal in the Z-axis direction of the visual code 20, the brightness changes according to the distance, and the brightness of the hue ring 221 and the selected color also change.

  At this time, each parameter is assigned to an equation as shown in the following equation (6) and changes. The user can acquire a desired color by selecting a desired color from the hue ring 221 and operating the button with a button.

[Variation / Operation of devices other than information terminals is also possible]
It is also possible to configure to input numerical values to an external electronic device via the terminal using the infrared communication means 4 or the wireless communication means 3 provided in the terminal. An example of operating a television using the visual code 20 and a terminal will be described. When the visual code 20 is “spinner” and the volume is adjusted, the volume is calculated in the same manner as in the above embodiment. This volume can be transmitted to the television using the infrared communication means 4 to adjust the volume of the television.
[Also possible with modifications and cyber codes]
Like the visual code 20 of the present embodiment, it is not possible to freely arrange characters and figures such as icons, which assist the user, but it is possible to detect the position and orientation relative to the camera such as cyber code and ARToolKit 2 It is also possible to configure using a dimension code. [References H. Kato, M. Billinghurst, I. Poupyrev, K. Imamoto, K. Tachibana: “Virtual Object Manipulation on a Table-Top AR Environment”, In Proceedings of ISAR 2000, Oct 5th-6th, 2000. ]
[Modifications / Visual code is free shape]
When calculating the positional relationship using only the area, the circumference length, and the center of gravity, the visual code can be configured as a free shape such as a circle or an ellipse as long as it can be detected from the image. .
[Modification / Visual code is polygon]
When the diagonal and side ratio is not used, and when four or more corresponding points are obtained between the visual code 20 in the camera image and the actual visual code 20, the camera parameter matrix can be calculated. It is also possible to configure the visual code 20 used in the form of an arbitrary polygon.
[Modifications / Special polygons]
As shown in FIG. 20, in the case of a shape having one or more sets of opposite sides parallel to each other, such as a regular hexagon, it is also possible to obtain a positional relationship element in the same manner as a square. A method of calculating the positional relationship element in “spinner, tilt” when the visual code 20 is a regular hexagon (230, 235) will be described as an example.

  The rotation angle around the Z-axis of the visual code 20 in the case of “spinner” can be obtained by calculating the angle formed by the vector from the center of gravity to a predetermined vertex with the Y-axis as in the case of a square (240). .

In the case of “tilt”, the ratio of the length of the vertex L of vertex number 3 and the side L3 of the vertex of vertex number 4 to the vertex L of vertex number 6 and the side L6 of the vertex of vertex number 1 A rotation angle about the Y axis can be obtained (250).
[Variation / Visual code presentation method]
In the present embodiment, the visual code 20 is printed on paper, but it may be configured to be displayed on a display screen such as a liquid crystal monitor or CRT.
[Matching gesture input]
It is also possible to perform gesture input using the position history information. This is realized by calculating a change in the positional relationship calculated in the past stored in the position history information within a certain period of time and checking whether or not this matches a preset pattern.

  The gesture database means stores the pattern, that is, the movement of the positional relationship within a predetermined time, and a numerical value corresponding to the pattern. A change in the positional relationship within a certain time can be input to the gesture database means, and if there is a matching gesture pattern, a numerical value held by the gesture database means can be input.

For example, it is possible to detect an operation of moving the terminal away from the visual code 20 within a predetermined time and immediately returning it, and assign this operation to an operation of determining by pressing a button.
[Modifications / Switching the conversion method at a predetermined timing]
It is also possible to switch to a numerical value by switching the positional relationship conversion method at a predetermined timing. For example, when a user browses an airplane timetable by scrolling, the CPU determines that the information is displayed densely by the number of characters in the text and the like, while scrolling the portion where the information is dense, It is possible to reduce the first order coefficient of the linear function for converting the distance into a numerical value and to reduce the scroll amount proportional to the change amount of the distance so that the information can be viewed in detail.

  The calculation result of the positional relationship between the visual code 20 and the mobile terminal described in the above embodiment is stored in a memory, and hardware resources are utilized.

  1 can be configured by a computer program, or the processing steps of the flowchart shown in FIG. 13 can be configured by a computer program to cause the computer to execute the program. Needless to say, a recording medium such as a flexible disk, a CD, a DVD, an MO, a program for realizing the functions of the computer or a program for causing the computer to execute the processing steps, such as a flexible disk, CD, DVD, MO, etc. It can be recorded on a ROM, a memory card, a removable disk, etc., and stored or distributed.

  Further, the above program can be provided through a network such as the Internet or electronic mail. The present invention can be implemented by installing the program from these recording media into a computer or by downloading the program from a network and installing the program into the computer.

The structure explanatory view showing the embodiment of the invention. Explanatory drawing of the visual code printed on the paper surface with the mobile phone terminal. Detailed configuration diagram of the visual code. Explanatory drawing of the visual marker printed on the visual code. Illustration of the color bar printed on the visual code. Illustration of the X, Y, and Z axes of the visual code. Explanatory drawing which shows the visual code area | region extraction process in a camera image | video. Explanatory drawing of the calculation method of the positional relationship using the shape of a visual code area | region, and a vertex coordinate. Explanatory drawing of the detection method of the normal vector in a camera coordinate system, the vector of a diagonal, and a position vector. Explanatory drawing of the calculation method of the rotation angle using the vector from a gravity center to a vertex. Explanatory drawing of the calculation method of the inclination of the visual code around the X and Y axes. Explanatory drawing of the calculation method of distance by an area, the circumference length, a diagonal line, etc. FIG. The flowchart which gives the outline | summary of the flow of a process of a system. Visual code information explanatory diagram. Explanatory drawing when operating a terminal with the feeling which turns the volume volume of an audio equipment. Explanatory drawing when browsing text paper scrolling up and down. Explanatory drawing when browsing while scrolling the timetable of an airplane. Explanatory drawing when moving a terminal and specifying time. Explanatory drawing of the example which assigns a hue and saturation to the position with respect to the visual code | cord | chord of a terminal, and adjusts a color. Explanatory drawing of the calculation method of the positional relationship element in a spinner and a tilt in case a visual code is a regular hexagon.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Cell-phone terminal 2 ... Built-in memory 3 ... Wireless communication means 4 ... Infrared communication means 5 ... Digital camera (imaging means)
6. Operation buttons (input means)
7 ... CPU
8 ... Liquid crystal monitor (display means)
9 ... External database means 11 ... Visual code database

Claims (9)

A numerical input method using a visual code having a predetermined shape and an imaging means provided in a portable device ,
Detecting a region of the visual code in the video by the imaging means based on the marker at the corner of the visual code;
Detecting an identification number of the visual code in the video by the imaging means based on the color barcode in the area of the visual code;
Image processing the visual code in the video by the imaging means, and calculating the positional relationship between the visual code and the imaging means according to the positional relationship defined by the identification number ;
Substituting a positional relationship vector indicating the positional relationship into a predetermined function and converting it into a numerical value;
Setting the numerical value as an input value of an input target defined by the identification number;
Numeric input method characterized by having. The numerical value input method according to claim 1,
The positional relationship vector is
A parameter indicating the inclination of the visual code with respect to the shooting direction of the imaging means;
A parameter indicating the distance between the imaging means and the visual code;
A numerical input method characterized in that it is a step in which one or any of parameters indicating the direction from the position of the imaging means to the position of the visual code is combined. The numerical value input method according to claim 1,
The numerical value input method characterized in that the step of converting into a numerical value substitutes the positional relationship vector and the positional relationship vector calculated a predetermined time ago into a predetermined function, and converts it into a numerical value. In the numerical input method according to claim 1 or 3,
A numerical value input method, characterized in that the calculated numerical value or positional relationship vector is displayed as a character or a figure in a display step. A numerical input device using a visual code having a predetermined shape and an imaging means provided in a portable device ,
Means for detecting an area of the visual code in the video by the imaging means based on the marker at the corner of the visual code;
Means for detecting the identification number of the visual code in the video by the imaging means based on the color barcode in the area of the visual code;
Means for image-processing a visual code in the video by the imaging means, and calculating a positional relationship between the visual code and the imaging means according to the positional relationship defined by the identification number ;
Means for substituting a positional relationship vector indicating the positional relationship into a predetermined function and converting it into a numerical value;
Means for setting the numerical value as an input value of an input target defined by the identification number;
Numeric input device characterized by having. The numerical input device according to claim 5 ,
The positional relationship vector is
A parameter indicating the inclination of the visual code with respect to the shooting direction of the imaging means;
A parameter indicating the distance between the imaging means and the visual code;
A numerical input device characterized in that it is a means combining one or any of the parameters indicating the direction from the position of the imaging means to the position of the visual code. The numerical input device according to claim 5 ,
The numerical value input device characterized in that the means for converting to a numerical value substitutes the positional relationship vector and the positional relationship vector calculated for a predetermined time into a predetermined function and converts it into a numerical value. In the numerical input device according to claim 5 or 7 ,
A numerical value input apparatus, wherein the calculated numerical value or positional relationship vector is displayed as a character or a figure by a display means. A numerical input program, characterized in that the step in the numerical input method according to any one of claims 1 to 4 is a program for causing a computer to execute.

Images