JP2008072322A - Personal digital assistant, acquisition/operation method of contents information, and program of its method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To ensure acquisition/operation of contents information, while making it unnecessary to continue photographing two-dimensional code. <P>SOLUTION: A contents information acquisition means 61 acquires contents information corresponding to the QR code 200A photographed with a capture means 10. A calculating means 62 for a code parameter computes a relative position and a relative posture of a personal digital assistant 100 over the QR code, and a code operation parameter is computed based on this result. A calculating means 63 for a flow parameter detects an optical flow from a captured picture, and furthermore, based on this result, a flow operation parameter of the personal digital assistant is computed. A parameter switching means 64 makes either a code operation parameter or a flow operation parameter a contents operation parameter, based on predetermined conditions. A contents control means 65 operates contents information by using contents operation parameters. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a portable information terminal for acquiring / manipulating content information read from a photographed two-dimensional code (such as a QR code) and a method for acquiring / manipulating content information, and in particular, portable information operated by a human gesture or movement. The present invention relates to a technique for acquiring / manipulating content information from a relative positional relationship (spatial position and orientation) between a terminal and a two-dimensional code.

  There is a handheld AR interface technology (hereinafter referred to as HAR using a two-dimensional code) using a two-dimensional code that applies augmented reality technology (AR) to manipulate information that can be read from the two-dimensional code by human movement. is there. This interface technology calculates the relative positional relationship between a two-dimensional code and a portable information terminal that reads it in real time, and can acquire / manipulate content information obtained from the two-dimensional code by the movement of the portable information terminal It is to make.

  A description will be given of a usage example in which a two-dimensional code for listening to music is attached to a promotional poster for a CD album. In this example, when a user reads a two-dimensional code with a digital camera included in a portable information terminal, an image corresponding to each song of the album and each song name are displayed on the screen of the terminal. When the user moves the terminal closer to or away from the two-dimensional code while photographing the two-dimensional code, the displayed music changes according to the distance between the terminal and the code. Then, when the user presses the enter button when a song to be auditioned is displayed, the user can audition the song. In the conventional interface, selection was made with a button, but in this system, music can be selected by movement of the terminal.

  If a two-dimensional code for volume control is attached to the poster, when the user turns the terminal around the normal vector of the code while reading the code for the volume while listening to the music, A function for adjusting the volume can also be provided.

  As another application example, a method for calculating a relative positional relationship when a visual code having a predetermined shape is photographed by a camera of a portable information terminal, converting the positional relationship into a numerical value, and inputting the numerical value as a numerical value is applied. A person has already proposed (for example, refer to Patent Document 1).

As described above, a system using HAR using a two-dimensional code has two advantages. One is that a smooth operation can be provided by replacing a button operation that has been conventionally performed after reading a two-dimensional code with an operation by movement. The other is that an intuitive operation method based on an operation method in the real world such as volume control can be provided.
JP 2006-114943 A

  There is a problem in operability in the conventional method for realizing HAR using a two-dimensional code. In the conventional method, the user must always store the two-dimensional code to be operated within the angle of view of the digital camera while moving and operating the portable information terminal. In other words, the user must move the terminal closer, away, and turn while storing the two-dimensional code in the angle of view of the digital camera included in the portable information terminal. If the two-dimensional code deviates from the angle of view of the digital camera while performing the volume operation, the operation is interrupted. For this reason, the two-dimensional code is carefully operated so as not to deviate from the angle of view, which causes a problem that the two-dimensional code cannot be operated smoothly.

  In order to promote the spread of services by HAR using a two-dimensional code, it is desirable that the operability can be improved. However, in the conventional operation method, since the parameter used for the operation is calculated only by photographing the two-dimensional code, the two-dimensional code needs to be included in the angle of view as described above, and the operability is deteriorated. there were. On the other hand, there is a method of using a device that detects the attitude of a portable information terminal such as a gyro device, but there is a problem that the introduction cost is high because a new device needs to be added.

  An object of the present invention is to provide a portable information terminal, a content information acquisition / operation method, and a program for this method, which ensure the acquisition / operation of content information while eliminating the need to continue photographing a two-dimensional code. There is.

  In the present invention, in order to detect the movement of the portable information terminal even in a situation where a two-dimensional code is not photographed, a flow parameter for calculating a flow manipulation parameter used for an operation by an optical flow and a two-dimensional code are photographed. When the computer is in use, a code operation parameter for calculating a code operation parameter used for the operation is used.

  In addition, in order to enable operation even when the two-dimensional code deviates from the angle of view, when the two-dimensional code deviates from the angle of view, the flow operation parameter is used to detect the two-dimensional code. Any one of the operation parameters is appropriately switched according to a situation in which the code operation parameter is used.

  Further, in the parameter switching, a two-dimensional code detection state is predicted for the flow operation parameter or the code operation parameter, and the operation parameter is appropriately switched based on the prediction result.

  Further, the flow operation parameter has a disadvantage that it is less accurate than the code operation parameter. In order to eliminate this disadvantage, the flow operation parameter is corrected by the code operation parameter.

  As described above, the present invention is characterized by the following apparatuses, methods, and programs.

(Invention of portable information terminal)
(1) A two-dimensional code is photographed using a portable information terminal provided with photographing means, and content information read from the two-dimensional code is acquired / operated by the relative position and orientation of the portable information terminal and the two-dimensional code. A portable information terminal,
Content information acquisition means for acquiring content information corresponding to the two-dimensional code read by the photographing means;
Using the shape of the two-dimensional code in the photographed image photographed by the photographing means, the relative position and orientation of the portable information terminal with respect to the two-dimensional code are calculated, and the code operation is performed based on the relative position and orientation A code parameter calculation means for calculating a parameter;
Anti-flow parameter calculating means for detecting an optical flow from an image photographed by the photographing means and calculating a flow operation parameter of the portable information terminal based on the optical flow;
Based on a predetermined condition, a parameter switching unit that uses either the code operation parameter or the flow operation parameter as a content operation parameter;
Content operating means for operating content information using the content operation parameters.

(2) Two-dimensional code state prediction means for calculating code state prediction information in which the detection state of the two-dimensional code in the captured image is predicted;
The parameter switching means is characterized in that either the code operation parameter or the flow operation parameter is set as a content operation parameter based on the code state prediction information.

  (3) A flow calculation correction unit that corrects the flow operation parameter using the code operation parameter is provided.

  (4) A flow calculation correction unit that corrects the flow operation parameter using the code state prediction information and the code operation parameter is provided.

(Invention of method)
(5) A two-dimensional code is photographed using a portable information terminal provided with photographing means, and content information read from the two-dimensional code is acquired / operated according to the relative position and orientation of the portable information terminal and the two-dimensional code. A method for acquiring / manipulating content information,
A content information acquisition step of acquiring content information corresponding to the two-dimensional code read by the photographing means;
Using the shape of the two-dimensional code in the photographed image photographed by the photographing means, the relative position and orientation of the portable information terminal with respect to the two-dimensional code are calculated, and the code operation is performed based on the relative position and orientation A code parameter calculation step for calculating parameters;
An anti-flow parameter calculating step of detecting an optical flow from an image captured by the imaging means and calculating a flow operation parameter of the portable information terminal based on the optical flow;
Based on a predetermined condition, a parameter switching step in which either the code operation parameter or the flow operation parameter is a content operation parameter;
And a content operation step of operating content information using the content operation parameter.

(6) a two-dimensional code state prediction step of calculating code state prediction information in which the detection state of the two-dimensional code in the captured image is predicted;
In the parameter switching step, one of the code operation parameter and the flow operation parameter is set as a content operation parameter based on the code state prediction information.

  (7) A flow calculation correction step of correcting the flow operation parameter using the code operation parameter is provided.

  (8) A flow calculation correction step of correcting the flow operation parameter using the code state prediction information and the code operation parameter is provided.

(Invention of the program)
(9) The processing steps in the content information acquisition / operation method according to the above (5) to (8) can be executed by a computer.

As described above, according to the present invention, acquisition / operation of content information is ensured while eliminating the need to continue photographing a two-dimensional code. In detail,
(1) During operation of a handheld AR interface using a two-dimensional code, two-dimensional code is used to calculate parameters for operation from two means of a photographed two-dimensional code and an optical flow, and switch between them for use. There is no need to continue shooting the code, and operability can be improved.

  (2) When the two-dimensional code is photographed and detected, by predicting a state that the two-dimensional code is likely to deviate from the angle of view, the operation parameters obtained from the means of the two-dimensional code and the optical flow are appropriately set. Can be switched.

  (3) By using the shape of the photographed two-dimensional code, it is possible to correct an error when calculating the operation parameters from the optical flow.

  (4) Since the timing for correcting the optical flow error is determined using the prediction result of the detection state of the two-dimensional code, the calculation cost can be reduced.

(Embodiment 1)
In the present embodiment, a mobile information terminal that reads a QR code that is a two-dimensional code using a mobile phone that is a mobile information terminal and operates the playback volume of a music that is content information embedded therein by the movement of the mobile phone And a method for acquiring / manipulating content information. Hereinafter, the case where the volume operation is performed will be described in detail. However, each embodiment described below is not limited to the volume, and can be applied to operation parameters in general.

(A) Device Block Configuration FIG. 1 shows a functional block configuration for volume control using a two-dimensional code photographed by a mobile phone (portable information terminal). The cellular phone terminal 100 includes an imaging unit 10 for capturing a moving image, a communication unit 20 for network communication, an input unit 30 for a user to input, and a display unit 40 for a user to check and browse information. , An audio reproduction means 50 for reproducing audio, and a general-purpose calculation means 60 for performing calculation processing. The general-purpose calculation means 60 includes a microcomputer mounted on a mobile phone and software described as a program in a program memory of the computer.

  The block configuration of the general-purpose calculation means 60 is a content information acquisition that reads a QR code (two-dimensional code) 200A from a photographed image of a poster 200 photographed by the photographing means 10 and obtains content information corresponding to the read two-dimensional code. Means 61, a code parameter calculation means 62 for calculating the position / posture parameter of the mobile phone 100 with respect to the QR code based on the shape of the QR code appearing in the captured image from the captured image, and calculating the code operation parameter; A flow parameter calculation unit 63 for detecting an optical flow from the captured image (moving image) and calculating a flow operation parameter, and an operation parameter calculated by the code parameter calculation unit 62 and the flow parameter calculation unit 63 are determined in advance. Switching parameters based on conditions Means 64, the content information the content information acquisition unit 61 acquires includes a content operation means 65 for applying an operation by the said code operating parameters or the flow operation parameter parameter switching means is switched.

(B) Outline of operation When the user captures the QR code 200A on the poster 200 by the imaging unit 10 of the mobile phone 100, the music data is obtained by the content information acquisition unit 61 based on the content identification information embedded in the QR code 200A. Acquired through the communication means 20, the music data is reproduced by the audio reproduction means 50. The user adjusts the volume of the music data to be reproduced by rotating the mobile phone 100 about the axis of the normal line of the QR code 200A while the music data is being reproduced.

(C) Details of Processing FIG. 2 shows a procedure (S1 to S7) of volume adjustment operation processing, which will be described in detail below.

(S1) Content Information Acquisition Step An operation is started when the user captures the QR code with the photographing means 10. In the content information acquisition step, the content information acquisition unit 61 extracts a QR code from the photographed image generated by the photographing unit 10, and uses the code information (content identification information) included in the QR code as a key (see FIG. 3). , Music data hoge.content, which is content information, from the music database via the communication means 20. mp3 is acquired (see FIG. 4). The acquired music data is stored as content information.

(S2) Shooting Step After the content information is acquired in the content information acquisition step, sequentially captured images are continuously generated.

(S3) Pair Code Parameter Calculation Step In the pair code parameter calculation step, the pair code parameter calculation means 62 calculates the position and orientation of the mobile phone 100 with respect to the QR code based on the shape of the QR code shown in the photographed image. .

  As a calculation method, for example, the method described in Patent Document 1 (Japanese Patent Application Laid-Open No. 2006-114943) can be used. As shown in FIG. 5, the outline of this calculation method is based on the fact that the QR code is on one plane, and the four corner points on the camera image (b in the figure) and the real-world QR code. The camera parameter matrix is calculated by associating the four corner points (a in the figure), and the real-world three-dimensional coordinates are calculated from the two-dimensional coordinates in the camera image from the camera parameter matrix. The normal vector of the QR code in the camera coordinate system and the vector of two orthogonal diagonal lines of the QR code as shown in (c), and the position vector of the center of gravity of the QR code as shown in (d) of FIG. The posture and position of the camera with respect to the QR code are calculated using these four vectors.

  By this method, a total of six position and posture parameters of the camera relative position x, y, z with respect to the QR code and the posture (rotation amounts φ, θ, ψ around the XYZ axes) can be calculated. In this embodiment, the rotation amount θ around the Y axis is used as a code operation parameter.

  However, if the QR code is not detected in the captured image, the code operation parameter is not calculated during this step, and the code operation parameter is output as NULL.

(S4) Anti-flow parameter calculation step In the anti-flow parameter calculation step, the anti-flow parameter calculation means 63 calculates an optical flow and calculates a flow operation parameter. The optical flow can be calculated by, for example, the Lucas & Kanade algorithm shown in the following non-patent document.

Non-patent literature “Lucas, B., and Kanade, T. An Iterative Image Registration Technology with an Application to Stereo Vision, Proc. Of 7th International Cont.
Using this optical flow, the angular velocity θv of the mobile phone is calculated. First, looking at the direction of each flow from the center point, an angle θn as shown in FIG. 6 is calculated for all flows. If there is no average flow in the four quadrants of these angles θn, it is determined that the angle θn is not rotated around the center of the angle of view, and this step is terminated. Subsequently, when the above condition is not satisfied, an average of all θn is calculated from the following formula (1), and is calculated as an angular velocity θv. The current rotation angle is θ _last , the calculated angular velocity θv is added to θ _last , and the rotation amount θ is calculated from the following equation (2).

  However, k is a weight and is 1 in this embodiment.

  In the present embodiment, the rotation amount θ around the Y axis is used as a flow operation parameter.

(S5) Parameter switching step In the parameter switching step, the parameter switching means 64 receives the code operation parameter and the flow operation parameter, and outputs the content operation parameter.

  When the code operation parameter is not NULL, the code operation parameter is output as a content operation parameter. When the code operation parameter is NULL, the flow operation parameter is output as a content operation parameter.

(S6) Content Operation Step The content operation means 65 reads the stored music data (content information read), and the music data hoge. mp3 is reproduced by the sound reproducing means. The sound volume at this time is determined based on the magnitude of the content operation parameter output in the parameter switching step. The volume is set according to the following formula (3) (see FIG. 7).

  Further, the content operation unit 65 presents a graphic representing the volume on the display unit 40 (see FIG. 8). By viewing this, the user can perform an operation while visually confirming the volume of the current playback volume.

(S7) Processing loop When the user presses the end button with the input means, the processing ends. If the end button is not pressed, the process returns to the photographing step and the series of steps is repeated.

  Therefore, according to the present embodiment, during the operation of the handheld AR interface using the two-dimensional code, the parameters for the operation are calculated from the two means of the photographed two-dimensional code and the optical flow, and they are switched. Since it is used, there is no need to continue photographing the two-dimensional code, and operability can be improved.

(Modification of Embodiment 1)
In the first embodiment, in the parameter switching step (S5), the flow operation parameter is output as the content operation parameter only when no QR code is detected. It is also possible to configure so that the flow operation parameter is always output as the content operation parameter after reading the QR code.

(Embodiment 2)
In the present embodiment, the detection state of the QR code is predicted, and the prediction result is used for the parameter switching step.

  In the first embodiment, when the QR code is no longer detected, the content operation parameter is switched from the code operation parameter to the flow operation parameter. In general, when a QR code approaches a detection limit due to flickering of a captured image, a state where it is not detected or is generated occurs. For this reason, when it is in the vicinity of the detection limit, there is a possibility that switching between the code operation parameter and the flow operation parameter occurs frequently. In this embodiment, in addition to the first embodiment, the parameter switching means includes a two-dimensional code state prediction means for predicting the detection state of the QR code, thereby smoothing this switching and reducing the calculation cost. is there. Hereinafter, the configuration of the portable information terminal and the content information acquisition / operation method will be described in detail.

(A) Device Block Configuration FIG. 9 shows a functional block configuration for voice volume operation using a two-dimensional code photographed by a mobile phone (portable information terminal). FIG. 10 shows a procedure (S1 to S8) of volume adjustment operation processing.

  The present embodiment is different from the first embodiment in that a two-dimensional code state prediction unit 66 that generates code state prediction information including state parameters indicating the current state of the QR code, and a two-dimensional code state prediction step (S8) ) Is added.

  The parameter switching means 64 uses either the flow operation parameter or the code operation parameter as the content operation parameter based on the two-dimensional code state prediction information.

(B) Details of Processing FIG. 11 shows a detailed procedure diagram of parameter switching processing based on two-dimensional code state prediction, and will be described for each step.

(S8) Two-dimensional code state prediction step In the two-dimensional code state prediction step, the two-dimensional code state prediction means has a circumscribing rectangle parameter for predicting whether the QR code is likely to deviate from the angle of view, and the inclination of the QR code is too large. Three parameters: a posture parameter for predicting whether detection is likely to be impossible, and a distance estimation parameter for predicting whether detection is impossible because the QR code distance is too far or too close Is calculated. These circumscribed rectangle parameters, posture parameters, and distance parameters are calculated by the following processing, and are transmitted to the parameter switching means 64 as code state prediction information.

(S8A) circumscribing rectangle parameter calculation step The circumscribing rectangle of the QR code in the photographed image is calculated, and four distances from the circumscribing rectangle to the edge of the image are calculated. For example, the distances x1, x2, y1, and y2 to be deviated from the four angles of view in the vertical and horizontal directions shown in FIG. These x1, x2, y1, and y2 are set as circumscribed rectangle parameters.

(S8B) Attitude parameter calculation step The attitude of the QR code with respect to the mobile phone is detected by the same method as the code operation parameter calculation step. For example, the posture is calculated as three parameters φ, θ, and ψ shown in FIG. 13, and these three parameters are used as posture parameters.

(S8C) Distance parameter calculation step The distance parameter is calculated using the area occupied by the QR code shape in the captured image (see FIG. 14). This area value is used as a distance parameter.

(S5) Parameter switching step The parameter switching means 64 detects detected stable parameter range data including the circumscribed rectangular parameter value range, the posture parameter value range, and the distance parameter value range when the QR code is detected with a margin. The parameter is switched in accordance with the following processing after previously saving in the nonvolatile memory.

(S5A) Determination result information storage step The parameter switching means 64 determines whether or not the circumscribed rectangle parameter, the posture parameter, and the distance parameter obtained in the above step are all within the range of the detected stable parameter range data. Here, the determination result information as to whether or not the value is within the range is stored in the determination result storage means (memory) together with the time information.

(S5B) Prediction Step The two-dimensional code state prediction unit 66 extracts determination result information included in a predetermined time (for example, 3 seconds) from the time information stored in the determination result storage unit, and performs the determination. When the result information indicates that all of the result information is not in the range, the flow operation parameter is set as the content operation parameter. In cases other than the case where all the judgment result information indicates that they are not within the range, the code operation parameter is set as the content operation parameter, and the content operation parameter is transferred to the content operation means 65.

  According to the present embodiment, when a two-dimensional code is photographed and detected, by predicting a state that it is likely to deviate from the angle of view, an operation obtained from the means of the two-dimensional code and the optical flow is performed. Parameters can be switched appropriately.

(Modification of Embodiment 2)
In the second embodiment, the time when the detection parameter is out of the detection stable parameter range is measured, and it is calculated from the detection stability parameter range that the QR code in the photographed image is out of the specified time, so that the QR code is not detected. As a result, the code state prediction information is generated.

  In this modification, code state prediction information is generated in consideration of the speed of movement. Depending on the speed at which the user moves the mobile phone, the time interval from when the detection stable parameter range falls into a state where it cannot actually be detected varies. In this modification, the speed of the two-dimensional code state parameter composed of the three parameters is calculated, and the parameter is switched according to the speed.

  As shown in FIG. 15, the same processing as in the second embodiment is performed up to the two-dimensional code state prediction step. The subsequent processing will be described step by step.

(S5C) Two-dimensional code state parameter speed calculation step The parameter switching means 64 reads past code state prediction information stored in the parameter temporary storage means as past code state prediction information. Next, the current code state prediction information is stored in the parameter temporary storage means.

  The parameter switching unit 64 subtracts the past code state prediction information from the code state prediction information to calculate the code state prediction information change amount. The code state prediction information change amount includes a circumscribed rectangle parameter change amount, a posture parameter change amount, and a distance parameter change amount.

  The parameter switching means 64 adds the circumscribed rectangle parameter, posture parameter, distance parameter, circumscribed rectangle parameter change amount, posture parameter change amount, distance parameter change amount for each parameter, and predicts circumscribed rectangle parameter, predicted posture parameter. Then, a prediction parameter including a prediction distance parameter is calculated.

(S5D) Prediction Step The parameter switching means 64 enters the predicted circumscribed rectangle parameter, predicted posture parameter, and predicted distance parameter into the circumscribed rectangle parameter value range, posture parameter value range, and distance parameter value range obtained from the detected stable parameter range data. Judgment is made.

  If any one of the parameters does not fall within the range, the flow operation parameter is set as the content operation parameter because it is highly likely that the QR code will not be detected after the next frame as a prediction result. In other cases, the code operation parameter is set as the content operation parameter. Finally, the parameter switching unit 64 transfers the content operation parameter to the content operation unit 65.

(Embodiment 3)
The flow operation parameter is calculated by the sum of angular velocities as described in the first embodiment. This is more likely to accumulate errors than the code operation parameter that constantly detects the angle. In this embodiment, as shown in the block configuration of FIG. 16, the mobile phone includes a flow calculation correction unit 67 that corrects the flow operation parameter using the code operation parameter. FIG. 17 shows a procedure (S1 to S7, S9) of the volume adjustment operation process, and the process by the flow calculation correction step will be described in detail below.

(S9) Flow Calculation Correction Step In the flow calculation correction step, the flow calculation correction means 67 acquires the code operation parameter and the flow operation parameter. These are a code operation parameter θ _code and a flow operation parameter θ _flow that can be expressed by the following equations (4) and (5).

Here, k _temp is calculated by equation (5) so that equation (4) holds. The calculated k _temp is stored in a weight storage unit (memory) included in the flow calculation correction unit 67. When the flow operation parameter is used as the content operation parameter in the parameter switching step S5, the average value k _{temp_ave} of all the k _temp stored in the weight storage means is calculated, and the expression (1) performed in the subsequent flow parameter calculation step thereafter. ) Of k) is updated with the average value k _{temp_ave} .

  Therefore, according to the present embodiment, it is possible to automatically correct an error when calculating a parameter for an operation from an optical flow by using the shape of a photographed two-dimensional code.

(Modification of Embodiment 3)
In the third embodiment, the flow calculation correction is always performed. It is computationally expensive to perform code operation parameter calculation, flow operation parameter calculation and flow correction simultaneously. Therefore, the flow correction indicates that the code state prediction information is switched to the flow operation parameter, and can be configured to perform correction only when a QR code is detected.

(Modification 1) Example of calculation in optical flow of vertical / horizontal translation In the embodiment, the rotation angle around the normal vector of the QR code of the terminal is used as the operation amount. In addition to this, it is also possible to realize by using a vertical and horizontal two-dimensional parallel movement amount as an operation amount.

  As the code operation parameter, the two-dimensional translation amount can be calculated using the x and y parameters described in the first embodiment. With respect to the flow operation parameter, vertical and horizontal movements can be calculated from the optical flow by the following method.

FIG. 18 shows an example of the calculated optical flow, where (a) is an example of the optical flow at the moment when the mobile phone is moving to the lower right, and (b) is the mobile phone moving in parallel to the left. It is the same example of the moment. When an optical flow as shown in the figure is calculated, each vector of the flow is two-dimensionally expressed as a vector v _n = (x _n , y _n ) (in the formula, it is indicated by adding → to v). ). The translation speed v _v is calculated from the average of the vectors v _n in the captured image by the following equation (6). The parallel movement amount is set as a vector p _last, and in the following equation (7), the calculated parallel movement speed (angular velocity) v _v is added to the p _last to calculate the parallel movement amount vector p.

However, k _x and k _y are weights, k _x = 1, k _y = 1.
This parallel movement amount can also be corrected using the code operation parameter in the same manner as described in the third embodiment. In the flow calculation correction step S9, the flow calculation correction means 67 acquires the code operation parameter and the flow operation parameter. _Assuming that the code operation parameter p _code and the flow operation parameter p _flow are those, k _xtemp and _kytemp are calculated by the equation (9) so that the following equation (8) is satisfied.

The calculated _kytemp and _kytemp are stored in a weight storage unit (memory) _included in the flow calculation correction unit 67.

In the parameter switching step S5, when utilizing a flow operating parameters as content operating parameters, the average value k _Xave values for all k _x, k _y stored in the weight storage means to calculate a k _Yave, since, versus flow calculation of k _x of formula is carried out by the parameter calculating step (7), the average value k _Xave value of k _y, is updated with k _Yave.

(Modification 2) Example of calculation of depth movement amount in optical flow In the embodiment, the rotation angle around the normal vector of the QR code of the terminal is used as the operation amount. Besides this, it is also possible to calculate the depth movement amount.

  As described in the first embodiment, the code operation parameter can use the z parameter to calculate the amount of movement of the depth. As the flow operation parameter, the depth movement amount can be calculated from the optical flow by the following method.

  First, the direction of movement in the depth direction and whether there is such movement is determined. As shown in FIG. 19 (a), the left quadrant optical flow in the first quadrant, the right quadrant optical flow in the second quadrant, the right quadrant optical flow in the third quadrant, and the fourth quadrant. When an optical flow upwards to the left appears, the depth movement amount is assumed to be a direction in which the mobile phone is moved away from the QR code. In addition, as shown in FIG. 19B, an upper right optical flow in the first quadrant, an upper left optical flow in the second quadrant, and a lower left optical flow in the third quadrant, When the optical flow pointing downward to the right appears in the quadrant, the depth movement amount is assumed to be a direction in which the mobile phone is brought closer to the QR code, and in other cases, the depth movement amount is assumed to be absent.

Next, the amount of movement of the depth is estimated. When the optical flow is calculated as shown in FIG. 19, the vectors of the respective flows are two-dimensionally expressed as vectors v _n = (x _n , y _n ). The average of the absolute values | v _n | of the vectors v _n in the captured image is calculated by the following equation (10) to calculate the absolute value of the translational speed | v _v |. The current flow depth movement amount is d _past, and in the following equation (11), the calculated parallel movement speed absolute value | v _v | is added to d _past to calculate the flow depth movement amount d.

This flow depth movement amount can also be corrected by using the code operation parameter in the same manner as described in the third embodiment and the modified example. In the flow calculation correction step S9, the flow calculation correction means 67 acquires the code operation parameter and the flow operation parameter. If these are the code operation parameter z and the flow operation parameter d,
K _temp is calculated by the following equation (13) so that the following equation (12) holds.

The calculated k _temp is stored in a weight storage unit (memory) included in the flow calculation correction unit. In the parameter switching step S5, when the flow operation parameter is used as the content operation parameter, an average value k _ave of all the k _temp values stored in the weight storage unit is calculated, and thereafter, the flow parameter calculation step is performed. The value of k in the calculation of Expression (11) is updated with the average value k _ave .

(Modification 3) Example of using a physical quantity The QR code to be used is configured by including physical size information indicating the physical size of the code itself, or the QR code is configured with a predetermined physical size. The x, y, and z parameters calculated when calculating the operation parameters can be reproduced in actual size. As a result, the flow operation parameter corrected by the flow calculation correction means can be used in the system as a virtual physical quantity.

(Modification 4) Example in which another QR code can be used In the present embodiment, the QR code is used as a two-dimensional code. However, two codes such as cyber code, ARTToolkit, and DataMatrix, which are also described in the following references, are used. It is also possible to configure using a dimension code.

Reference "Knowledge of barcodes and two-dimensional codes" Nippon Kogyo Publishing (2001)
(Modification 5) Optical Flow Calculation Method In the present embodiment, the Lucas & Kanade algorithm is used for optical flow calculation. However, it may be configured to use other methods such as block matching.

(Modification 6) Example of enabling processing on the server side In the present embodiment, every step is executed on the general-purpose computing means of the mobile phone. It is also possible to configure the server means on the network to perform all the processes performed by the general purpose computing means.

  In the present invention, some or all of the processing functions of the method shown in FIG. 2 and the like can be configured as a program and executed by a computer. In addition, a computer-readable recording medium such as a flexible disk, MO, ROM, or memory card can be used to store a program for realizing the processing function of each unit by the computer or a program for causing the computer to execute the processing procedure. It can be recorded on a CD, a DVD, a removable disk, etc., stored, provided, and distributed via a communication network such as the Internet.

The functional block block diagram of the mobile telephone which shows Embodiment 1 of this invention. FIG. 3 is a procedure diagram of volume adjustment operation processing according to the first embodiment. The schematic diagram of a two-dimensional code. The data structural example of the music database for music data. FIG. 3 is a schematic diagram of code parameter calculation. The schematic diagram of the optical flow which calculates rotation amount. Operation image for adjusting the playback volume. Graphic example during playback volume adjustment. The functional block block diagram of the mobile telephone which shows Embodiment 2 of this invention. FIG. 10 is a flowchart illustrating a sound volume adjustment operation process according to the second embodiment. The detailed procedure figure of the parameter switching process based on two-dimensional code state prediction. The outline figure of circumscribed rectangle parameter. FIG. The outline figure of a distance parameter. The detailed procedure figure of a two-dimensional code state prediction process. The functional block block diagram of the mobile telephone which shows Embodiment 3 of this invention. FIG. 10 is a flowchart illustrating a sound volume adjustment operation process according to the third embodiment. Optical flow image for translation. Image of optical flow for depth movement.

Explanation of symbols

100 Mobile phone (personal digital assistant)
200 poster 200A QR code 10 photographing unit 20 communication unit 61 content information acquisition unit 62 pair code parameter calculation unit 63 pair flow parameter calculation unit 64 parameter switching unit 65 content operation unit 66 two-dimensional code state prediction unit 67 flow calculation correction unit

Claims (9)

A portable information terminal that captures / manipulates content information read from a two-dimensional code by photographing a two-dimensional code using a portable information terminal including a photographing unit and relative positions and postures of the portable information terminal and the two-dimensional code Because
Content information acquisition means for acquiring content information corresponding to the two-dimensional code read by the photographing means;
Using the shape of the two-dimensional code in the photographed image photographed by the photographing means, the relative position and orientation of the portable information terminal with respect to the two-dimensional code are calculated, and the code operation is performed based on the relative position and orientation A code parameter calculation means for calculating a parameter;
Anti-flow parameter calculating means for detecting an optical flow from an image photographed by the photographing means and calculating a flow operation parameter of the portable information terminal based on the optical flow;
Based on a predetermined condition, a parameter switching unit that uses either the code operation parameter or the flow operation parameter as a content operation parameter;
Content operating means for operating content information using the content operating parameters;
A portable information terminal comprising: Two-dimensional code state prediction means for calculating code state prediction information predicting the detection state of the two-dimensional code in the captured image;
The portable information terminal according to claim 1, wherein the parameter switching unit sets one of the code operation parameter and the flow operation parameter as a content operation parameter based on the code state prediction information.   The portable information terminal according to claim 1, further comprising: a flow calculation correction unit that corrects the flow operation parameter using the code operation parameter.   The portable information terminal according to claim 2, further comprising: a flow calculation correction unit that corrects the flow operation parameter using the code state prediction information and the code operation parameter. Content information for capturing / manipulating content information read from a two-dimensional code by photographing a two-dimensional code using a portable information terminal equipped with a photographing unit and relative position and orientation between the portable information terminal and the two-dimensional code An acquisition / operation method,
A content information acquisition step of acquiring content information corresponding to the two-dimensional code read by the photographing means;
Using the shape of the two-dimensional code in the photographed image photographed by the photographing means, the relative position and orientation of the portable information terminal with respect to the two-dimensional code are calculated, and the code operation is performed based on the relative position and orientation A code parameter calculation step for calculating parameters;
An anti-flow parameter calculating step of detecting an optical flow from an image captured by the imaging means and calculating a flow operation parameter of the portable information terminal based on the optical flow;
Based on a predetermined condition, a parameter switching step in which either the code operation parameter or the flow operation parameter is a content operation parameter;
A content operation step of operating content information using the content operation parameter;
A method of acquiring / manipulating content information, comprising: A two-dimensional code state prediction step of calculating code state prediction information in which the detection state of the two-dimensional code in the captured image is predicted;
6. The content information acquisition / operation method according to claim 5, wherein the parameter switching step uses one of the code operation parameter and the flow operation parameter as a content operation parameter based on the code state prediction information. .   6. The content information acquisition / operation method according to claim 5, further comprising a flow calculation correction step of correcting the flow operation parameter using the code operation parameter.   The content information acquisition / operation method according to claim 6, further comprising: a flow calculation correction step of correcting the flow operation parameter using the code state prediction information and the code operation parameter.   9. A program that enables a computer to execute the processing steps in the content information acquisition / operation method according to claim 5.