WO2023218886A1

WO2023218886A1 - Information input device, information input method, and information input program

Info

Publication number: WO2023218886A1
Application number: PCT/JP2023/015665
Authority: WO
Inventors: 卓吾岩間; 政樹武田
Original assignee: ゼブラ株式会社
Priority date: 2022-05-12
Filing date: 2023-04-19
Publication date: 2023-11-16

Abstract

This information input system comprises: an acquisition unit for acquiring from a sensor an observation value regarding the action of an input apparatus; an estimation unit for estimating the estimate value of state of the input apparatus on the basis of the observation value; and an output unit for outputting information indicated by the locus of the estimate value to a display device as input information by the input apparatus. The estimation unit includes: a model acquisition unit for acquiring an input apparatus model in which the input apparatus is modeled; a disturbance calculation unit for calculating a disturbance on the input apparatus model, using the observation value and mass information that indicates the mass of the input apparatus; an analysis execution unit for giving an disturbance, including mass information, to the input apparatus model, so as to execute the simulation of action of the input apparatus using the input apparatus model; and an estimate value derivation unit for associating the state of the input apparatus model indicated by the result of execution of simulation with the current state of the input apparatus, so as to derive an estimate value.

Description

Information input system, information input method, and information input program

The present disclosure relates to an information input system, an information input method, and an information input program.

As technologies related to information input systems, for example, the technologies described in Patent Documents 1 to 5 are known. For example, Patent Document 1 discloses a writing instrument that detects a writing motion and provides writing information to a user. In this writing instrument, an inclined portion is provided at the tip of the barrel, and a motion sensor is housed in the inclined portion. The motion sensor measures the handwriting motion, and handwriting information is acquired based on the measurement results of the motion sensor.

JP 2021-033563 Publication JP2018-067131A Special table 2016-530612 publication Japanese Patent Application Publication No. 2013-125487 Japanese Patent Application Publication No. 2003-114754

In the above-mentioned writing instrument, the motion sensor is placed near the tip of the writing instrument to suppress errors between the writing information measured by the motion sensor and the actual writing motion. However, since the motion sensor cannot be placed at the tip of the writing instrument, there is a limit to improving the accuracy of handwritten information by bringing the motion sensor closer to the tip of the writing instrument. On the other hand, it is also conceivable to perform a correction calculation to convert the handwriting information measured at the position of the motion sensor into the handwriting information at the position of the tip of the writing instrument. However, in this case, there is a possibility that the movement of the tip of the writing instrument obtained by the correction calculation will be an unnatural movement that differs from the actual writing motion. Therefore, this method may give the user a sense of discomfort.

The present disclosure provides an information input system, an information input method, and an information input program that can improve the accuracy of input information while suppressing the discomfort that a user feels during input operations.

An information input system according to an embodiment of the present disclosure includes an acquisition unit that acquires observed values regarding the operation of an input device from a sensor, an estimation unit that estimates an estimated value of a state of the input device based on the observed values, and an estimated value. and an output unit that outputs information indicated by the trajectory to a display device as input information from an input device. The estimation unit includes a model acquisition unit that acquires an input device model that models the input device, and a disturbance calculation unit that calculates a disturbance to the input device model using the observed value and mass information indicating the mass of the input device. , an analysis execution unit that simulates the operation of the input device using the input device model by applying a disturbance to the input device model that includes mass information, and the current state of the input device model indicated by the simulation execution result. and an estimated value deriving unit that derives the estimated value by associating it with the state of the input device.

An information input method according to one embodiment of the present disclosure is an information input method executed by an information input system including a processor. This information input method includes the steps of acquiring observed values regarding the operation of the input device from the sensor, estimating the estimated value of the state of the input device based on the observed values, and inputting information indicated by the trajectory of the estimated value. and outputting to a display device as input information from the device. The step of estimating the estimated value includes a step of obtaining an input device model that models the input device, and a step of calculating a disturbance to the input device model using the observed value and mass information indicating the mass of the input device. , the step of simulating the operation of the input device using the input device model by applying a disturbance to the input device model including mass information, and the step of simulating the operation of the input device using the input device model, and calculating the state of the input device model indicated by the simulation execution result based on the current input The method includes the step of deriving the estimated value by associating it with the state of the device.

An information input program according to an embodiment of the present disclosure includes a step of acquiring an observed value regarding the operation of an input device from a sensor, a step of estimating an estimated value of the state of the input device based on the observed value, and a trajectory of the estimated value. outputting the information indicated by the input device to the display device as input information by the input device. The step of estimating the estimated value includes a step of obtaining an input device model that models the input device, and a step of calculating a disturbance to the input device model using the observed value and mass information indicating the mass of the input device. , the step of simulating the operation of the input device using the input device model by applying a disturbance to the input device model including mass information, and the step of simulating the operation of the input device using the input device model, and calculating the state of the input device model indicated by the simulation execution result based on the current input The method includes the step of deriving the estimated value by associating it with the state of the device.

In the above embodiment, observed values regarding the operation of the input device are acquired from the sensor, and disturbances to the input device model are calculated using the observed values and mass information. Then, by applying a disturbance to the input device model, a simulation of the operation of the input device using the input device model is executed. The state of the input device model indicated by the simulation execution result is associated with the current state of the input device. As a result, an estimated value of the state of the input device is derived. By using a simulation using an input device model in this way, information obtained at the position of a sensor can be converted into information at an arbitrary position of the input device and processed. Therefore, the estimated value can be derived with high accuracy based on the observed value from the sensor, regardless of the position of the sensor with respect to the input device. Furthermore, by applying a disturbance to the input device model that includes mass information indicating the mass of the input device, it is possible to give the input device model natural motion that takes the mass of the input device into consideration. In other words, it is possible to bring the change in the state of the input device indicated by the estimated value closer to the actual change in the state of the input device accompanying the input operation. As a result, it is possible to suppress the discomfort felt by the user during the input operation.

The input device may be a writing instrument that includes a pen tip at one end in the axial direction. The estimation unit may estimate an estimated value of the position of the pen tip and an estimated value of the angle in the axial direction as the estimated values. In this case, it is possible to accurately acquire information indicated by the trajectory of the pen tip as handwriting information, while suppressing the discomfort felt by the user during the writing operation.

The information input system may further include a setting unit that sets a virtual plane in a space where the input device is placed, and a recognition unit that recognizes a trajectory of the estimated value with respect to the virtual plane. In this case, the user's writing motion on the virtual surface can be recognized.

The estimation unit may further include a monitoring unit that monitors changes in the estimated value for the virtual surface, and an adjustment unit that adjusts the estimated value for the virtual surface. With this configuration, when the change in the estimated value with respect to the virtual plane is large, the estimated value can be adjusted so that the change in the estimated value is small. Thereby, it is possible to suppress a situation in which the estimated value deviates significantly from the virtual plane. As a result, it becomes possible to more reliably recognize the locus of the estimated value with respect to the virtual surface.

The monitoring unit includes a calculation unit that calculates an amount of change in the estimated value from the reference value using a predetermined state of the input device with respect to the virtual surface as a reference value, and a calculation unit that determines whether the amount of change is within a first tolerance range. A determination unit may also be included. The adjustment unit improves the simulation execution result by making the disturbance applied to the input device model smaller than the disturbance calculated by the disturbance calculation unit when the determination unit determines that the amount of change is not within the first tolerance range. The amount of change in the estimated value shown may be reduced. With this configuration, when the estimated value changes significantly from the reference value, by reducing the disturbance applied to the input device model, it is possible to reduce changes in the state of the input device model caused by execution of the simulation. Thereby, the movement of the estimated value indicated by the simulation execution result can be reduced. As a result, it is possible to suppress a situation in which the estimated value deviates significantly from the virtual plane, and it becomes possible to more reliably recognize the trajectory of the estimated value with respect to the virtual plane.

The determination unit may determine whether the amount of change is within a second tolerance range that is larger than the first tolerance range. The adjustment unit may return the estimated value to the reference value when the determination unit determines that the amount of change is not within the second tolerance range. According to this configuration, when the estimated value changes further from the reference value, by returning the estimated value to the reference value, it becomes possible to more reliably recognize the locus of the estimated value with respect to the virtual plane.

When the determination unit determines that the amount of change is within the second tolerance range and not within the first tolerance range, the adjustment unit adjusts the amount of change to the disturbance calculated by the disturbance calculation unit as the amount of change deviates from the first tolerance range. The reduction rate of the disturbance applied to the input device model may be increased. In this configuration, the larger the amount of change in the estimated value from the reference value, the smaller the movement of the estimated value indicated by the simulation execution result. This allows the user to recognize that the estimated value deviates from the reference value.

According to one embodiment of the present disclosure, it is possible to improve the accuracy of input information while suppressing the discomfort that a user feels during input operations.

FIG. 1 is a diagram illustrating an example of application of an information input system according to an embodiment. FIG. 2 is a diagram showing an example of a hardware configuration related to the information input system. FIG. 3 is a sectional view showing an example of the configuration of a writing instrument included in the information input system. FIG. 4 is a diagram illustrating an example of a functional configuration related to the information input system. FIG. 5 is a diagram showing an example of a virtual plane set by the setting unit. FIG. 6 is a diagram illustrating an example of the configuration of the estimation section. FIG. 7 is a diagram illustrating a part of the configuration of the estimating section in FIG. 6 in more detail. FIG. 8(a) is a diagram showing the positional relationship between the virtual plane and the writing instrument. FIG. 8(b) is a diagram of FIG. 8(a) viewed from another angle. FIG. 9 is a diagram showing an example of the configuration of the recognition section. FIG. 10 is a diagram showing how drawing points are set on a virtual plane. FIGS. 11(a) and 11(b) are diagrams showing an example of an image of the calibration process. FIGS. 12(a) and 12(b) are diagrams showing an example of the image of the calibration process. FIG. 13 is a diagram showing a part of the configuration of the recognition unit shown in FIG. 8 in more detail. FIG. 14 is a diagram showing an example of an image in which handwritten information is extracted. FIG. 15(a) is a diagram showing an example of current handwriting information. FIG. 15(b) is a diagram showing an example of past handwriting information. FIG. 16 is a flowchart showing an example of the processing contents of the information input method implemented in the information input system. FIG. 17 is a flowchart illustrating an example of estimation processing. FIG. 18 is a flowchart illustrating an example of feedback processing. FIG. 19 is a flowchart illustrating an example of drift correction processing. FIG. 20 is a flowchart illustrating an example of recognition processing. FIG. 21 is a flowchart illustrating an example of adjustment processing. FIG. 22 is a flowchart illustrating an example of the appraisal process.

[Information input system configuration]
FIG. 1 is a diagram showing an example of application of the information input system 1 according to the present embodiment. The information input system 1 is a system for inputting handwriting information D4 (an example of input information) regarding a writing action by the user U to a display screen of an electronic device. The information input system 1 includes, for example, a terminal 10, a writing instrument 20 (an example of an input device), and a database 30. The writing instrument 20 is a tool used for writing characters, symbols, illustrations, and the like. The writing instrument 20 may be a pen capable of writing using ink or graphite, such as a ballpoint pen, fountain pen, marker, or mechanical pencil, or may be a pointing device such as a stylus pen. The user U may be a scribe who writes using the writing instrument 20.

The writing instrument 20 is connected to the terminal 10 by short-range wireless communication. The short-range wireless communication may be, for example, a communication method such as Bluetooth (registered trademark) or Wi-Fi (registered trademark). The communication method between the writing instrument 20 and the terminal 10 is not limited. In this embodiment, a case is illustrated in which one writing instrument 20 communicates with the terminal 10, but the number of writing instruments 20 is not limited. For example, two or more writing instruments 20 may communicate with one terminal 10.

The terminal 10 is a computer used by a scribe. The terminal 10 is, for example, a high-performance mobile phone (smartphone), a tablet terminal, a wearable terminal (for example, a head mounted display (HMD), smart glasses, or a smart watch), a laptop personal computer, or a mobile phone. Good as a mobile device. The terminal 10 may be a stationary terminal such as a desktop personal computer. The terminal 10 is connected to a database 30 via a communication network. The communication network may include, for example, the Internet or an intranet.

The database 30 is a non-temporary storage device that stores various data used by the information input system 1. In this embodiment, the database 30 records threshold information D11, past handwriting information D12, and a writing instrument model D13 (an example of an input device model). The database 30 may store threshold information D11 and past handwriting information D12 regarding a plurality of users U, for example. The database 30 may store multiple types of writing instrument models D13. The database 30 may be constructed as a single database or may be a collection of multiple databases.

FIG. 2 is a diagram showing an example of a hardware configuration related to the information input system 1. FIG. 2 shows a terminal computer 100 functioning as a terminal 10. As shown in FIG. The terminal computer 100 includes, for example, a processor 101, a main storage section 102, an auxiliary storage section 103, a communication section 104, an input interface 105, and an output interface 106 as hardware components. Processor 101 is a computing device that executes an operating system and application programs. The processor 101 may be, for example, a CPU or a GPU, but the type of processor 101 is not limited thereto.

The main storage unit 102 is a device that stores programs for implementing the terminal 10, calculation results output from the processor 101, and the like. The main storage unit 102 includes, for example, at least one of a ROM and a RAM. The auxiliary storage unit 103 is generally a device that can store a larger amount of data than the main storage unit 102. The auxiliary storage unit 103 is configured by a nonvolatile storage medium such as a hard disk or flash memory. The auxiliary storage unit 103 stores a client program P1 for causing the terminal computer 100 to function as the terminal 10 and various data. The communication unit 104 is a device that performs data communication with other computers via a communication network. The communication unit 104 is configured by, for example, a network card or a wireless communication module.

The input interface 105 is a device that receives data based on user U's operations or actions. For example, the input interface 105 is configured by at least one of a keyboard, an operation button, a pointing device, a microphone, a sensor, and a camera. The keyboard and operation buttons may be displayed on the touch panel. The data input to the input interface 105 is not limited. For example, input interface 105 may accept input or selected data via a keyboard, operating buttons, or pointing device. Alternatively, the input interface 105 may accept audio data input through a microphone. Alternatively, the input interface 105 may accept image data (eg, video data or still image data) captured by a camera.

The output interface 106 is a device that outputs data processed by the terminal computer 100. For example, the output interface 106 is configured by at least one of a monitor, a touch panel, an HMD, and a speaker. Display devices such as monitors, touch panels, and HMDs display processed data on screens. The speaker outputs audio indicated by the processed audio data.

Each functional element of the terminal 10 is realized by loading a client program P1, which is an example of the information input system 1, into the processor 101 or the main storage unit 102, and causing the processor 101 to execute the program. The client program P1 includes codes for realizing each functional element of the terminal 10. The processor 101 operates the communication unit 104, input interface 105, or output interface 106 in accordance with the client program P1, and reads and writes data in the main storage unit 102 or the auxiliary storage unit 103. Through this processing, each functional element of the terminal 10 is realized.

The client program P1 may be provided after being recorded non-temporarily on a tangible recording medium such as a CD-ROM, a DVD-ROM, or a semiconductor memory. Alternatively, the client program P1 may be provided via a communication network as a data signal superimposed on a carrier wave.

FIG. 3 is a schematic diagram showing an example of the configuration of the writing instrument 20. FIG. 3 shows a cross section of the writing instrument 20 taken along the axial direction L. As shown in FIG. In this embodiment, a case where the writing instrument 20 is a ballpoint pen will be exemplified. As shown in FIG. 3, the writing instrument 20 includes, for example, a cylinder portion 201, a refill 203, a substrate 206, a pressure sensor 207, and a motion sensor 208 (an example of a sensor).

The cylindrical portion 201 may be a substantially cylindrical member extending along the axial direction L of the writing instrument 20. The cylindrical portion 201 has an opening 202 at the tip in the axial direction L. The refill 203 may be a cylindrical member filled with ink. The refill 203 has an outer diameter smaller than the inner diameter of the cylindrical portion 201 and is housed inside the cylindrical portion 201 . When the refill 203 is housed in the cylindrical portion 201, the tip of the refill 203 (hereinafter referred to as the “pen nib 204”) is exposed from the opening 202 of the cylindrical portion 201. The substrate 206 is housed inside the cylindrical portion 201 behind the base end 205 of the refill 203 (that is, the end opposite to the pen tip 204 in the axial direction L). When the scribe holds the cylinder portion 201 and presses the pen tip 204 against the medium, the ink inside the refill 203 oozes out from the pen tip 204. Therefore, the scribe can write by moving the writing instrument 20 while pressing the pen tip 204 against the medium.

The pressure sensor 207 is provided, for example, inside the cylindrical portion 201 between the base end 205 of the refill 203 and the substrate 206. The pressure sensor 207 detects the pressure that the pen tip 204 receives from the medium when writing with the writing instrument 20 as writing pressure. However, when a writing operation is performed in the air, no writing pressure is generated on the writing instrument 20, so the pressure sensor 207 does not detect the writing pressure. In this embodiment, it is assumed that a writing operation is performed in the air, and a description of the process by which the pressure sensor 207 detects the writing pressure will be omitted.

The motion sensor 208 is provided, for example, on the substrate 206 inside the cylindrical portion 201. The motion sensor 208 detects the motion of the writing instrument 20. For example, the motion sensor 208 may be a six-axis sensor that includes an acceleration sensor that detects acceleration in three axes that are orthogonal to each other, and a gyro sensor that detects angular velocity in the three axes. In this case, the motion sensor 208 detects observed values of acceleration in three axial directions and observed values of angular velocity in three axial directions when writing with the writing instrument 20 .

The writing instrument 20 includes, for example, a communication section 21 and an acquisition section 22 as functional elements. The acquisition unit 22 acquires observed values of acceleration and angular velocity detected by the motion sensor 208 during a writing operation, either continuously or intermittently at a predetermined frequency. The communication unit 21 transmits the observed values of acceleration and angular velocity acquired by the acquisition unit 22 to the terminal 10. As described above, the communication unit 21 is capable of communicating with the terminal 10, for example, by short-range wireless communication such as Bluetooth (registered trademark). The communication unit 21 and the acquisition unit 22 may be incorporated into the writing instrument 20 as a simple computer device.

FIG. 4 is a diagram showing an example of a functional configuration related to the information input system 1. The terminal 10 includes, as functional elements, a communication section 11, an acquisition section 12, a display section 13 (an example of a display device), a setting section 14, an estimation section 15, a recognition section 16, and an adjustment section 17. , an appraisal section 18, and an output section 19. The communication unit 11 receives information transmitted from the communication unit 21 of the writing instrument 20. The information transmitted from the communication unit 21 of the writing instrument 20 includes, for example, observed values of acceleration and angular velocity acquired by the acquisition unit 22 of the writing instrument 20. The acquisition unit 12 acquires the observed values of acceleration and angular velocity that the communication unit 11 receives.

The display unit 13 displays, for example, a virtual object associated with real space (ie, world coordinate system). The display unit 13 may be realized, for example, by an information processing terminal including an HMD worn by the user U, or may be realized by a tablet terminal, projection mapping, or the like. The display unit 13 captures an image in the same imaging direction as the line of sight of the user U, and displays the captured image with the virtual space superimposed thereon. Through the display unit 13, the user U can view virtual objects that do not exist in reality, which correspond to the arrangement of real objects arranged in real space.

The display unit 13 displays virtual objects using a virtual space that is three-dimensional coordinates. The display unit 13 places a virtual object at a preset position on the virtual space, and calculates the correspondence between the virtual space and the real space. In the following explanation, the coordinate system in the virtual space will be referred to as the virtual coordinate system α. The display unit 13 displays an image of the virtual object as viewed from a position and direction in virtual space that respectively correspond to the imaging position and imaging direction in real space. That is, the display unit 13 displays an image of the virtual object viewed from the line of sight of the virtual user U. The display of the virtual object described above can be performed using conventional MR (Mixed Reality) technology.

The setting unit 14 sets a virtual surface S in the virtual space. The virtual surface S may be a virtual writing surface for recognizing the writing motion of the writing instrument 20 by the user U. The setting unit 14 sets the virtual plane S at an arbitrary position in the virtual space. The virtual plane S set by the setting unit 14 is displayed in the virtual space by the display unit 13. FIG. 5 is a diagram showing an example of the virtual surface S. As shown in FIG. 5, the virtual surface S may be, for example, a two-dimensional virtual plane set at an arbitrary position in the virtual coordinate system α. Hereinafter, the coordinate system that defines the virtual surface S will be referred to as the screen coordinate system β.

The virtual surface S is not limited to the example shown in FIG. 5, and may be a virtual curved surface that changes three-dimensionally, for example. In this case, the virtual surface S can be set on the surface of any three-dimensional virtual object displayed in the virtual space. The type of virtual surface S is determined by receiving input from user U, for example. Therefore, when receiving an input from the user U, the setting unit 14 sets the virtual plane S corresponding to the input in the virtual space. The setting unit 14 does not need to set the virtual plane S in response to the input from the user U, and may set the virtual plane S in advance at a fixed position in the virtual space, for example. The virtual surface S does not need to be fixed in the virtual space, and may be set to move in the virtual space.

FIG. 6 is a diagram showing an example of the configuration of the estimation unit 15. The estimation unit 15 estimates an estimated value of the state of the writing instrument 20 in the virtual space. The state of the writing instrument 20 in the virtual space may be the position and orientation of the writing instrument 20 arranged in the virtual space. Therefore, the estimated value of the state of the writing instrument 20 may be the estimated value of the position of the writing instrument 20, or the estimated value of the posture of the writing instrument 20. The estimated value of the position of the writing instrument 20 may be, for example, the estimated value of the position of the pen tip 204 (see FIG. 5).

The estimated value of the position of the pen tip 204 is represented by coordinate values of the virtual coordinate system α indicating the position of the pen tip 204. The estimated value of the posture of the writing instrument 20 may be, for example, an estimated value of the angle in the axial direction L that changes as the writing instrument 20 moves. The estimated value of the angle of the writing instrument 20 may be, for example, the current angle of the axial direction L of the writing instrument 20 with respect to the initial angle of the axial direction L of the writing instrument 20 in the virtual coordinate system α. Hereinafter, the observed value of acceleration and the observed value of angular velocity will be collectively referred to as "observed value D10", and the estimated value of the position of writing instrument 20 and the estimated value of the posture of writing instrument 20 will be collectively referred to as "estimated value D20". .

The estimation unit 15 estimates the estimated value D20 by executing a simulation of the operation of the writing instrument 20 using a writing instrument model D13 that models the writing instrument 20. As shown in FIG. 6, the estimation unit 15 includes, for example, a model processing unit 151, an analysis unit 152, an estimated value derivation unit 153, and a feedback unit 154. The model processing unit 151 includes, for example, a model acquisition unit 155 and an initialization processing unit 156. The model acquisition unit 155 acquires the writing instrument model D13 from the database 30. The writing instrument model D13 is, for example, CAD data of the writing instrument 20, and is divided into a finite number of elements that can be numerically analyzed. Possible for numerical analysis means that it can be handled by a numerical analysis method such as the finite element method, finite volume method, finite difference method, or boundary element method, for example. The writing instrument model D13 does not need to be stored in the database 30 in advance, and may be generated by the terminal 10.

Model information D21 indicating the specifications of the writing instrument 20 is given to the writing instrument model D13. The specifications of the writing instrument 20 may include, for example, the shape, dimensions, material, and mass of the writing instrument 20. These pieces of information are given to each element constituting the writing instrument model D13 and are reflected in the writing instrument model D13. The information indicating the mass of the writing instrument 20 may be, for example, mass information indicating the distribution of mass for each part of the writing instrument 20. The mass information is given to the writing instrument model D13 as information indicating the distribution of mass for each element of the writing instrument model D13. By referring to the mass information, it is possible to determine which parts have a higher mass and which parts have a lower mass in the writing instrument model D13. The thickness of the pen tip 204 can be determined from the information indicating the shape and dimensions of the writing instrument 20. The model information D21 may include color information indicating the color of the pen tip 204. The model information D21 may be provided to the writing instrument model D13 in advance, or may be provided to the writing instrument model D13 at the time of simulation execution.

The initialization processing unit 156 initializes the state (for example, position and orientation) of the writing instrument model D13. Initializing means setting the position and orientation of the writing instrument model D13 in the simulation space to initial values. The initial value of the position of the writing instrument model D13 may be any coordinate value of the coordinate system in the simulation space. The initial value of the angle of the writing instrument model D13 may be an angle from an arbitrary reference line of the coordinate system in the simulation space.

The coordinate system in the simulation space does not necessarily have to match the virtual coordinate system α in the virtual space, but is associated with the virtual coordinate system α. The coordinate values of the coordinate system in the simulation space can be converted to the coordinate values of the virtual coordinate system α. The initialization processing unit 156 initializes the position and orientation of the writing instrument model D13 at the timing when the model acquisition unit 155 acquires the writing instrument model D13, or at any timing when initialization is required. The model processing unit 151 provides the analysis unit 152 with the writing instrument model D13 initialized by the initialization processing unit 156 and the model information D21 given to the writing instrument model D13.

The analysis unit 152 includes, for example, a disturbance calculation unit 157 and an analysis execution unit 158. The disturbance calculation unit 157 uses the observed value D10 and the model information D21 to calculate a disturbance to be applied to the writing instrument model D13. The disturbance may be, for example, a force and torque applied to the writing instrument 20 by a user during operation of the writing instrument 20. The disturbance calculation unit 157 calculates the force and torque to be applied to each element of the writing instrument model D13 from the acceleration and angular velocity indicated by the observed value D10 and the mass information indicated by the model information. The force F to be applied to the element of the writing instrument model D13 is expressed as in the following equation (1), where m is the mass of the element and Sa is the acceleration. In equation (1), A is an arbitrary function defined to derive force F from mass m and acceleration Sa. The disturbance calculation unit 157 calculates torque based on the force F. The disturbance calculation section 157 provides the calculated disturbance as a disturbance condition D22 to the analysis execution section 158 together with the writing instrument model D13 and model information D21.

The analysis execution unit 158 executes a simulation of the operation of the writing instrument 20 using the writing instrument model D13. The analysis execution unit 158 sets the disturbance condition D22 as an analysis condition for executing a simulation of the operation of the writing instrument 20. The analysis execution unit 158 causes the writing instrument model D13 to operate in the simulation space by applying a disturbance condition D22 to the writing instrument model D13. As a result, the position and orientation of the writing instrument model D13 in the simulation space change. When the simulation is executed, the force F and torque included in the disturbance condition D22 and the mass information included in the model information D21 are given to each element of the writing instrument model D13.

Here, since the force F is proportional to the mass m, a large force F is applied to an element with a high mass m, and a small force F is applied to an element with a low mass m. For example, when the mass m near the pen tip 204 is higher than other parts, a relatively large force F is applied to the pen tip 204, and the amount of change in the position of the pen tip 204 in the simulation space becomes large. On the other hand, when the mass m near the pen tip 204 is lower than other parts, a relatively small force F is applied to the pen tip 204, and the amount of change in the position of the pen tip 204 in the simulation space becomes small.

As a result, the motion of the writing instrument model D13 in the simulation space becomes a natural motion that takes into account the mass m of the writing instrument 20, that is, a motion that is close to the motion of the actual writing instrument 20. By executing such a simulation, in the simulation space, the position and orientation of the writing instrument model D13 after execution of the simulation change from the position and orientation of the writing instrument model D13 before execution of the simulation. The analysis execution unit 158 provides the simulation execution result to the estimated value derivation unit 153. The execution result of the simulation may be a result indicating the position and orientation of the writing instrument model D13 after execution of the simulation, or a result indicating the amount of change in the position and orientation of the writing instrument model D13 caused by execution of the simulation. Good too.

The estimated value deriving unit 153 derives the estimated value D20 of the current position and orientation of the writing instrument 20 from the simulation execution result. For example, the estimated value deriving unit 153 derives the estimated value D20 by associating the position and orientation of the writing instrument model D13 in the simulation space with the position and orientation of the writing instrument 20 in the virtual space. Associating the position and orientation of the writing instrument model D13 with the position and orientation of the writing instrument 20 involves converting the position and orientation of the writing instrument model D13 in the coordinate system of the simulation space to the position and orientation of the writing instrument model D13 in the virtual coordinate system α, respectively. , means that the position and orientation of the converted writing instrument model D13 are respectively regarded as the position and orientation of the writing instrument 20 in the virtual space.

In this way, the estimated value deriving unit 153 utilizes the correspondence between the coordinate system in the simulation space and the virtual coordinate system α in the virtual space, for example, as indicated by the simulation execution result (i.e., after the simulation execution). By reflecting the position and orientation of the writing instrument model D13 on the current position and orientation of the writing instrument 20 in the virtual space, an estimated value D20 of the current position and orientation of the writing instrument 20 is derived.

The estimated value deriving unit 153 converts the amount of change in the position and orientation of the writing instrument model D13 caused by the execution of the simulation into the amount of change in the position and orientation of the writing instrument model D13 in the virtual space, and calculates the amount of change in the position and orientation of the writing instrument model D13 after the conversion. The amount of change in position and orientation may be added to the position and orientation of the writing instrument 20 before the operation in the virtual space (that is, before the simulation is executed). In this case, the estimated value deriving unit 153 derives the position and orientation of the writing instrument 20 after the amount of change has been added as the estimated value D20 of the current position and orientation of the writing instrument 20, respectively. The estimated value deriving unit 153 provides the estimated value D20 to the recognizing unit 16 and the feedback unit 154, respectively.

In the simulation space, the position of the pen tip 204 of the writing instrument model D13 can be determined using the observed value D10 from the motion sensor 208. The observed value D10 from the motion sensor 208 indicates not the position of the pen tip 204 but the acceleration and angular velocity at the position where the motion sensor 208 is provided. By integrally calculating at least one of these accelerations and angular velocities, the position of the motion sensor 208 is determined. The position of the motion sensor 208 can be converted to the position of the pen tip 204 using, for example, the following equation (2).

In equation (2), VS is a position vector indicating the position of the motion sensor 208, VS ₀ is a position vector indicating the initial value of the position of the motion sensor 208, VH is a position vector indicating the position of the pen tip 204, and VH ₀ is the position vector of the pen tip 204. RS is a rotation vector indicating the rotation of the _motion sensor 208, RS is a rotation vector indicating the initial value of the rotation of the motion sensor 208, and Rot is obtained from the angle between VH and VS. The transformation rotation matrix shown is shown below. Since the position of the pen tip 204 with respect to the motion sensor 208 does not change, (VH-VS) is always constant (see FIG. 5). In Equation (2), VS, RS, and RS ₀ are obtained from the acceleration and angular velocity indicated by the observed value D10. Therefore, the estimated value deriving unit 153 can estimate the position of the pen tip 204 and the orientation of the writing instrument 20 from the observed value D10 using equation (2).

FIG. 7 is a diagram showing an example of the configuration of the feedback section 154. The feedback unit 154 monitors the estimated value D20 of the current position and orientation of the writing instrument 20 in the virtual coordinate system α, and feeds back the monitoring results to the analysis unit 152. As shown in FIG. 7, the feedback unit 154 includes, for example, a monitoring unit 141 and an adjustment unit 142. The monitoring unit 141 monitors the amount of change in the temporary current position and orientation of the writing instrument 20 in the virtual coordinate system α. The monitoring unit 141 includes, for example, a calculation unit 143 and a determination unit 144. The calculation unit 143 uses the predetermined position and orientation of the writing instrument 20 with respect to the virtual plane S as a reference value, and calculates the amount of change in the estimated value D20 from the reference value.

The reference value may be the initial value of the position and orientation of the writing instrument 20 in the virtual coordinate system α. The initial values may be, for example, the position and orientation of the writing instrument 20 in a state in which the tip of the pen tip 204 faces the center Sc of the virtual plane S in the virtual coordinate system α. The point of the pen nib 204 facing the center Sc of the virtual surface S may be defined as a virtual line extending from the pen nib 204 intersecting the center Sc of the virtual surface S. If the tip of the pen nib 204 faces the center Sc of the virtual surface S, the axial direction L of the writing instrument 20 may be inclined with respect to the normal direction of the virtual surface S, for example, or may be inclined with respect to the normal direction of the virtual surface S. It may be along the direction. The initial state is not limited to a state in which the tip of the pen tip 204 faces the center Sc of the virtual surface S, but also a state in which the tip of the pen tip 204 points at a position offset from the center Sc of the virtual surface S. It's okay.

FIG. 8(a) is a diagram showing the positional relationship between the virtual surface S and the writing instrument 20. FIG. 8(b) is a diagram of FIG. 8(a) viewed from another angle. FIGS. 8A and 8B show the writing instrument 20A indicated by the reference value and the current writing instrument 20 indicated by the estimated value D20. The amount of change in the estimated value D20 from the reference value can be expressed, for example, by taking the inner product of the unit direction vector V1 toward which the pen tip 204 of the writing instrument 20A is directed and the unit direction vector V2 toward which the pen tip 204 of the writing instrument 20A is directed. I can do it. By taking the inner product of the unit directional vector V1 and the unit directional vector V2, the angle θ formed by the unit directional vector V1 and the unit directional vector V2 can be calculated.

The larger the amount of change in the estimated value D20 from the reference value, the larger the angle θ formed by the unit direction vector V1 and the unit direction vector V2, and the smaller the amount of change in the estimated value D20 from the reference value, the larger the angle θ becomes. becomes smaller. Therefore, the calculation unit 143 calculates the angle θ formed by the unit direction vector V1 and the unit direction vector V2 as an index indicating the amount of change in the estimated value D20 from the reference value. The calculation unit 143 provides the determination unit 144 with a calculation result D41 indicating the calculated angle θ.

The determination unit 144 determines whether the amount of change (for example, angle θ) indicated by the calculation result D41 is within an allowable range. The determination unit 144 first sets a first region R1 and a second region R2 on the virtual surface S. As shown in FIGS. 8A and 8B, the first region R1 may be, for example, a region inside the outer edge of the virtual surface S and including at least the center Sc. The boundary of the first region R1 may be located midway between the outer edge of the virtual surface S and the center Sc, or may be shifted to the inside or outside of the midpoint. The second region R2 may be larger than the first region R1. The second region R2 may be, for example, a region inside the outer edge of the virtual surface S.

The determining unit 144 sets the range of the angle θ formed by the unit directional vector V2 with respect to the unit directional vector V1 to a first tolerance range when the intersection PV2 of the unit directional vector V2 and the virtual plane S is located in the first region R1. Set. The fact that the angle θ is within the first tolerance range means that the intersection PV2 is within the second region R2. The determination unit 144 sets the range of angle θ when the intersection PV2 is located in the second region R2 as a second tolerance range. The fact that the angle θ is within the second tolerance range means that the intersection PV2 between the unit direction vector V2 and the virtual surface S is within the second region R2. The second tolerance range is a range of angles θ larger than the first tolerance range.

The determining unit 144 first determines whether the angle θ is within the second tolerance range. When determining that the angle θ is not within the second tolerance range, that is, when determining that the intersection PV2 between the unit direction vector V2 and the virtual surface S is not within the second region R2, the determining unit 144 determines that the angle θ is not within the second tolerance range. The adjustment unit 142 is provided with a determination result D42 indicating that the value is not within the second allowable range. On the other hand, when determining that the angle θ is within the second tolerance range, that is, when determining that the intersection PV2 is within the second region R2, the determination unit 144 determines that the angle θ is within the first tolerance range. Determine whether it exists or not. When determining that the angle θ is not within the first tolerance range, that is, when determining that the intersection PV2 is not within the first region R1, the determination unit 144 determines that the angle θ is within the second tolerance range. The adjustment unit 142 is provided with a determination result D42 indicating that the value is not within the allowable range. On the other hand, when the determination unit 144 determines that the angle θ is within the first tolerance range, the determination unit 144 repeats the above determination again for the angle θ obtained based on the next estimated value D20.

The adjustment unit 142 adjusts the estimated value D20 of the current position and orientation of the writing instrument 20 in the virtual coordinate system α based on the monitoring result by the monitoring unit 141 (specifically, the determination result D42 by the determination unit 144). When the adjustment unit 142 receives the determination result D42 indicating that the angle θ is not within the second tolerance range, the adjustment unit 142 performs a calibration process to return the estimated value D20 of the current position and orientation of the writing instrument 20 in the virtual coordinate system α to the reference value. Execute. For example, the adjustment unit 142 adjusts the position and orientation of the writing instrument model D13 in the simulation space so that the estimated value D20 becomes the reference value (that is, so that the pen tip 204 of the writing instrument 20 faces the center Sc of the virtual surface S). adjust.

For example, the adjustment unit 142 calculates the disturbance necessary to move the unit direction vector V2 of the writing instrument 20 indicated by the estimated value D20 to the unit direction vector V1 of the writing instrument 20A indicated by the reference value, and analyzes the calculated disturbance. Department 158. This disturbance can be set by adjusting the function A of the above-mentioned equation (1). The analysis execution unit 158 changes the position and orientation of the writing instrument model D13 in the simulation space by applying the disturbance provided from the adjustment unit 142 to the writing instrument model D13. Then, the adjustment unit 142 reflects the position and orientation of the writing instrument model D13 after execution of the simulation on the current position and orientation of the writing instrument 20 in the virtual space. Thereby, the adjustment unit 142 returns the estimated value D20 of the current position and orientation of the writing instrument 20 to the reference value.

When the adjustment unit 142 receives the determination result D42 indicating that the angle θ is within the second tolerance range and not within the first tolerance range, the adjustment unit 142 adjusts the amount of change in the estimated value D20 with respect to the reference value. The amount of change in the estimated value D20 corresponds to the amount of change in the position and orientation of the writing instrument model D13 caused by execution of the simulation. The amount of change in the position and orientation of the writing instrument model D13 depends on disturbances set as analysis conditions when executing the simulation. The disturbance can be adjusted by the magnitude of the function A shown in equation (1).

Therefore, the adjustment unit 142 makes the disturbance function A given to the writing instrument model D13 during simulation execution smaller than the disturbance function A calculated by the disturbance calculation unit 157. The disturbance function A calculated by the disturbance calculation unit 157 is a value calculated based on the observed value D10 and the model information D21. This disturbance may be regarded as a disturbance applied to the actual writing instrument 20. Therefore, by reducing the function A, the disturbance applied to the writing instrument model D13 can be made smaller than the disturbance applied to the actual writing instrument 20.

In other words, the amount of change in the position and orientation of the writing instrument model D13 caused by the execution of the simulation can be made smaller than the amount of change in the actual position and orientation of the writing instrument 20. As a result, the motion of the writing instrument model D13 in the simulation space becomes smaller than the motion of the actual writing instrument 20. Correspondingly, the movement of the estimated value D20 in the virtual space indicated by the simulation execution result becomes smaller. In other words, the movement of the estimated value D20 becomes slower than the actual movement of the writing instrument. This prevents the estimated value D20 from deviating further from the reference value.

When reducing the disturbance applied to the writing instrument model D13, the adjustment unit 142 reduces the disturbance calculated by the disturbance calculation unit 157 as the estimated value D20 deviates from the reference value, that is, the angle θ deviates from the first tolerance range. The function A of the disturbance applied to the writing instrument model D13 is set so that the reduction rate of the function A of the disturbance applied to the writing instrument model D13 at the time of execution of the simulation with respect to the function A becomes large. For example, when the angle θ does not deviate significantly from the first tolerance range (for example, when the angle θ is closer to the upper limit of the first tolerance range than the upper limit of the second tolerance range), the adjustment unit 142 adjusts the writing instrument model D13 to The disturbance function A to be applied to is set to half of the disturbance function A calculated by the disturbance calculation unit 157. In this case, the amount of movement of the estimated value D20 indicated by the execution result of the simulation can be suppressed to about half the amount of movement of the actual writing instrument 20.

On the other hand, if the angle θ is significantly outside the first tolerance range (for example, if the angle θ is closer to the upper limit of the second tolerance range than the upper limit of the first tolerance range), the disturbance imparted to the writing instrument model D13 Set function A to zero. As a result, the disturbance applied to the writing instrument model D13 in the simulation space becomes zero, and the operation of the actual writing instrument 20 is no longer reflected in the operation of the writing instrument model D13. In this case, the estimated value D20 indicated by the execution result of the simulation does not move in the virtual space, so even if the user moves the writing instrument 20 significantly, the estimated value D20 does not move in the virtual space. In this way, by changing the reduction rate of the disturbance applied to the writing instrument model D13 according to the magnitude of the angle θ, the movement of the estimated value D20 is made smaller as the estimated value D20 deviates from the reference value in the virtual space. can. As a result, the user can be made aware that the current position and orientation of the writing instrument 20 deviate greatly from the virtual plane S.

The monitoring unit 141 may monitor whether or not to perform drift correction on the estimated value D20. In this case, the monitoring unit 141 sets a drift correction condition that requires drift correction of the estimated value D20. A detection error generally referred to as a drift error may occur in the motion sensor 208. When such errors accumulate, a phenomenon occurs in which the estimated value D20 gradually deviates from the virtual surface S in the virtual space. If such a phenomenon occurs, the center of the distribution of estimated values D20 within a certain time when the distribution of estimated values D20 is reflected on the virtual surface S will be The estimated value D20 gradually shifts from the center of the distribution. The center of the distribution of estimated values D20 within a certain period of time may be the average value or median value of the distribution of estimated values D20 within a certain period of time. When drift errors are not accumulated, the center of the distribution of estimated values D20 within a certain period of time is located near the center Sc of the virtual surface S, for example.

Therefore, the calculation unit 143 calculates the shift amount of the center of the distribution of the current estimated value D20 within a certain period of time, using the center of the distribution of the estimated value D20 within a certain period of time as a reference value when drift errors have not accumulated. You may. In this case, the determination unit 144 determines that the drift correction condition is satisfied when the calculated amount of deviation exceeds a preset threshold. On the other hand, the determination unit 144 determines that the drift correction condition is not satisfied when the calculated amount of deviation does not exceed the threshold. The above “threshold” can be obtained from the threshold information D11 stored in the database 30. The above threshold value may be set in advance by, for example, the information input system 1, the provider of the information input system 1, or the user. The above threshold value may be set, for example, based on a past statistical value indicating the center of the distribution of the estimated value D20 within a certain period of time, or may be set arbitrarily by the user.

If the determining unit 144 determines that the drift correction condition is satisfied, the adjusting unit 142 executes a calibration process to return the estimated value D20 to the reference value. For example, the adjustment unit 142 adjusts the position and orientation of the writing instrument model D13 in the simulation space so that the estimated value D20 becomes the reference value (that is, so that the pen tip 204 of the writing instrument 20 faces the center Sc of the virtual surface S). adjust. For example, the adjustment unit 142 performs the same process as the calibration process described above. Thereby, the center of the distribution of the current estimated value D20 within a certain period of time can be returned to the vicinity of the center Sc of the virtual surface S. On the other hand, if the determination unit 144 determines that the drift correction condition is not satisfied, the determination unit 144 repeats the above determination again for the next estimated value D20.

FIG. 9 is a diagram showing an example of the configuration of the recognition unit 16. The recognition unit 16 recognizes the trajectory of the writing instrument 20 based on the estimated value D20 estimated by the estimation unit 15. As shown in FIG. 9, the recognition unit 16 includes, for example, a virtual line setting unit 161, a drawing point setting unit 162, and a writing information acquisition unit 163. The virtual line setting unit 161 sets a virtual line VL (for example, a virtual ray) in the virtual space. FIG. 10 is a diagram showing how the drawing point P is set on the virtual surface S by the virtual line VL. The virtual line VL may be, for example, a straight line virtually set to set the drawing point P on the virtual surface S. As shown in FIG. 10, the virtual line VL extends linearly from the pen tip 204 of the writing instrument 20 toward the virtual surface S, for example.

The drawing point P may be an input point set on the virtual surface S in order to recognize the writing operation of the writing instrument 20 on the virtual surface S. The estimated value D20 indicates the coordinate value of the position of the pen tip 204 in the virtual coordinate system α and the attitude of the writing instrument 20 in the virtual coordinate system α (that is, the angle in the axial direction L). Therefore, the virtual line setting unit 161 uses the estimated value D20 to set a straight line passing through the position of the pen tip 204 indicated by the estimated value D20 and along the axial direction L of the writing instrument 20 as the virtual line VL. The position of the pen tip 204 indicated by the estimated value D20 may be, for example, a coordinate point (that is, an input point) indicating the position of the pen tip 204 in the virtual coordinate system α.

The posture of the writing instrument 20 in the virtual coordinate system α can be expressed by a direction vector V along the axial direction L of the writing instrument 20. For example, let the coordinate values of the origin of the screen coordinate system β in the virtual coordinate system α be (x ₁ , y ₁ , z ₁ ), and the coordinate values of the position of the pen tip 204 in the virtual coordinate system α be (x ₂ , y ₂ , z ₂ ), and the direction vector V is (Vx, Vy, Vz). In this case, the virtual line VL is represented as a straight line along a direction vector (Vx, Vy, Vz) passing through the coordinate values (x ₂ , y ₂ , z ₂ ).

For example, the virtual line VL is expressed by the following equation (3), where t is an arbitrary real number (that is, a parameter). The virtual line setting unit 161 sets a virtual line VL passing through the position of the pen tip 204 and along the axial direction L of the writing instrument 20 using the estimated value D20 and equation (3). The virtual line VL extends linearly from the pen tip 204 in the direction toward which the pen tip 204 faces, and intersects the virtual surface S. The virtual line setting section 161 provides the drawing point setting section 162 with virtual line information D1 indicating the virtual line VL.

The drawing point setting unit 162 sets, as a drawing point P, the intersection between the virtual line VL indicated by the virtual line information D1 and the virtual plane S set in the virtual space. When the normal vector of the virtual surface S in the virtual coordinate system α is (Nx, Ny, Nz), the plane equation representing the virtual surface S is expressed by the following equation (4). Therefore, the drawing point setting unit 162 calculates the intersection between the virtual line VL and the virtual surface S by substituting equation (3) representing the virtual line VL into equation (4) representing the virtual surface S. For example, by substituting equation (3) into equation (4), equation (5) is obtained. By solving equation (5) for t, equation (6) is obtained. Then, by substituting equation (6) into equation (3), the intersection between the virtual surface S and the virtual line VL can be found.

The drawing point setting unit 162 sets the intersection of the virtual plane S and the virtual line VL as the drawing point P. For example, the drawing point setting unit 162 sets the position of the drawing point P as a coordinate value in the screen coordinate system β. The drawing point setting section 162 provides coordinate information D2 including the coordinate values of the position of the drawing point P to the adjustment section 17 and the writing information acquisition section 163. The coordinate information D2 may include the coordinate values of the position of the pen tip 204 in addition to the coordinate values of the position of the drawing point P.

Refer to FIG. 9 again. The adjustment unit 17 refers to the coordinate information D2 and determines whether it is necessary to adjust the positions of the virtual plane S and the writing instrument 20 in the virtual coordinate system α. When the adjustment unit 17 determines that the position between the virtual surface S and the writing instrument 20 needs to be adjusted, it performs an adjustment process to adjust the position between the virtual surface S and the writing instrument 20. The adjustment process may include, for example, a process similar to the feedback process by the feedback unit 154 described above. The adjustment unit 17 may perform the adjustment process together with the feedback process described above, or may perform the adjustment process when the feedback process is not performed. Alternatively, the adjustment unit 17 does not need to perform the adjustment process when the feedback process is performed.

As shown in FIG. 8(a), the adjustment unit 17 first sets a first region R1 and a second region R2 on the virtual surface S, similarly to the feedback process. Then, the adjustment unit 17 determines whether the drawing point P is located within the second region R2 of the virtual surface S. That is, the adjustment unit 17 determines whether the drawing point P falls within the virtual plane S or not. If the adjustment unit 17 determines that the drawing point P is not located within the second region R2, the adjustment unit 17 determines that the position of the virtual surface S and the writing instrument 20 needs to be adjusted. In this case, the adjustment unit 17 executes a calibration process to return the position of the virtual surface S with respect to the writing instrument 20 to the reference position. The calibration process here is different from the calibration process by the adjustment unit 142 described above.

11(a), FIG. 11(b), FIG. 12(a), and FIG. 12(b) are diagrams showing an example of the image of the calibration process by the adjustment unit 17. In the state shown in FIG. 11(a), the pen tip 204 of the writing instrument 20 is facing the virtual surface S, and the drawing point P is located within the virtual surface S. When this state changes to the state shown in FIG. 11(b), the direction of the pen tip 204 deviates from the virtual surface S, so the drawing point P is not set on the virtual surface S. In such a case, the adjustment unit 17 performs calibration to return the position of the virtual surface S with respect to the writing instrument 20 to the reference position so that the drawing point P is set within the virtual surface S.

The state in which the position of the virtual surface S with respect to the writing instrument 20 is at the reference position may be, for example, a state in which the virtual surface S and the writing instrument 20 are arranged such that the tip of the pen tip 204 faces the center Sc of the virtual surface S. The fact that the tip of the pen nib 204 points toward the center Sc of the virtual surface S means that the virtual line VL extending from the pen nib 204 intersects the center Sc of the virtual surface S (that is, the intersection of the virtual line VL and the virtual surface S). may be located at the center Sc). If the tip of the pen nib 204 faces the center Sc of the virtual surface S, the axial direction L of the writing instrument 20 may be inclined with respect to the normal direction of the virtual surface S, for example, or may be inclined with respect to the normal direction of the virtual surface S. It may be along the direction. The state in which the virtual surface S and the writing instrument 20 are at the reference position is not limited to the state in which the tip of the pen tip 204 faces the center Sc of the virtual surface S, but also the state in which the tip of the pen tip 204 points to the center Sc in the virtual surface S. It may also be in a state where it is facing a position shifted from the original position. That is, as long as the virtual line VL extending from the pen tip 204 intersects the virtual surface S, the intersection of the virtual line VL and the virtual surface S may be offset from the center Sc of the virtual surface S.

When executing the calibration process, the adjustment unit 17 moves at least one of the virtual surface S and the writing instrument 20 in the virtual coordinate system α so that the positions of the virtual surface S and the writing instrument 20 become the reference positions. At this time, the adjustment unit 17 may move only the virtual surface S without moving the writing instrument 20 in the virtual coordinate system α, or may move only the writing instrument 20 without moving the virtual surface S. , both the virtual surface S and the writing instrument 20 may be moved.

In the example shown in FIGS. 12(a) and 12(b), the adjustment unit 17 rotates only the virtual plane S with respect to the writing instrument 20 in the virtual coordinate system α. At this time, as the virtual surface S rotates, the position of the screen coordinate system β that defines the virtual surface S in the virtual coordinate system α also rotates. As a result, the virtual plane S and the screen coordinate system β move to the position shown in FIG. 12(b). In the state shown in FIG. 12(b), the positions of the writing instrument 20 and the virtual surface S are adjusted so that the tip of the pen nib 204 faces the center Sc of the virtual surface S, so the virtual line extending from the pen nib 204 VL intersects the center Sc of the virtual plane S. This completes the calibration process that returns the positions of the virtual plane S and the writing instrument 20 in the virtual coordinate system α to the reference positions.

If the adjustment unit 17 determines that the drawing point P is within the second region R2, it determines whether the drawing point P is within the first region R1. When the determination unit 144 determines that the drawing point P is not within the first region R1, the determination unit 144 performs an adjustment to adjust the amount of change in the estimated value D20 with respect to the reference value using a predetermined position and orientation of the writing instrument 20 with respect to the virtual surface S as a reference value. Perform processing. This adjustment process may be the same process as the adjustment process by the adjustment unit 142 described above. In this case, the adjustment unit 17 makes the disturbance function A given to the writing instrument model D13 during simulation execution smaller than the disturbance function A calculated by the disturbance calculation unit 157.

Thereby, the amount of change in the position and orientation of the writing instrument model D13 caused by execution of the simulation can be made smaller than the amount of change in the actual position and orientation of the writing instrument 20, respectively. As a result, the movement of the estimated value D20 in the virtual space indicated by the simulation execution result becomes smaller. This prevents the estimated value D20 from deviating further from the reference value. On the other hand, when the determination unit 144 determines that the drawing point P is within the first region R1, the determination unit 144 repeats the above determination again for the drawing point P obtained based on the next estimated value D20.

FIG. 13 is a diagram showing an example of the configuration of the handwritten information acquisition section 163. As shown in FIG. 13, the handwritten information acquisition section 163 includes, for example, a trajectory information acquisition section 164 and a handwritten information extraction section 165. The trajectory information acquisition unit 164 includes, for example, a coordinate string recording unit 166 and a distance calculation unit 167. Upon receiving the coordinate information D2 from the drawing point setting section 162, the coordinate string recording section 166 records the coordinate values of the drawing point P in chronological order and also records the coordinate values of the pen tip 204 in chronological order. The coordinate string recording unit 166 records the coordinate values of the drawing point P and the coordinate values of the pen tip 204 in a time-series manner in association with each other. The coordinate string recording unit 166 stores drawing point trajectory information D31 in which the coordinate values of the drawing point P are recorded in time series, and pen tip trajectory information D32 in which the coordinate values of the pen tip 204 are recorded in time series, in the distance calculation unit 167. and provided to the handwritten information extraction unit 165. The drawing point trajectory information D31 may be information indicating a two-dimensional trajectory of the coordinate values of the drawing point P on the virtual surface S. The pen tip trajectory information D32 may be information indicating a three-dimensional trajectory of the coordinate values of the pen tip 204 in the virtual space.

The distance calculation unit 167 calculates the distance between the drawing point P and the pen tip 204 using the drawing point trajectory information D31 and the pen tip trajectory information D32. For example, the distance calculation unit 167 calculates the coordinate values of the drawing point P at a certain time and the coordinate values of the pen tip 204 at that time (i.e., the coordinates of the drawing point P) from the drawing point trajectory information D31 and the pen tip trajectory information D32. (coordinate value of the pen tip 204 associated with the value). The distance calculation unit 167 records the calculated distance in chronological order and associates it with the coordinate values of the drawing point P and the coordinate values of the pen tip 204 in chronological order. The distance calculation unit 167 provides the writing information extraction unit 165 with distance information D33 indicating the distance between the coordinate values of the drawing point P and the coordinate values of the pen tip 204. As a result, the trajectory information acquisition unit 164 provides the writing information extraction unit 165 with information including the drawing point trajectory information D31, the pen tip trajectory information D32, and the distance information D33 as the trajectory information D3 indicating the trajectory of the writing instrument 20. .

The writing information extraction unit 165 extracts writing information D4 indicating the trajectory of the drawing point P that the user U intends to write on the virtual surface S from the drawing point trajectory information D31 while referring to the distance information D33. The locus of the drawing point P indicated by the writing information D4 is expressed as a character written on the virtual surface S using the writing instrument 20, for example. Therefore, the written information D4 may be character information written by the user U, for example. The trajectories of the drawing points P indicated by the drawing point trajectory information D31 are connected like one stroke. Therefore, for example, when the writing information D4 includes character information, from the drawing point trajectory information D31, the trajectory of the drawing point P that the user U intended to write on the virtual surface S, and the trajectory of the drawing point P that the user U intended to write on the virtual surface S. It is necessary to extract the characters written on the virtual surface S by determining the locus of the drawing point P that is not intended to be written on the virtual surface S.

For example, when the writing information D4 includes character information, the trajectory of the drawing point P that the user U intends to write on the virtual surface S is the trajectory of the coordinate values of the drawing point P excluding the breaks in the character stroke. good. The character stroke may be an action of drawing lines constituting a character on the virtual surface S. A break in a character stroke may be a location between any line that makes up a character and the next line. When the user U does not intend to write on the virtual surface S (that is, when the user U performs an action corresponding to a break in a character stroke), when the user U intends to write on the virtual surface S ( That is, compared to the case where the user U performs a character stroke motion, the user U performs a motion of separating the writing instrument 20 from the virtual surface S.

As a result, when the user U operates the writing instrument 20 without intending to write on the virtual surface S, the virtual surface S and the writing instrument The distance from 20 becomes larger. Therefore, the writing information extraction unit 165 refers to the change in the distance between the virtual surface S and the writing instrument 20 and extracts the drawing point P that the user U intends to write on the virtual surface S from the drawing point locus information D31. The trajectory of is extracted as written information D4. For example, the handwriting information extraction unit 165 determines whether the change in the distance between the drawing point P and the pen tip 204 is less than a preset threshold. When the writing information extraction unit 165 determines that the change in the distance between the drawing point P and the pen tip 204 is less than the threshold, the user U can virtually trace the trajectory of the drawing point P associated with the change in distance. It is extracted as handwriting information D4 intended to be written on the surface S.

On the other hand, if the writing information extraction unit 165 determines that the change in the distance between the drawing point P and the pen tip 204 is not less than the threshold, the user U can trace the trajectory of the drawing point P associated with the change in distance on the virtual surface. It is determined that the written information D4 is not intended to be written in S. The above “threshold” can be obtained from threshold information stored in the database 30. The above threshold value may be set in advance by the information input system 1, the provider of the information input system 1, or the user U, for example. The above threshold value may be set, for example, based on past statistical values indicating changes in the distance between the drawing point P and the pen tip 204 of the same user U, or may be set arbitrarily by the user U. . A threshold value for a change in the distance between the drawing point P of the same user U and the pen tip 204 may be set in advance using artificial intelligence (AI) technology.

FIG. 14 is a diagram showing an example of an image in which the written information D4 is extracted. As shown in FIG. 14, when the user U performs an action of writing a line constituting a character and then writing the next line, the distance between the virtual surface S and the writing instrument 20 in the Sz axis direction increases. In FIG. 14, trajectories T1 and T2 indicate the trajectories of the pen tip 204 when it is assumed that the pen tip 204 is on the virtual surface S during writing. A trajectory T1 indicated by a solid line indicates a trajectory of the pen tip 204 that the user U intends to write on the virtual surface S. On the other hand, a trajectory T2 indicated by a broken line indicates a trajectory of the pen tip 204 in mid-air where the user U does not intend to write on the virtual surface S.

As shown in FIG. 14, compared to the distance between the drawing point P and the pen tip 204 when the user U performs an action intended to write on the virtual surface S (that is, an action corresponding to the trajectory T1), the user U The distance between the drawing point P and the pen tip 204 becomes larger when the user performs an action that is not intended to be written on the virtual surface S (that is, an action corresponding to the trajectory T2). Therefore, the writing information extraction unit 165 determines whether or not the change Δd in the distance between the drawing point P and the pen tip 204 is less than a preset threshold value, thereby determining the trajectory T1 from the drawing point trajectory information D31. The coordinate values of the indicated drawing point P are extracted as writing information D4.

Instead of extracting the writing information D4 based on the change Δd in the distance between the drawing point P and the pen tip 204, the writing information extraction unit 165 extracts the writing information D4 based on the change in the moving speed of the pen tip 204, for example. may be extracted. Alternatively, the handwriting information extraction unit 165 may extract the handwriting information D4 based on both the change Δd in the distance between the drawing point P and the pen tip 204 and the change in the moving speed of the pen tip 204.

The handwritten information extraction unit 165 provides the extracted handwritten information D4 to the output unit 19. The output unit 19 outputs the written information D4 to the display unit 13. The display unit 13 displays the handwritten information D4 in real time on a virtual surface S arranged in the virtual space. Thereby, the user U can see the written information D4 displayed on the virtual surface S. The output unit 19 may output the handwritten information D4 not only to the display unit 13 but also to another display device. For example, the output unit 19 may output the handwritten information D4 to a display screen of a personal computer, a cloud server, a smart device (for example, a smartphone or a tablet terminal), or the like.

Refer to FIG. 13 again. When the writing operation by the user U is completed, the appraisal unit 18 receives the trajectory information D3 recorded in the trajectory information acquisition unit 164, and evaluates the writing information D4. The appraisal unit 18 may determine that the writing action by the user U has ended when it receives a signal indicating that the user U has finished writing, or may determine that the writing action by the user U has ended when a predetermined period of time has elapsed since the writing action stopped. It may be determined that the writing operation has ended. The appraisal section 18 includes, for example, a non-written information extraction section 181 and a comparison section 182. The non-writing information extraction unit 181 extracts the trajectory of the pen tip 204 that the user U does not intend to write on the virtual surface S from the pen tip trajectory information D32 as non-writing information D5. The non-written information D5 can be extracted in the same manner as the written information D4.

That is, when the non-written information extraction unit 181 determines that the change Δd in the distance between the drawing point P and the pen tip 204 is not less than a preset threshold, the non-written information extraction unit 181 detects the trajectory of the pen tip 204 associated with the change in distance. is extracted as non-written information D5 that the user U does not intend to write on the virtual surface S. On the other hand, if the non-written information extraction unit 181 determines that the change Δd in the distance between the drawing point P and the pen tip 204 is not less than the threshold, the user U It is determined that the non-written information D5 is not intended to be written on the virtual surface S.

The comparison unit 182 receives the non-written information D5 from the non-written information extraction unit 181, and converts the information including the written information D4 and the non-written information D5 into current handwriting information D45 indicating the current handwriting of the user U (see FIG. 15 (described later)). a) Reference). The comparison unit 182 may extract the written information D4 from the trajectory information D3, or may receive the written information D4 from the written information extraction unit 165. Furthermore, the comparison unit 182 acquires past handwriting information D12 from the database 30. The past handwriting information D12 is information indicating the past handwriting of the user U, and corresponds to the current handwriting information D45. The comparison unit 182 compares the current handwriting information D45 and the past handwriting information D12. For example, the comparison unit 182 calculates the degree of matching between the current handwriting information D45 and the past handwriting information D12.

The above-mentioned "matching degree" may be an index indicating how similar the current handwriting information D45 and the past handwriting information D12 are. If each coordinate value indicated by the current handwriting information D45 is similar to each coordinate value indicated by the past handwriting information D12, the degree of coincidence between the current handwriting information D45 and the past handwriting information D12 will be high; otherwise, the corresponding The degree of agreement will be low. For example, the comparison unit 182 compares each coordinate value indicated by the current handwriting information D45 and each coordinate value indicated by the past handwriting information D12 in a time series, and determines the degree of deviation of the coordinate values for each time series as a degree of coincidence. It may be calculated. In this case, the comparison unit 182 may calculate a statistical value (for example, an average value or a median value) of the degree of deviation of coordinate values for each time series as the degree of coincidence.

The comparison unit 182 determines whether the calculated degree of matching is greater than or equal to a preset threshold. If the comparison unit 182 determines that the degree of matching is greater than or equal to the threshold, it determines that the current handwriting information D45 and the past handwriting information D12 are the same, and compares the user U who provided the current handwriting information D45 with the past handwriting information D12. It is determined that the user U who provided the information is the same person. On the other hand, if the comparison unit 182 determines that the degree of matching is not equal to or higher than the threshold, it determines that the current handwriting information D45 and the past handwriting information D12 are not the same, and the comparison unit 182 determines that the current handwriting information D45 and the past handwriting information D12 are It is determined that the user U who provided the is a different person.

The above "threshold" can be obtained from the threshold information D11 stored in the database 30. The above threshold value may be set in advance by the information input system 1, the provider of the information input system 1, or the user U, for example. The above threshold value may be set based on a past statistical value indicating the degree of matching between the current handwriting information D45 and the past handwriting information D12, or may be set arbitrarily by the user U, for example. A threshold value for the degree of matching between the current handwriting information D45 and the past handwriting information D12 may be set in advance using artificial intelligence (AI) technology.

FIG. 15(a) is a diagram showing an example of an image of the current handwriting information D45. FIG. 15(a) is a diagram showing an example of an image of past handwriting information D12. In the example shown in FIGS. 15A and 15B, the handwriting information D4 of the current handwriting information D45 is similar to the past handwriting information D14 corresponding to the handwriting information D4 of the past handwriting information D12. However, the non-written information D5 of the current handwriting information D45 is significantly different from the past non-written information D15 corresponding to the non-written information D5 of the past handwriting information D12. In this case, the comparing unit 182 compares the current handwriting information D45 including the past written information D14 and the non-written information D5 with the past handwriting information D12 including the written information D4 and the past non-written information D15.

As a result, the comparison unit 182 determines that the degree of coincidence between the current handwriting information D45 and the past handwriting information D12 is not equal to or greater than the threshold value, and determines that the current handwriting information D45 and the past handwriting information D12 are not the same. In this case, the comparison unit 182 determines that the user U who provided the current handwriting information D45 and the user U who provided the past handwriting information D12 are different people. In this way, the comparison unit 182 not only compares the written information D4 and the past written information D14, but also includes a comparison between the non-written information D5, which the user U does not intend to write, and the past non-written information D15. Then, the written information D4 is evaluated. Therefore, in this embodiment, as shown in FIGS. 15(a) and 15(b), even if the written information D4 and the past written information D14 are similar, the non-written information D5 and the past written information D14 are similar. Based on the difference from the handwritten information D15, it can be determined that the user U is not the original user (that is, the user U is an impersonator).

The comparison unit 182 may extract the past written information D14 and the past non-written information D15 from the past handwritten information D12, and compare the degree of matching between the written information D4 and the past written information D14 and the non-written information D5 and the past non-written information. The degree of matching with D15 may be calculated respectively. In this case, the comparison unit 182 may determine whether each degree of matching is greater than or equal to a threshold value. The comparison unit 182 determines whether or not the degree of matching between the current handwriting information D45 and the past handwriting information D12 is greater than or equal to a threshold value, thereby determining whether or not the user U who provided the handwriting information D4 is the user. . Then, the comparison unit 182 outputs to the output unit 19 an appraisal result D6 indicating whether or not the user U who provided the written information D4 is the user himself/herself. The output unit 19 may output the appraisal result D6 to the display unit 13, or may output the appraisal result D6 to a notification unit such as a speaker.

[Operation of information input system]
Next, the operation of the information input system 1 according to this embodiment will be explained. FIG. 16 is a flowchart illustrating an example of the processing contents of the information input method implemented in the information input system 1.

First, the overall process performed by the information input system 1 will be described with reference to FIG. 16. The processing by the information input system 1 includes an observation process (step S1) to acquire the observed value D10 from the motion sensor 208, an estimation process (step S2) to estimate the estimated value D20, and a recognition process (step S2) to recognize the trajectory of the writing instrument 20. step S3), and an output process (step S4) for outputting the written information D4. Steps S1 to S4 are, for example, repeatedly executed at predetermined intervals.

In step S1, the motion sensor 208 of the writing instrument 20 acquires observed values D10 of acceleration and angular velocity in accordance with the operation of the writing instrument 20. Next, the acquisition unit 12 of the terminal 10 acquires the observed value D10 from the motion sensor 208. In step S2, the estimation unit 15 estimates the estimated value D20 of the position and orientation of the writing instrument 20 based on the observed value D10. In step S3, the recognition unit 16 sets a drawing point P on the virtual surface S based on the estimated value D20. Next, the recognition unit 16 acquires writing information D4 for the virtual surface S based on the trajectory indicated by the drawing point P. In step S4, the output unit 19 outputs the written information D4 to the display unit 13. By repeating the above steps S1 to S4, the user U can visually check the handwritten information D4 written with the writing instrument 20 via the display unit 13 in real time.

FIG. 17 is a flowchart illustrating an example of the estimation process performed by the estimation unit 15. First, the model acquisition unit 155 acquires a writing instrument model D13 that is a model of the writing instrument 20 from the database 30 (step S21). Next, the initialization processing unit 156 initializes the state (eg, position and orientation) of the writing instrument model D13 (step S22). On the other hand, when the acquisition unit 12 acquires the observed value D10, the disturbance calculation unit 157 calculates the disturbance to be applied to the writing instrument model D13 (step S23). The disturbance may be a force and torque applied to the writing instrument 20 by a user during operation of the writing instrument 20. Next, the analysis execution unit 158 sets the disturbance condition D22 as an analysis condition for executing a simulation of the operation of the writing instrument 20 (step S24).

Next, the analysis execution unit 158 executes a simulation of the operation of the writing instrument 20 using the writing instrument model D13 (step S25). The analysis execution unit 158 causes the writing instrument model D13 to operate in the simulation space by applying a disturbance condition D22 to the writing instrument model D13. As a result, the position and orientation of the writing instrument model D13 in the simulation space change. Next, the estimated value deriving unit 153 derives the estimated value D20 of the current position and orientation of the writing instrument 20 from the simulation execution result (step S26). For example, the estimated value deriving unit 153 derives the estimated value D20 by associating the position and orientation of the writing instrument model D13 in the simulation space with the position and orientation of the writing instrument 20 in the virtual space, respectively. Through the above steps S21 to S26, the estimated value D20 is estimated based on the observed value D10.

FIG. 18 is a flowchart illustrating an example of feedback processing performed by the feedback unit 154. When the estimation unit 15 estimates the estimated value D20, the calculation unit 143 calculates the amount of change in the estimated value D20 from the reference value using the predetermined position and orientation of the writing instrument 20 with respect to the virtual plane S as the reference value (step S31). . The amount of change in the estimated value D20 from the reference value can be expressed, for example, by taking the inner product of the unit direction vector V1 toward which the pen tip 204 of the writing instrument 20A is directed and the unit direction vector V2 toward which the pen tip 204 of the writing instrument 20A is directed. can. By taking the inner product of the unit directional vector V1 and the unit directional vector V2, the angle θ formed by the unit directional vector V1 and the unit directional vector V2 can be calculated. Therefore, the calculation unit 143 calculates the angle θ formed by the unit direction vector V1 and the unit direction vector V2 as an index indicating the amount of change in the estimated value D20 from the reference value.

Next, the determination unit 144 determines whether the angle θ (that is, the amount of change in the estimated value D20) calculated by the calculation unit 143 is within the second tolerance range (step S32). If the determining unit 144 determines that the angle θ is not within the second tolerance range (step S32: No), the adjusting unit 142 performs calibration to return the estimated value D20 to the reference value (step S33). For example, the adjustment unit 142 calculates the disturbance necessary to move the unit direction vector V2 of the writing instrument 20 indicated by the estimated value D20 to the unit direction vector V1 of the writing instrument 20A indicated by the reference value. The adjustment unit 142 changes the position and orientation of the writing instrument model D13 in the simulation space by applying the calculated disturbance to the writing instrument model D13. Then, the adjustment unit 142 reflects the position and orientation of the writing instrument model D13 after execution of the simulation in the current position and orientation of the writing instrument 20 in the virtual space, thereby obtaining an estimated value of the current position and orientation of the writing instrument 20. Return D20 to standard value.

On the other hand, when determining that the angle θ is within the second tolerance range, the determination unit 144 determines whether the angle θ is within the first tolerance range (step S34). When determining that the angle θ is not within the first tolerance range (step S34: No), the determining unit 144 adjusts the amount of change in the estimated value D20 with respect to the reference value. For example, the adjustment unit 142 makes the disturbance function A given to the writing instrument model D13 during simulation execution smaller than the disturbance function A calculated by the disturbance calculation unit 157. Thereby, the adjustment unit 142 makes the disturbance applied to the writing instrument model D13 smaller than the disturbance applied to the actual writing instrument 20.

As a result, the writing instrument model D13 in the simulation space becomes smaller than the actual operation of the writing instrument 20. In other words, the movement of the estimated value D20 indicated by the simulation execution result becomes smaller. When the determination unit 144 determines that the angle θ is within the first tolerance range (step S34: Yes), the determination unit 144 returns to step S31 again and sets the angle θ obtained based on the next estimated value D20 in step S31. ~Repeat step S35.

FIG. 19 is a flowchart illustrating an example of the drift correction process performed by the feedback unit 154. When the estimation unit 15 estimates the estimated value D20, the determination unit 144 determines whether the drift correction condition is satisfied (step S41). The drift correction condition is such that when the center of the distribution of the current estimated value D20 within a certain period of time exceeds a threshold value, the center of the distribution of the estimated value D20 within a certain period of time is set as the reference value when drift errors are not accumulated. It is filled. When the determination unit 144 determines that the drift correction condition is satisfied (step S41: Yes), the adjustment unit 142 performs calibration to return the estimated value D20 to the reference value (step S42). On the other hand, if the determining unit 144 determines that the drift correction condition is not satisfied (step S41: No), the determining unit 144 returns to step S41 again and performs steps S41 and S42 for the next estimated value D20. repeat.

FIG. 20 is a flowchart showing an example of the recognition process performed by the recognition unit 16. When the estimation unit 15 estimates the estimated value D20, the virtual line setting unit 161 sets a straight line passing through the position of the pen tip 204 and along the axial direction L of the writing instrument 20 as the virtual line VL (step S51). The virtual line VL extends linearly from the pen tip 204 in the direction toward which the pen tip 204 faces, and intersects the virtual surface S. Next, the drawing point setting unit 162 calculates the intersection between the virtual line VL and the virtual surface S (step S52). Next, the drawing point setting unit 162 sets the intersection of the virtual line VL and the virtual surface S as a drawing point P. The position of the drawing point P is set, for example, as a coordinate value in the screen coordinate system β.

Next, the trajectory information acquisition unit 164 acquires trajectory information D3 indicating the trajectory of the writing instrument 20 (step S54). The trajectory information D3 includes, for example, drawing point trajectory information D31 in which the coordinate values of the drawing point P are recorded in chronological order, pen tip trajectory information D32 in which the coordinate values of the pen tip 204 are recorded in chronological order, and the coordinates of the drawing point P. It includes distance information D33 indicating the distance between the value and the coordinate value of the pen tip 204. Next, the writing information extraction unit 165 extracts writing information D4 indicating the trajectory of the drawing point P that the user U intends to write on the virtual surface S from the drawing point trajectory information D31 while referring to the distance information D33. (Step S55). The handwriting information extraction unit 165 determines, for example, whether a change in the distance between the drawing point P and the pen tip 204 indicated by the distance information D33 is less than a preset threshold. Then, when the writing information extraction unit 165 determines that the change in the distance between the drawing point P and the pen tip 204 is less than the threshold value, the writing information extraction unit 165 extracts the trajectory of the drawing point P associated with the change in distance as writing information D4. do.

FIG. 21 is a flowchart illustrating an example of the adjustment process performed by the adjustment unit 17. When the drawing point setting unit 162 sets the drawing point P, the adjustment unit 17 determines whether the drawing point P is located within the second region R2 of the virtual surface S (step S61). When the adjustment unit 17 determines that the drawing point P is not located within the second region R2 (step S61: No), the adjustment unit 17 executes a calibration process to return the position of the virtual surface S with respect to the writing instrument 20 to the reference position ( Step S62). On the other hand, when the adjustment unit 17 determines that the drawing point P is within the second region R2 (step S61: Yes), it determines whether the drawing point P is within the first region R1 (step S63). . If the determination unit 144 determines that the drawing point P is not within the first region R1 (step S63: No), the determination unit 144 uses the predetermined position and orientation of the writing instrument 20 with respect to the virtual surface S as a reference value, and calculates the estimated value D20 with respect to the reference value. The amount of change is adjusted (step S64). When the adjustment unit 17 determines that the drawing point P is within the first region R1 (step S63: Yes), the adjustment unit 17 returns to step S61 again and adjusts the drawing point P obtained based on the next estimated value D20. Steps S61 to S64 are repeated.

FIG. 22 is a flowchart illustrating an example of the appraisal process performed by the appraisal unit 18. First, the appraisal unit 18 determines whether or not the writing operation by the user U has been completed (step S71). When determining that the writing action by the user U has not been completed (step S71: No), the appraisal unit 18 repeats step S71 until it is determined that the writing action by the user U has been completed. On the other hand, when determining that the writing action by the user U has ended (step S71: Yes), the appraisal unit 18 extracts the non-writing information D5 from the trajectory information D3. For example, based on the change Δd in the distance between the drawing point P and the pen tip 204, the non-writing information extraction unit 181 determines, as the non-writing information D5, the trajectory of the pen tip 204 that the user U does not intend to write on the virtual surface S. Extract (step S72).

Next, the comparison unit 182 acquires information including the written information D4 and the non-written information D5 as current handwriting information D45 indicating the current handwriting of the user U, and the degree of matching between the current handwriting information D45 and the past handwriting information D12. is calculated (step S73). Next, the comparison unit 182 evaluates the handwriting information D4 based on the degree of matching between the current handwriting information D45 and the past handwriting information D12 (step S74). For example, the comparison unit 182 determines whether the degree of matching is greater than or equal to a preset threshold. If the comparison unit 182 determines that the degree of matching is greater than or equal to the threshold, it determines that the user U who provided the current handwriting information D45 and the user U who provided the past handwriting information D12 are the same person. That is, the comparison unit 182 determines that the user U who provided the written information D4 is the user himself/herself. On the other hand, if the comparison unit 182 determines that the degree of matching is not equal to or greater than the threshold, it determines that the user U who provided the current handwriting information D45 and the user U who provided the past handwriting information D12 are different people. In other words, the comparison unit 182 determines that the user U who provided the written information D4 is an impersonator.

[Effect]
The effects of the information input system 1, the information input method, and the information input program according to the present embodiment described above will be explained. In this embodiment, an observed value D10 regarding the operation of the writing instrument 20 is acquired from the motion sensor 208, and a disturbance to the writing instrument model D13 is calculated using the observed value D10 and mass information. Then, by applying a disturbance to the writing instrument model D13, a simulation of the operation of the writing instrument 20 is executed using the writing instrument model D13. The state of the writing instrument model D13 indicated by the simulation execution result is associated with the current state of the writing instrument 20. As a result, an estimated value D20 of the state of the writing instrument 20 is derived. By using the simulation using the writing instrument model D13 in this manner, information obtained at the position of the motion sensor 208 can be converted into information at an arbitrary position of the writing instrument 20 and processed. Therefore, regardless of the position of the motion sensor 208 with respect to the writing instrument 20, the estimated value D20 can be derived with high accuracy based on the observed value D10 from the motion sensor 208. Furthermore, by applying a disturbance to the writing instrument model D13 having mass information indicating the mass of the writing instrument 20, it is possible to give the writing instrument model D13 a natural movement that takes the mass of the writing instrument 20 into consideration. In other words, it is possible to bring the change in the state of the writing instrument 20 indicated by the estimated value D20 closer to the actual change in the state of the writing instrument 20 accompanying the writing operation. As a result, it is possible to suppress the discomfort that the user U feels during the writing operation.

As in this embodiment, the writing instrument 20 having the pen tip 204 at one end in the axial direction L may be used as an input device. In this case, the information indicated by the trajectory of the pen tip 204 can be accurately acquired as handwriting information, and the discomfort felt by the user U during the writing operation can be suppressed.

As in the present embodiment, the information input system 1 includes a setting unit 14 that sets a virtual plane S in the space where the writing instrument 20 is placed, and a recognition unit 16 that recognizes the trajectory of the estimated value D20 with respect to the virtual plane S. You may prepare. In this case, the writing motion of the user U on the virtual surface S can be recognized.

As in this embodiment, the estimating unit 15 may include a monitoring unit 141 that monitors changes in the estimated value D20, and an adjusting unit 142 that adjusts the estimated value D20. In this case, with this configuration, when the change in the estimated value D20 is large, the estimated value D20 can be adjusted so that the change in the estimated value D20 becomes small. Thereby, a situation in which the estimated value D20 deviates significantly from the virtual plane S can be suppressed. As a result, it becomes possible to recognize the locus of the estimated value D20 with respect to the virtual surface S more reliably.

As in the present embodiment, the monitoring unit 141 includes a calculation unit 143 that calculates the angle θ as the amount of change in the estimated value D20 from the reference value, using a predetermined state of the writing instrument 20 with respect to the virtual plane S as a reference value; may also include a determination unit 144 that determines whether or not is within a first tolerance range. The adjustment unit 142 improves the simulation by making the disturbance applied to the writing instrument model D13 smaller than the disturbance calculated by the disturbance calculation unit 157 when the determination unit 144 determines that the angle θ is not within the first tolerance range. The angle θ of the estimated value D20 indicated by the execution result may be made smaller. With this configuration, when the estimated value D20 changes significantly from the reference value, by reducing the disturbance applied to the writing instrument model D13, it is possible to reduce the change in the state of the writing instrument model D13 caused by execution of the simulation. Thereby, the movement of the estimated value D20 indicated by the simulation execution result can be reduced. As a result, it is possible to suppress a situation in which the estimated value D20 greatly deviates from the virtual surface S, and it becomes possible to recognize the locus of the estimated value D20 with respect to the virtual surface S more reliably.

As in the present embodiment, the determination unit 144 may determine whether the angle θ is within a second tolerance range that is larger than the first tolerance range. The adjustment unit 142 may return the estimated value D20 to the reference value when the determination unit 144 determines that the angle θ is not within the second tolerance range. According to this configuration, when the estimated value D20 changes further from the reference value, by returning the estimated value D20 to the reference value, the trajectory of the estimated value D20 with respect to the virtual surface S can be more reliably recognized. It becomes possible.

As in the present embodiment, when the determination unit 144 determines that the angle θ is within the second tolerance range and not within the first tolerance range, the adjustment unit 142 adjusts the angle θ as it deviates from the first tolerance range. The reduction rate of the disturbance applied to the writing instrument model D13 relative to the disturbance calculated by the disturbance calculation unit 157 may be increased. In this configuration, the larger the angle θ, the smaller the movement of the estimated value D20 indicated by the simulation execution result. This makes it possible for the user U to recognize that the estimated value D20 deviates from the reference value.

[Modified example]
The detailed description has been made above based on the embodiments of the present disclosure. However, the present disclosure is not limited to the embodiments described above. The present disclosure can be modified in various ways without departing from the gist thereof.

In the embodiment described above, the terminal 10 includes the communication unit 11, the acquisition unit 12, the display unit 13, the setting unit 14, the estimation unit 15, the adjustment unit 17, the recognition unit 16, and the appraisal unit as functional elements. A case is illustrated in which the output unit 18 and the output unit 19 are included. However, at least one of these functional elements may be implemented in the writing instrument 20. When all of these functional elements are implemented in the writing instrument 20, the information input system 1 can acquire the writing information D4 without using the terminal 10. The information input system 1 may include a server separate from the terminal 10. In this case, at least one of the above functional elements included in the terminal 10 may be implemented in the server. In the embodiment described above, the terminal 10 or the database 30 stores various information such as the threshold information D11, the past handwriting information D12, and the writing instrument model D13. However, when the writing instrument 20 includes a storage device, the storage device may store various information.

In the embodiment described above, the case where the motion sensor 208 includes an acceleration sensor and a gyro sensor is exemplified. However, the motion sensor 208 may include a magnetic field sensor that detects magnetic fields in three axes orthogonal to each other instead of the acceleration sensor or the gyro sensor. Alternatively, motion sensor 208 may include a magnetic field sensor in addition to an acceleration sensor and a gyro sensor. In the embodiment described above, the writing instrument 20 is used as an example of the input device. However, the input device may be any device other than the writing instrument 20 as long as it is capable of detecting an input operation by the user U.

The processing procedure executed in the information input system 1 is not limited to the example shown in the embodiment described above. For example, some of the steps (processes) described above may be omitted, or each step may be executed in a different order. Any two or more of the steps described above may be combined, or some of the steps may be modified or deleted. Alternatively, other steps may be performed in addition to each of the above steps.

The purpose and scene of use of the writing instrument 20 as an example of an input device are not limited. The writing implement 20 may be used in a learning scene that involves writing, or may be used in a scene other than a learning scene. Learning that involves writing may be learning that takes place at school, home, or other places. The writing instrument 20 may be used in the creation of illustrations or charts. The input information system is not limited to the above-mentioned situations, but can be applied to various situations in which an input device such as the writing instrument 20 is used.

The gist of the present disclosure is shown below.
[1] An acquisition unit that acquires observed values regarding the operation of the input device from the sensor;
an estimation unit that estimates an estimated value of the state of the input device based on the observed value;
an output unit that outputs information indicated by the trajectory of the estimated value to a display device as input information from the input device,
The estimation unit is
a model acquisition unit that acquires an input device model that models the input device;
a disturbance calculation unit that calculates a disturbance to the input device model using the observed value and mass information indicating the mass of the input device;
an analysis execution unit that performs a simulation of the operation of the input device using the input device model by applying the disturbance to the input device model including the mass information;
An information input system comprising: an estimated value deriving unit that derives the estimated value by associating a state of the input device model indicated by the execution result of the simulation with a current state of the input device.
[2] The input device is a writing instrument including a pen tip at one end in the axial direction,
The information input system according to [1], wherein the estimator estimates, as the estimated values, an estimated value of the position of the pen tip and an estimated value of the angle in the axial direction.
[3] a setting unit that sets a virtual plane in a space in which the input device is arranged;
The information input system according to [1] or [2], further comprising a recognition unit that recognizes a trajectory of the estimated value with respect to the virtual surface.
[4] The estimation unit includes:
a monitoring unit that monitors changes in the estimated value with respect to the virtual surface;
The information input system according to [3], further comprising: an adjustment unit that adjusts the estimated value for the virtual plane.
[5] The monitoring unit:
a calculation unit that calculates an amount of change in the estimated value from the reference value, using a predetermined state of the input device with respect to the virtual plane as a reference value;
a determination unit that determines whether the amount of change is within a first tolerance range,
The adjustment unit makes the disturbance applied to the input device model smaller than the disturbance calculated by the disturbance calculation unit when the determination unit determines that the amount of change is not within the first tolerance range. The information input system according to [4], wherein the amount of change in the estimated value indicated by the execution result of the simulation is reduced.
[6] The determination unit determines whether the amount of change is within a second tolerance range that is larger than the first tolerance range,
The information input system according to [5], wherein the adjustment unit returns the estimated value to the reference value when the determination unit determines that the amount of change is not within the second tolerance range.
[7] The adjustment unit adjusts the amount of change so that the amount of change deviates from the first tolerance range when the determination unit determines that the amount of change is within the second tolerance range and not within the first tolerance range. , the information input system according to [6], wherein the reduction rate of the disturbance given to the input device model with respect to the disturbance calculated by the disturbance calculation unit is increased.
[8] An information input method executed by an information input system including a processor,
Obtaining observed values regarding the operation of the input device from the sensor;
estimating an estimate of the state of the input device based on the observed value;
outputting information indicated by the trajectory of the estimated value to a display device as input information by the input device,
The step of estimating the estimated value includes:
obtaining an input device model that models the input device;
calculating a disturbance to the input device model using the observed value and mass information indicating the mass of the input device;
performing a simulation of the operation of the input device using the input device model by applying the disturbance to the input device model including the mass information;
An information input method comprising: deriving the estimated value by associating the state of the input device model indicated by the execution result of the simulation with the current state of the input device.
[9] Obtaining observed values regarding the operation of the input device from the sensor;
estimating an estimate of the state of the input device based on the observed value;
causing a computer to perform a step of outputting information indicated by the trajectory of the estimated value to a display device as input information from the input device;
The step of estimating the estimated value includes:
obtaining an input device model that models the input device;
calculating a disturbance to the input device model using the observed value and mass information indicating the mass of the input device;
performing a simulation of the operation of the input device using the input device model by applying the disturbance to the input device model including the mass information;
An information input program comprising: deriving the estimated value by associating the state of the input device model indicated by the execution result of the simulation with the current state of the input device.

DESCRIPTION OF SYMBOLS 1... Information input system, 12... Acquisition part, 13... Display part (an example of a display device), 14... Setting part, 15... Estimation part, 16... Recognition part, 19... Output part, 20... Writing implement (an example of an input device) ), 101... Processor, 141... Monitoring section, 143... Calculating section, 142... Adjusting section, 144... Judging section, 153... Estimated value deriving section, 155... Model acquiring section, 157... Disturbance calculating section, 158... Analysis execution section , 204... Pen tip, 208... Movement sensor (an example of a sensor), D4... Writing information (an example of input information), D10... Observed value, D13... Writing instrument model (an example of an input device model), D20... Estimated value, L ...Axis direction, S...Virtual surface, T1...Trajectory, θ...Angle (an example of the amount of change).

Claims

an acquisition unit that acquires observed values regarding the operation of the input device from the sensor;
an estimation unit that estimates an estimated value of the state of the input device based on the observed value;
an output unit that outputs information indicated by the trajectory of the estimated value to a display device as input information from the input device,
The estimation unit is
a model acquisition unit that acquires an input device model that models the input device;
a disturbance calculation unit that calculates a disturbance to the input device model using the observed value and mass information indicating the mass of the input device;
an analysis execution unit that performs a simulation of the operation of the input device using the input device model by applying the disturbance to the input device model including the mass information;
An information input system comprising: an estimated value deriving unit that derives the estimated value by associating a state of the input device model indicated by the execution result of the simulation with a current state of the input device.
The input device is a writing instrument including a pen tip at one end in the axial direction,
The information input system according to claim 1, wherein the estimator estimates, as the estimated values, an estimated value of the position of the pen tip and an estimated value of the angle in the axial direction.
a setting unit that sets a virtual plane in a space where the input device is placed;
The information input system according to claim 2, further comprising a recognition unit that recognizes a trajectory of the estimated value with respect to the virtual surface.
The estimation unit is
a monitoring unit that monitors changes in the estimated value with respect to the virtual surface;
The information input system according to claim 3, further comprising: an adjustment unit that adjusts the estimated value for the virtual plane.
The monitoring unit is
a calculation unit that calculates an amount of change in the estimated value from the reference value, using a predetermined state of the input device with respect to the virtual plane as a reference value;
a determination unit that determines whether the amount of change is within a first tolerance range,
The adjustment unit makes the disturbance applied to the input device model smaller than the disturbance calculated by the disturbance calculation unit when the determination unit determines that the amount of change is not within the first tolerance range. The information input system according to claim 4, wherein the amount of change in the estimated value indicated by the execution result of the simulation is reduced.
The determination unit determines whether the amount of change is within a second tolerance range that is larger than the first tolerance range,
The information input system according to claim 5, wherein the adjustment unit returns the estimated value to the reference value when the determination unit determines that the amount of change is not within the second tolerance range.
When the determination unit determines that the amount of change is within the second tolerance range and not within the first tolerance range, the adjustment unit is configured to adjust the disturbance so that the amount of change deviates from the first tolerance range. 7. The information input system according to claim 6, wherein the reduction rate of the disturbance applied to the input device model with respect to the disturbance calculated by the calculation unit is increased.
An information input method performed by an information input system comprising a processor, the method comprising:
Obtaining observed values regarding the operation of the input device from the sensor;
estimating an estimate of the state of the input device based on the observed value;
outputting information indicated by the trajectory of the estimated value to a display device as input information by the input device,
The step of estimating the estimated value includes:
obtaining an input device model that models the input device;
calculating a disturbance to the input device model using the observed value and mass information indicating the mass of the input device;
performing a simulation of the operation of the input device using the input device model by applying the disturbance to the input device model including the mass information;
An information input method comprising: deriving the estimated value by associating the state of the input device model indicated by the execution result of the simulation with the current state of the input device.
Obtaining observed values regarding the operation of the input device from the sensor;
estimating an estimate of the state of the input device based on the observed value;
causing a computer to perform a step of outputting information indicated by the trajectory of the estimated value to a display device as input information from the input device;
The step of estimating the estimated value includes:
obtaining an input device model that models the input device;
calculating a disturbance to the input device model using the observed value and mass information indicating the mass of the input device;
performing a simulation of the operation of the input device using the input device model by applying the disturbance to the input device model including the mass information;
An information input program comprising: deriving the estimated value by associating the state of the input device model indicated by the execution result of the simulation with the current state of the input device.