WO2015049934A1

WO2015049934A1 - Information processing device, information processing method, and program

Info

Publication number: WO2015049934A1
Application number: PCT/JP2014/072227
Authority: WO
Inventors: 健太郎井田; 由幸小林; 宏之水沼; 山野　郁男
Original assignee: ソニー株式会社
Priority date: 2013-10-02
Filing date: 2014-08-26
Publication date: 2015-04-09

Abstract

The present invention controls the display of a prediction locus predicted in accordance with legacy locus information. The present invention has a data processing unit for processing the control of locus display corresponding to input position information, the data processing unit displaying, as an actual locus, a locus for which a locus calculation process corresponding to the input position information has been completed, estimating, as a prediction locus, a locus in a region for which the locus calculation process corresponding to the input position information has not been completed, and displaying the estimated prediction locus in a manner different from the already displayed actual locus. For example, at least one of the color, permeability, and thickness of the prediction locus is displayed in a manner different from that of the actual locus so that the prediction locus is displayed in a manner less conspicuous than the actual locus. Also, the prediction locus is processed so as to be hidden in accordance with reliability.

Description

Information processing apparatus, information processing method, and program

The present disclosure relates to an information processing apparatus, an information processing method, and a program. Specifically, for example, the present invention relates to an information processing apparatus, an information processing method, and a program that perform display control for displaying a real trajectory and a predicted trajectory in an easily distinguishable manner in displaying a drawing trajectory on a touch panel display.

In recent years, a touch panel type input display device is widely used. A touch panel type input display device is a device that enables information input by touching a display surface with a finger or a pen as well as information display processing using, for example, a liquid crystal display.
A contact position with a finger or a pen on the display surface is detected, and processing according to the detected position, for example, drawing processing is enabled. There are various methods such as an electromagnetic induction method and a capacitance method for detecting the position of the pen or finger.

However, in such an apparatus, for example, when a line is drawn according to the pen trajectory using a pen as an input device, a “deviation” occurs between the position of the pen tip and the tip position of the display line on the display. There is.
This is caused by a delay in processing time from the pen position detection process to the line drawing process, and becomes more prominent when the pen is moved at high speed.

For example, Patent Document 1 (Japanese Patent Laid-Open No. 09-190275) is known as a prior art that discloses a configuration for eliminating this deviation. Patent Document 1 discloses a configuration in which a static trajectory (function) is used to predict and draw a locus that has not been processed.

In this disclosed method, a linear approximation line is generated using the latest locus region in a locus for which drawing has been completed, and the generated linear approximation line is extended and displayed as an unprocessed (undrawn) portion locus. It is. In this method, it is also possible to generate an approximate curve by changing the approximate order setting or using a trigonometric function.

However, this disclosed technique only performs a process of predicting the previous trajectory by an approximation process using only the drawing trajectory immediately before the unprocessed area. For example, a sudden speed change or a direction change is performed. In some cases, an overshoot of the predicted point or a large discrepancy between the actual pen tip position and the predicted point may occur.

Also, if the actual trajectory and the predicted trajectory are displayed in the same manner, the user (drawer) may be confused if an incorrect predicted trajectory is displayed.

JP 09-190275 A

The present disclosure has been made in view of the above-described problems, for example, by clearly distinguishing and displaying the actual trajectory and the predicted trajectory so that the user can perform smooth drawing processing without causing a war. It is an object to provide an information processing apparatus, an information processing method, and a program that are made possible.

The first aspect of the present disclosure is:
A data processing unit that performs display control processing of a locus according to input position information generated by user operation input;
The data processing unit
An actual trajectory identified based on the input position information;
A trajectory of an area where the identification of the real trajectory is not completed, and a predicted trajectory identified by a predetermined prediction process;
Is in an information processing apparatus that performs display control for displaying the image in a different manner.

Furthermore, in an embodiment of the information processing apparatus according to the present disclosure, the predicted trajectory is a trajectory predicted based on an actual trajectory specified in the past.

Furthermore, in an embodiment of the information processing apparatus according to the present disclosure, the data processing unit displays at least one of a color, a transparency, and a thickness of the predicted locus in a manner different from the actual locus.

Furthermore, in an embodiment of the information processing apparatus according to the present disclosure, the data processing unit sets a color, transparency, or thickness of the predicted trajectory to be different from the actual trajectory, It is displayed in a mode that is less conspicuous than the actual trajectory.

Furthermore, in an embodiment of the information processing apparatus according to the present disclosure, the data processing unit determines a display mode of the predicted trajectory according to the reliability of the predicted trajectory.

Furthermore, in an embodiment of the information processing apparatus according to the present disclosure, the data processing unit determines at least one of a color, a transparency, or a thickness of the predicted trajectory according to the reliability of the predicted trajectory. To display.

Furthermore, in an embodiment of the information processing apparatus according to the present disclosure, the data processing unit changes at least one of a color, transparency, or thickness of the predicted trajectory according to the reliability of the predicted trajectory. As the reliability decreases, the predicted trajectory is displayed in a less conspicuous manner.

Furthermore, in an embodiment of the information processing apparatus according to the present disclosure, the data processing unit controls the display unit so that the predicted trajectory is not displayed when the reliability of the predicted trajectory is determined to be lower than a predetermined threshold. .

Furthermore, in an embodiment of the information processing apparatus according to the present disclosure, when the data processing unit determines that the reliability of the predicted trajectory is lower than a predetermined threshold value, the display of the predicted trajectory is changed to an inconspicuous display mode. To do.

Furthermore, in one embodiment of the information processing apparatus according to the present disclosure, the input position information is input position information obtained by detecting a contact position of the input object with respect to the touch panel, and the data processing unit is configured so that the input object touches the touch panel. A pressure value representing a pressure is acquired, and a display mode of the predicted locus is determined according to the pressure value.

Furthermore, in an embodiment of the information processing apparatus according to the present disclosure, the data processing unit may not display the predicted trajectory when the data processing unit detects that the pressure value decrease is greater than a predetermined threshold value. To control the display.

Furthermore, in an embodiment of the information processing apparatus according to the present disclosure, the data processing unit acquires a movement amount per unit time of the input position information, and determines a display mode of the predicted locus according to the movement amount. .

Furthermore, in one embodiment of the information processing apparatus of the present disclosure, the data processing unit stops displaying the predicted trajectory when detecting that the movement amount is less than a predetermined threshold value.

Furthermore, in an embodiment of the information processing apparatus according to the present disclosure, the input object is a stylus.

Furthermore, in an embodiment of the information processing apparatus according to the present disclosure, the data processing unit detects a plurality of similar trajectories similar to the immediately preceding trajectory of the predicted trajectory calculation area as the predicted trajectory calculation process from past trajectories, A subsequent trajectory of each similar trajectory is estimated based on the detected plurality of similar trajectories, and a predicted trajectory is calculated by averaging or weighting addition processing of the estimated plurality of subsequent trajectories.

Furthermore, the second aspect of the present disclosure is:
An information processing method executed in an information processing apparatus,
The data processing unit performs a trajectory display control process according to the input position information generated by the user's operation input,
The data processing unit, in the display control process,
An actual trajectory identified based on the input position information;
A trajectory of an area where the identification of the real trajectory is not completed, and a predicted trajectory identified by a predetermined prediction process;
There is an information processing method for executing display control for displaying the image in a different manner.

Furthermore, the third aspect of the present disclosure is:
A program for executing information processing in an information processing apparatus;
Causing the data processing unit to perform display control processing of the locus according to the input position information generated by the user's operation input;
In the display control process,
An actual trajectory identified based on the input position information;
A trajectory of an area where the identification of the real trajectory is not completed, and a predicted trajectory identified by a predetermined prediction process;
Is in a program for executing display control for displaying the image in a different manner.

Note that the program of the present disclosure is a program that can be provided by, for example, a storage medium or a communication medium provided in a computer-readable format to an image processing apparatus or a computer system that can execute various program codes. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the information processing apparatus or the computer system.

Further objects, features, and advantages of the present disclosure will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

According to the configuration of an embodiment of the present disclosure, display control of a predicted trajectory predicted according to past trajectory information is realized.
Specifically, it has a data processing unit that performs a display control process of the trajectory according to the input position information, and the data processing unit displays the trajectory after the trajectory calculation process according to the input position information is completed as an actual trajectory, A trajectory in an area where the trajectory calculation process according to the input position information is not completed is estimated as a predicted trajectory, and the estimated predicted trajectory is displayed in a manner different from the displayed actual trajectory. For example, at least one of the color, transparency, and thickness of the predicted trajectory is displayed in a manner different from the actual trajectory, and the predicted trajectory is displayed in a manner that is less conspicuous than the actual trajectory. Further, a process for hiding the predicted trajectory is performed according to the reliability.
With this configuration, display control of a predicted trajectory predicted according to past trajectory information is realized.
Note that the effects described in the present specification are merely examples and are not limited, and may have additional effects.

It is a figure explaining the problem in case a line segment (line) is drawn on an input display apparatus using a pen-type input device. It is a figure explaining the example of generation | occurrence | production of the prediction error in the drawing process of a prediction locus. It is a figure explaining the prediction process of the locus | trajectory which the information processing apparatus of this indication performs. It is a figure explaining the outline | summary of the prediction process of the locus | trajectory which the information processing apparatus of this indication performs. It is a figure which shows the flowchart explaining the estimation of the prediction locus | trajectory which applied the kNN method, and the drawing process sequence. It is a figure explaining the example of the estimation of the prediction locus | trajectory which applied kNN method, and a drawing process. It is a figure explaining the feature-value applied to estimation of a prediction locus. It is a figure explaining the extraction process example of a similar locus | trajectory. It is a figure explaining the reliability of a prediction locus. It is a figure which shows the flowchart explaining the sequence of the calculation drawing process of the prediction locus | trajectory accompanied by the process change according to a standard deviation calculation process and a process result. It is a figure which shows an example of the standard deviation of each coordinate position of multiple (k pieces) auxiliary prediction locus | trajectory applied to the determination process of a prediction locus | trajectory, and an auxiliary prediction locus | trajectory. It is a figure which shows the example of calculation of the reliability of each future frame U = t + 1, t + 2, t + 3. It is a figure explaining the example of concrete display control of a prediction locus. It is a figure explaining the example of concrete display control of a prediction locus. It is a figure explaining the specific example of display control according to the reliability of a prediction locus. It is a figure explaining the specific example of display control according to the reliability of a prediction locus. It is a figure explaining the example of a process which erases the displayed prediction locus in the next drawing frame, and draws a new prediction locus. It is a figure explaining the process which does not draw the prediction locus | trajectory just before reaching the prediction point, when the reliability value index value is not less than the predetermined threshold value. It is a figure explaining the example of a process which performs drawing of a prediction locus | trajectory according to a condition, ie, switching display or non-display. It is a figure explaining the detection process that the fall amount of the input device pressure value per unit time is more than regulation threshold value [Thp]. It is a figure explaining the example of a process which detects that the input device movement amount per unit time is less than regulation threshold value [Thd]. It is a figure explaining the example of a process which performs control which changes the display mode of a prediction locus, and makes a prediction locus inconspicuous when a prediction locus deviates from an actual locus. It is a figure which shows the flowchart explaining the process sequence of display control of a prediction locus | trajectory. It is a figure which shows the flowchart explaining the process sequence of display control of a prediction locus | trajectory. It is a figure explaining an example of the effect by the display of a prediction locus. It is a figure explaining the hardware structural example of the information processing apparatus of this indication.

Hereinafter, the details of the information processing apparatus, the information processing method, and the program of the present disclosure will be described with reference to the drawings. The description will be made according to the following items.
1. 1. Problems in the drawing process 2. Trajectory prediction processing executed by the information processing apparatus of the present disclosure 3. Example of prediction process using coordinate information of drawing trajectory 3-1. Similarity determination processing 3-2. 3. Prediction locus determination process 4. Example of processing according to the reliability of the predicted trajectory 5. Process applying pressure detection information of input device 6. Modification of predicted trajectory calculation and drawing processing 7. Example of controlling the drawing mode of the predicted trajectory 8. Processing sequence of display control of predicted trajectory An example of the effect of displaying a highly accurate predicted trajectory 10. 10. Hardware configuration example of information processing apparatus of present disclosure Summary of composition of this disclosure

[1. About problems in drawing processing]
First, a description will be given of problems in the case where drawing is performed on an input display device using an input device (input object) such as a touch pen.
FIG. 1 is a diagram showing an example in which a line segment (line) is drawn on a touch panel type input display device 10 using a pen type input device 11.
The input device 11 sequentially moves in the direction of the arrow, and a locus according to the locus of the input device 11 is drawn. That is, a line (line) as a locus of the input device is displayed.
However, when the moving speed of the input device 11 is fast, the locus drawing process cannot follow the moving speed of the input device 11, and the display of the locus line segment may be delayed.

The drawn trajectory 21 shown in the figure is a line displayed on the display according to the trajectory of the input device. The latest drawing position 22 of the drawn trajectory 21 does not coincide with the current pen tip position 15 of the input device 11, and is at the position of the trajectory of the input device 11 before a predetermined time. The drawing delay area 23 shown in the figure is an area where the pen has already been moved according to a predetermined locus, but a line (line segment) corresponding to the locus is not displayed in time, and is a blank area. Such a blank area occurs due to a delay in the drawing process.

As a process for eliminating such a blank area, there is a process using the static filter (function) described in the above [Background Art] section. This is a process of predicting and drawing the line ahead by, for example, linear approximation using a line area of a predetermined length including the latest drawing position 22 in the drawn locus 21 shown in FIG.

In other words, as described above, a linear approximation straight line is generated from the latest drawing portion of the drawn locus, and the line corresponding to the predicted locus is drawn by extending beyond the latest drawing position 22.
However, when such a process is performed, a predicted locus different from the actual pen locus may be generated, and an incorrect locus may be drawn.

An example of occurrence of such a prediction error will be described with reference to FIG.
FIG. 2 shows two examples of prediction errors.
(A) Prediction error example 1 is an example of a prediction error due to overshoot when the pen trajectory suddenly curves and the pen traveling direction fluctuates greatly.
At the latest drawing position 32 of the drawn line 31, as shown by the pen locus 34 shown in the drawing, the pen rapidly changes its traveling direction.

However, when the prediction process using the static filter (function) described above is performed, the data used for the prediction is only a drawn trajectory having a predetermined length including the latest drawing position 32 in the drawn trajectory 31. That is, only the approximate application trajectory 33 shown in the figure is data applied to the prediction.

For example, a line extended according to the line direction of the approximate application locus 33 is set as the predicted locus 35 and displayed.
As a result, as shown in the figure, a predicted track 35 is set at a position completely different from the actual pen track 34.

Another prediction error example shown in FIG. 2 is (b) prediction error example 2, which is an example in a case where a pen as an input device suddenly stops.
This is an example of processing when the pen stops moving and stops at the latest drawing position 42 of the drawn trajectory 41.

When the prediction process using the static filter (function) described above is performed, the data used for the prediction is a drawn line of a predetermined length including the latest drawing position 42 in the drawn locus 41. The approximate application locus 43 shown in the figure is data applied to the prediction.

A line extended according to the line direction of the approximate application locus 43 is set as the predicted locus 45. As a result, as shown in the figure, the predicted trajectory 45 is set before the actual pen position.

As described above, the line segment prediction process using the static filter (function) is a process of applying the line segment immediately before the drawn trajectory to the approximation process. Therefore, if the pen movement mode is different from the approximate application trajectory, The prediction result is completely different from the actual locus of the pen, and an incorrect prediction locus is displayed.

[2. Trajectory Prediction Process Performed by Information Processing Device of Present Disclosure]
Next, a trajectory prediction process example performed by the information processing apparatus according to the present disclosure will be described.
FIG. 3 is a diagram illustrating an example of a trajectory prediction process of the information processing apparatus according to the present disclosure. By applying the processing of the information processing apparatus of the present disclosure, it is possible to accurately estimate and display the predicted trajectory 53 that is a prediction line from the latest drawing position 52 of the drawn trajectory 51 to the pen position 54. This highly accurate estimation and display process of the predicted trajectory 53 reduces the feeling of delay felt by the user and enables drawing processing without a sense of incongruity.

FIG. 4 is a diagram illustrating an outline of a trajectory prediction process performed by the information processing apparatus according to the present disclosure.
In the present embodiment, a learning process (machine learning) to which the configuration information of the drawn trajectory 71 is applied is executed, and a predicted trajectory 73 is set by applying the learning result to perform the drawing process.
The predicted locus 73 is drawn ahead of the latest drawing position 72.
In the present embodiment, the predicted trajectory is estimated by a learning process using not only the information of the drawn trajectory immediately before the latest drawing position but also the information of the past drawing trajectory. This process reduces prediction errors as described above with reference to FIG. 2 and realizes highly accurate prediction.

The trajectory prediction process executed by the information processing apparatus of the present disclosure is a dynamic prediction process different from the static prediction using the conventional fixed static filter (function) described above.
In the static prediction, the line segment immediately before the drawn trajectory is applied to the approximation process, and the past trajectory information does not affect the predicted trajectory. Therefore, when the immediately preceding trajectory applied to prediction is constant, the predicted trajectory is a constant trajectory.
On the other hand, in dynamic prediction, trajectory prediction is performed with reference to past trajectory information other than the line segment immediately before the drawn trajectory, and these past trajectories affect the predicted trajectory. Therefore, even if the previous trajectory applied to the prediction is constant, the predicted trajectory is different if the past trajectory is different.
The past trajectory includes a trajectory before the previous trajectory and includes a trajectory that is not displayed on the display unit.

Hereinafter, as an example of the learning process applied in the present embodiment,
"K Nearest Neighbors (kNN method), k neighborhood method"
A process for estimating a predicted trajectory by a learning process to which is applied will be described.

FIG. 5 is a flowchart illustrating a prediction trajectory estimation and drawing process sequence to which the kNN method is applied.
Note that the processing shown in this flowchart is executed under the control of a data processing unit of the information processing apparatus according to the present disclosure, specifically, a data processing unit including, for example, a CPU having a program execution function. The program is stored, for example, in the memory of the information processing apparatus.

First, the processing of each step of the flow shown in FIG. 5 will be described in order, and then a specific processing example of each step will be described with reference to FIG.
The information processing apparatus performs a process of drawing a line according to the locus of an input device (input object) such as a dedicated pen (stylus).
However, as described with reference to FIG. 1 and the like, a drawing delay area that does not coincide with the drawing process occurs in the area immediately before the current position of the input device (dedicated pen). The flow shown in FIG. 5 is a sequence of processing for drawing a line (line) along the estimated locus by drawing the locus of the input device in the drawing delay area as a predicted locus. For example, it is an estimated drawing process of the predicted trajectory 73 shown in FIG.

(Step S101)
First, in step S101, a plurality (k) of trajectory regions (similar trajectories) similar to the latest drawn trajectory (immediate trajectory) are searched from the drawn trajectories.

The drawn trajectory is a trajectory region in which the trajectory analysis of the input device is completed and the line drawing processing corresponding to the trajectory, that is, the output display processing for the display unit is completed.
The latest drawing trajectory (immediately preceding trajectory) is the latest trajectory region in the drawn trajectory, and is a trajectory region in contact with the drawing delay unit.
Specifically, the immediately preceding locus is a drawn area of a predetermined length including the latest drawing position 72 shown in FIG. In step S101, a plurality (k) of trajectory regions (similar trajectories) similar to the latest drawing trajectory (immediate previous trajectory) are searched from the drawn trajectories. For example, k is a predetermined number such as 3, 5, 10, 20, 30 or the like.

Note that, for example, in the initial stage of drawing, when k similar trajectories determined in advance cannot be searched, a process using only the searched similar trajectories may be performed, or the estimation process may be stopped.

(Step S102)
Next, in step S102, subsequent trajectories of a plurality (k) of trajectory regions (similar trajectories) searched in step S101 are estimated or selected, and these multiple subsequent trajectories are connected to the latest drawing position.
The latest drawing position corresponds to the latest drawing position 72 shown in the example shown in FIG.

(Step S103)
Next, in step S103, an average trajectory of a plurality of connected subsequent trajectories is calculated.

(Step S104)
Finally, in step S104, a drawing process is executed with the average trajectory calculated in step S103 as the final determined predicted trajectory.

A specific example of processing according to the flow shown in FIG. 5 will be described with reference to FIG. FIG. 6 shows drawing lines similar to those shown in FIG. This is an example in which an input device such as a dedicated pen draws a locus from the left to the right, and a line along the locus is drawn.

The dedicated pen that is an input device proceeds ahead of the latest drawing position 81 of the drawn locus, and determines and draws a predicted locus according to the estimated locus before the latest drawing position 81. In the example shown in FIG. 6, the finally set prediction trajectory is the final determined prediction trajectory (Qf) 86 shown in FIG.
The final determined predicted trajectory (Qf) 86 is determined by a process such as averaging the k auxiliary predicted trajectories Q1 to Q3 set according to the extracted k similar trajectories.

The processing in steps S101 to S104 in FIG. 5 will be described with reference to FIG.
(Step S101)
First, in step S101, a plurality (k) of trajectory areas (similar trajectories) similar to the latest drawing trajectory (immediate trajectory) are searched from the drawn trajectories.

This process will be described with reference to the example shown in FIG.
First, the latest drawing locus (immediately preceding locus P) 82 is extracted from the drawn locus 80 shown in FIG.
Further, a plurality (k) of trajectory regions similar to the extracted latest drawn trajectory (previous trajectory P) 82 are searched from the drawn trajectories.
In the example shown in FIG. 6, it is indicated that the following similar trajectory (Rn) of the following three previous trajectories (P) is retrieved with k = 3.
Similar trajectory (R1) 83-1, immediately preceding trajectory (P),
Similar trajectory (R2) 83-2 of the previous trajectory (P),
Similar locus (R3) 83-3 of the immediately preceding locus (P),

The process of step S101 is a process of searching k similar trajectories similar to the latest drawing trajectory (previous trajectory P) 82 from the drawn trajectories.

(Step S102)
Next, in step S102, the subsequent trajectories of the plural (k) trajectory regions (similar trajectories) searched in step S101 are selected, and the selected plural subsequent trajectories are connected to the end of the latest drawing position.

This process will be described with reference to the example shown in FIG.
In step S101, three similar trajectories shown in FIG.
Similar trajectory (R1) 83-1, immediately preceding trajectory (P),
Similar trajectory (R2) 83-2 of the previous trajectory (P),
Similar locus (R3) 83-3 of the immediately preceding locus (P),
These similar trajectories were searched.

In step S102, the subsequent trajectory of these similar trajectories is selected.
In the example shown in FIG. 6, the following three following trajectories are selected.
Subsequent trajectory (A1) 84-1, similar trajectory (R1)
Subsequent locus (A2) 84-2 of the similar locus (R2),
Subsequent locus (A3) 84-3 of the similar locus (R3),

Further, these three subsequent loci 84-1 to 8-3 are connected to the tip of the latest drawing position 81.
Note that the angle between the immediately preceding locus P and the subsequent locus to be connected is an angle that matches the connection angle between the succeeding locus to be connected and the similar locus corresponding to the succeeding locus.

In this way, as shown in FIG. 6, the three subsequent trajectories A1 to A3 are connected to the latest drawing position 81. As shown in FIG. 6, the connected trajectories are the following three auxiliary prediction trajectories.
Auxiliary predicted trajectory (Q1) 85-1, corresponding to the subsequent trajectory (A1),
Auxiliary predicted trajectory (Q2) 85-2 corresponding to the following trajectory (A2),
Auxiliary predicted trajectory (Q3) 85-3 corresponding to the following trajectory (A3),

The process in step S102 is a process for connecting the subsequent locus of the similar locus detected in step S101 to the end of the latest drawing position.

(Step S103)
Next, in step S103, an average trajectory of a plurality of connected subsequent trajectories is calculated.
This process will be described with reference to FIG.

In FIG. 6, the following trajectory connected to the end of the latest drawing position 81 is the following three auxiliary prediction trajectories.
Auxiliary predicted trajectory (Q1) 85-1, corresponding to the subsequent trajectory (A1),
Auxiliary predicted trajectory (Q2) 85-2 corresponding to the following trajectory (A2),
Auxiliary predicted trajectory (Q3) 85-3 corresponding to the following trajectory (A3),

In step S103, average trajectories of these three auxiliary predicted trajectories (Q1 to Q3) 85-1 to 3 are calculated.
As a result, the average trajectory obtained is the trajectory shown in the final determined predicted trajectory (Qf) 86 shown in FIG.

(Step S104)
Finally, in step S104, a process of drawing the average locus calculated in step S103 as the final decision prediction line (final decision prediction locus) is executed.
This process will be described with reference to FIG.
In step S104, a process of drawing the average trajectory calculated in step S103, that is, the final determined predicted trajectory (Qf) 86 shown in FIG. 6 as a predicted trajectory is executed.

[3. Prediction processing example using drawing trajectory coordinate information]
The processing according to the flow shown in FIG. 5 can be executed using the coordinate information (x _t , y _t ) of the drawing trajectory updated for each image frame (t) displayed on the display unit of the information processing apparatus. It is.
Hereinafter, the process of determining the predicted trajectory using this coordinate information will be described.

For example, the information processing apparatus stores, in a memory, coordinate information (x _t , y _t ) of a drawing trajectory newly displayed for each image frame (t) displayed on the display unit. The information processing apparatus stores, in a memory, coordinate information corresponding to a drawing trajectory of a past fixed time from the latest drawing position, and using this coordinate information, a trajectory similarity determination process, a subsequent trajectory connection process, and a connection The final predicted trajectory determination process is performed by calculating the average value of the subsequent trajectories.

The coordinate information (x, y) is stored in the memory corresponding to each display frame (t). For example, the coordinates indicating the latest drawing trajectory position corresponding to the frame t are stored in the memory as (x _t , y _t ). The coordinates indicating the latest drawing locus position corresponding to the next frame t + 1 are stored in the memory as (x _{t + 1} , y _{t + 1} ). In this way, coordinate information as trajectory position information associated with each frame is stored in the memory, and trajectory similarity determination processing and the like are executed using this information.
Hereinafter, a specific processing example using the locus coordinate information will be described.

(3-1. Similarity determination processing)
In step S101 described with reference to the flow shown in FIG. 5, a process of searching the latest drawn locus (immediately preceding locus) and a similar locus from the drawn locus is performed.
In this similarity determination process, the feature amount of the locus is calculated using the coordinate information of the locus, and the similarity is determined by comparing the feature amounts.

For example, the following feature amounts are calculated as the feature amounts applied to the similarity determination.
(1) Speed: Z1
(2) Acceleration: Z2
(3) Angle: Z3
(4) Angle difference: Z4
Each of these feature amounts is calculated using coordinate information corresponding to the drawing trajectory.
Furthermore, the feature amount calculated from the coordinate information constituting the latest drawing locus (immediately preceding locus) is compared with the feature amount calculated from the coordinate information constituting the drawn locus, and the drawn locus having a more similar feature amount. Are extracted as similar trajectories. By these processes, for example, similar trajectories (R1 to R3) 84-1 to 3 shown in FIG. 6 are extracted.

Formulas for calculating the feature quantities Z1 to Z4 are shown below.
(1) Speed: Z1 (t) = sqrt {(x _t −x _t−1 ) ² + (y _t −y _t−1 ) ² }
(2) Acceleration: Z2 (t) = sqrt {(x _t −x _t−1 ) ² + (y _t −y _t−1 ) ² } / sqrt {(x _t−1 −x _t−2 ) ² + (Y _t-1 -y _t-2 ) ² }
(3) Angle: Z3 (t) = atan {(x _t −x _t−1 ) / (y _t −y _t−1 )}
(4) Angular difference: Z4 (t) = atan {(x _t −x _t−1 ) / (y _t −y _t−1 )} − atan {(x _t−1 −x _t−2 ) / (y _t-1 -y _t-2 )}

In addition,
Z1 (t) is the velocity at frame t,
Z2 (t) is the acceleration at frame t,
Z3 (t) is the angle at frame t,
Z4 (t) is the angular difference at frame t,
These are shown.
Also,
sqrt is the square root,
atan is arc tangent
Means.

Note that t is a parameter indicating a frame number in the present embodiment, but processing in which t is set as time information instead of the frame number is also possible.
That is, in the following description, frame t and frame u can be replaced with time t and time u.

The feature amounts Z1 to Z4 will be described with reference to FIG.
FIG. 7 shows a part of the drawn trajectory.
In the figure, the coordinates (x _t−2 , y _t−2 ) to (x _{t + 1} , y _{t + 1} ) of the latest positions of the drawing trajectories at the time of displaying each frame from frame t−2 to frame t + 1 displayed on the display unit are shown. Show. That is, the following four points P1 to P4.
As shown in the lower left of FIG. 7, x corresponds to the horizontal direction in the figure, and y corresponds to the vertical direction.

P1: Latest position coordinates (x _t-2 , y _t-2 ) of the drawing trajectory displayed in the frame t-2
P2: Latest position coordinates (x _t−1 , y _t−1 ) of the drawing trajectory displayed in the frame t−1
P3: Latest position coordinates (x _t , y _t ) of the drawing trajectory displayed in frame t
P4: Latest position coordinates (x _{t + 1} , y _{t + 1} ) of the drawing trajectory displayed in frame t + 1

The above-described feature amounts Z1 (t) to Z4 (t) are calculated as feature amounts corresponding to the coordinates (x _t , y _t ) corresponding to the frame t.
Similar feature values are calculated at coordinate positions corresponding to the frames other than the frame t.

(About speed Z1 (t))
A velocity Z1 (t), which is one of the feature amounts corresponding to the latest coordinates (x _t , y _t ) of the frame t, is calculated by the following equation.
Speed: Z1 (t) = sqrt {(x _t −x _t−1 ) ² + (y _t −y _t−1 ) ² }
This corresponds to the distance La between P3 (x _t , y _t ) and P2 (x _t−1 , y _t−1 ) shown in FIG.
That is, the distance traveled by the trajectory from frame t-1 to frame t, and corresponds to the moving speed of the trajectory between one frame.

(About acceleration Z2 (t))
The acceleration Z2 (t), which is one of the feature amounts corresponding to the latest coordinates (x _t , y _t ) of the frame t, is calculated by the following equation.
Acceleration: Z2 (t) = sqrt {(x _t −x _t−1 ) ² + (y _t −y _t−1 ) ² } / sqrt {(x _t−1 −x _t−2 ) ² + (y _{t -1} -y _t-2 ) ² }
This is because the distance La between P3 (x _t , y _t ) and P2 (x _t−1 , y _t−1 ) shown in FIG. 7, P2 (x _t−1 , y _t−1 ) and P1 (x _This corresponds to the ratio La / Lb between the distance Lb and the distance _t−2 , y _t−2 ).
That is, the value of how many times the trajectory distance La from the frame t-1 to the frame t corresponds to the multiple of the trajectory distance Lb from the frame t-2 to the frame t-1. Yes, this corresponds to the magnification of the trajectory moving speed between the current frames and the trajectory moving speed between the preceding frames.

Note that the feature quantity Z2 (t) indicating acceleration is calculated as a magnification of the speed indicating how many times the subsequent speed corresponds to the previous speed in the above example, but the subsequent speed and the previous speed May be calculated as the difference between
In this case, the feature amount Z2 (t) is calculated by the following equation.
Acceleration: Z2 (t) = sqrt {(x _t −x _t−1 ) ² + (y _t −y _t−1 ) ² } −sqrt {(x _t−1 −x _t−2 ) ² + (y _{t -1} -y _t-2 ) ² }
This is because the distance La between P3 (x _t , y _t ) and P2 (x _t−1 , y _t−1 ) shown in FIG. 7, P2 (x _t−1 , y _t−1 ) and P1 (x _This corresponds to the difference La−Lb between the distance Lb and _t−2 , y _t−2 ).

(About angle Z3 (t))
An angle Z3 (t), which is one of the feature amounts corresponding to the latest coordinates (x _t , y _t ) of the frame t, is calculated by the following equation.
Angle: Z3 (t) = atan {(x _t −x _t−1 ) / (y _t −y _t−1 )}
This is a triangle composed of a line segment La having vertices P3 (x _t , y _t ) and P2 (x _t−1 , y _t−1 ) shown in FIG. 7, a horizontal line Wa, and a vertical line Ha. It is expanded as follows.
Z3 (t) = atan {(x _t −x _t−1 ) / (y _t −y _t−1 )}
= Atan (Wa / Ha)
Therefore, Z3 (t) = atan {(x _t −x _t−1 ) / (y _t −y _t−1 )}
Is, P3 shown in FIG. 7 _(x _{t, y} t) and _{_{P2 (x t-1, y}} t-1) and the apex line segments La and the horizontal line Wa, the apex of the triangle formed by the vertical line Ha P3 This corresponds to the angle α of (x _t , y _t ).

(Regarding the angle difference Z4 (t))
An angle difference Z4 (t), which is one of the feature amounts corresponding to the latest coordinates (x _t , y _t ) of the frame t, is calculated by the following equation.
Angular difference: Z4 (t) = atan {(x _t −x _t−1 ) / (y _t −y _t−1 )} − atan {(x _t−1 −x _t−2 ) / (y _t−1 -Y _t-2 )}
This is a triangular shape composed of a line segment La having apexes P3 (x _t , y _t ) and P2 (x _t−1 , y _t−1 ) shown in FIG. 7, a horizontal line Wa, and a vertical line Ha. The angle α of the vertex P3 (x _t , y _t ),
A triangle constituted by a line segment Lb having P2 (x _t−1 , y _t−1 ) and P1 (x _t−2 , y _t−2 ) as vertices, a horizontal line Wb, and a vertical line Hb shown in FIG. Angle β of vertex P2 (x _t−1 , y _t−1 ) of
It corresponds to the difference (α−β) of each degree.

These four feature quantities Z1 to Z4 are sequentially obtained for the latest coordinate point of each frame and stored in the memory. When the memory capacity is limited, for example, the latest drawing position 81 shown in FIG. 6 is used to set the coordinate information corresponding to the predetermined number of frames, such as the past 100 frames, to a newer frame. The update of the memory data in which the coordinate information of the oldest frame is deleted may be executed in response to the input of the corresponding coordinate information.

If the memory capacity is sufficient, handwriting-corresponding trajectory data, including past data, is stored in the non-volatile memory, and it remains in the memory even when the information processing device is turned off. When drawing is started, the prediction trajectory may be estimated by performing similarity determination using the past accumulated data.
Further, the memory storage data may be associated with the user identifier (user ID), and processing using the user-corresponding storage data may be performed according to the user.

The search for the similar trajectory is performed by calculating the feature amount using the coordinate information corresponding to the trajectory for a plurality of frames stored in the memory, and comparing the feature amounts.
Note that the comparison target is the feature amount of the immediately preceding locus and the feature amount obtained from the other past locus.
By this comparison processing, a predetermined number k of past locus regions having a high degree of similarity are selected, for example, three in the example of FIG.

A determination formula for similarity is shown below.
D (t, t ′) = Σ _i Σ ⁴ _{j = 1} w _j (Z _j (t i) −Z _j (t′−i)) ²
The above is the similarity determination formula,
It is determined that the similarity is higher as the feature amount distance D (t, t ′) calculated by the above equation is smaller.

In the above similarity determination formula,
t ′ corresponds to the frame number in which the locus of the coordinates (x _{t ′} , y _{t ′} ) of the latest drawing position 81 shown in FIG. 6 is displayed as the update locus. That is, it is the frame number on which the latest locus just before the predicted locus has been drawn.
t corresponds to a past arbitrary frame displaying coordinates (x _t , y _t ) of an arbitrary position on the drawn trajectory.

i corresponds to a frame section to be subjected to similarity comparison, and corresponds to, for example, the number of frames necessary for drawing the locus of the immediately preceding locus 82 shown in FIG.
For example, when the immediately preceding trajectory 82 shown in FIG. 6 is a trajectory generated by the display process for five frames, a comparison process using information on coordinate positions for five frames is executed.

That is, as shown in FIG. 8, when the previous trajectory 82 is a trajectory generated by the display process of 5 frames, the similarity determination target for the previous product is selected from the drawn trajectories 80 before the previous trajectory 82. This is the trajectory information for all five frames.

j is a coefficient corresponding to the type of feature quantity. That is, it corresponds to the respective coefficients 1 to 4 of Z1 to Z4.
wj is a weight set for each feature quantity Zj = Z1 to Z4. This weight can be set to various values depending on the situation. Specifically, all the weights wj = 1 may be set to one value.

For example, as an example of weight setting when the display device is full HD,
Weight w1 = 1 corresponding to speed Z1,
Weight w2 = 100 corresponding to acceleration Z2;
Weight w3 = 10 corresponding to angle Z3,
Weight w4 = 1000 corresponding to angle difference Z4,
The above weight setting may be used.

The feature amount distance D (t, t ′) is calculated, and a predetermined selection number, that is, k pieces, is selected in ascending order of the feature amount distance. The selected k areas are set as similar trajectories. In this way, for example, similar trajectories R1 to R3 shown in FIG. 6 are selected. In the example of FIG. 6, k = 3.

In addition, the weight wj set for each feature amount may be configured to increase the weight for newer data, for example. In other words, it may be configured to calculate feature amount distance data in which a weight according to the distance is set so as to preferentially select a trajectory that is close to the previous trajectory.
The formula for calculating the feature amount distance D (t, t ′) when the similarity is calculated by increasing the weight for newer data is set as follows.
D (t, t ′)
= Σ _i pow (ε, i) Σ ⁴ _{j = 1} w _j (Z _j (t i) −Z _j (t′−i)) ²
However,
ε: Weight attenuation rate set in advance (0.0 <ε ≦ 1.0)
pow (ε, i): Weight (function that outputs 1.0 at a newer locus position and outputs a smaller value as it gets older)
As described above, the feature distance may be calculated by setting a larger weight of a new locus.

(3-2. Predictive locus determination process)
Next, processing for determining a predicted trajectory using a plurality of similar trajectories selected by the above-described processing will be described.
In steps S102 to S104 described with reference to the flow shown in FIG. 5, a plurality of similar trajectories selected in step S101 are connected to the tip of the latest drawing position, and the averaging process is performed to obtain a final predicted trajectory. The process of determining is performed.
This predicted locus determination process will be described.

The coordinates (x _u , y _u ) constituting the predicted trajectory are calculated according to the following formula (predicted trajectory coordinate calculation formula).
x _u = (1 / k) Σ ^k _{n = 1} x _u (n) ′
_yu = (1 / k) [Sigma] ^kn _{= 1} _yu (n) '

Note that u in the predicted trajectory coordinate calculation formula corresponds to a frame number. For example, when the frame number of the display frame at the latest drawing position 81 shown in FIG. 6 is t, the frame number is after the frame number t, and t <u.
The frame u corresponds to a future frame in which the drawing of the locus has not been completed.

The other parameters are set as follows.
k is the number of extracted similar trajectories.
n is a variable of 1 to k.
x _u (n) ′ is the x coordinate in the frame u of the prediction line corresponding to each of the similar trajectories of n = 1 to k,
y _u (n) ′ is the y coordinate in the frame u of the prediction line corresponding to each of the similar trajectories of n = 1 to k,

The calculation process of the x coordinate: x _u (n) ′ and the y coordinate: _yu (n) ′ in the frame u of the predicted line corresponding to each of the n = 1 to k similar trajectories used in the predicted trajectory coordinate calculation formula will be described. To do.

First, the velocity V _t (n), angle a _t (n), coordinates (x _t (n) of the tip positions (latest positions) of k similar trajectories extracted in the above-described processing (step S101 in the flow shown in FIG. 5). n) ′, y _t (n) ′) is calculated as follows.
(1) Latest speed: v _t (n) = sqrt {(x _t −x _t−1 ) ² + (y _t −y _t−1 ) ² }
(2) Latest angle: a _t (n) = atan {(x _t −x _t−1 ) / (y _t −y _t−1 )}
(3) Latest coordinates: x _t (n) ′ = x _t , y _t (n) ′ = y _t
Here, n is a variable corresponding to the number of extracted similar trajectories (k), and n = 1 to k.

Next, the velocity: V _u (n) and the angle: a _u (n) in the future frame (frame u) predicted according to the nth (n = 1 to k) similar trajectories are calculated according to the following calculation formula. calculate.
V _u (n) = V _u−1 (n) Z ₂ (sn + ut)
a _u (n) = a _u−1 (n) Z ₄ (sn + u−t)
Here, sn is a frame number of each of k similar trajectories with n = 1 to k.

Furthermore, by applying the above calculation formula, the xy coordinates in the future frame (frame u) predicted according to the nth (n = 1 to k) similar trajectories: (x _u (n) ′, _yu ( n) ') is calculated according to the following calculation formula.
x _u (n) ′ = x _u−1 (n) ′ + V _u (n) cos (a _u (n))
_yu (n) '= _yu-1 (n)' + _Vu (n) sin ( _au (n))

In accordance with the above formula, the xy coordinates in the future frame u corresponding to the k similar trajectories are calculated. These coordinates are xy coordinates constituting the k auxiliary prediction trajectories shown in FIG.

Applying the xy coordinates (x _u (n) ′, _yu (n) ′) corresponding to the frame u corresponding to the k similar trajectories calculated according to the above formula, the predicted trajectory coordinate calculation formula described above is used. Use to calculate the constituent coordinates of the final predicted trajectory.
That is, as described above, the coordinates (x _u , _yu ) constituting the predicted trajectory are calculated according to the following formula (predicted trajectory coordinate calculation formula).
x _u = (1 / k) Σ ^k _{n = 1} x _u (n) ′
_yu = (1 / k) [Sigma] ^kn _{= 1} _yu (n) '
The trajectory calculated according to the above formula is the final determined predicted trajectory (Qf) 86 shown in FIG.

In the above trajectory calculation formula, the final predicted trajectory is calculated by simply averaging the constituent coordinates of the predicted trajectory calculated based on all similar trajectories. A configuration may be adopted in which the final predicted trajectory coordinates are calculated by increasing the weight of the trajectory and performing addition averaging.

This process will be described.
Set the parameters as follows:
s _n : time (or frame) of k similar points
r _n : similarity rank of k similar points (most similar point = 1, least similar point = k)
_{_{q n = pow (β, r}} n-1): n -th weights similarities (now in the most similar point _(r n = 1) 1.0, the smaller the value of _{r n} increases)
β: preset weight decay rate (0.0 <β ≦ 1.0)

Using the above parameters, the coordinates (x _u , _yu ) constituting the predicted trajectory are calculated according to the following formula (predicted trajectory coordinate calculation formula).
x _u = (Σ ^k _{n = 1} q _n x _u (n) ′) / (Σ ^k _{n = 1} q _n )
y _u = (Σ ^k _{n = 1} q _n y _u (n) ′) / (Σ ^k _{n = 1} q _n )

In this manner, the coordinates of the predicted trajectory may be calculated by performing weighting according to the similarity.

[4. Example of processing according to the reliability of the predicted trajectory]
When a prediction process based on so-called machine learning such as the kNN method described above is executed, the prediction accuracy decreases when a “relatively infrequent locus” is drawn.
“Infrequent trajectory” is, for example, “rapid direction change (often used for kanji characters)”, “stop”, “continuous curve”, and the like.

That is, the prediction accuracy by the learning process is increased for a frequently appearing trajectory, but the prediction accuracy is decreased for a trajectory having a low frequency. In extreme terms, accuracy is reduced except for straight lines and relatively smooth curves. This is particularly noticeable when learning progress is not sufficient. If the accuracy of the prediction decreases, the probability that the predicted locus is drawn at a position different from the position of the input device such as the dedicated pen increases.
Such drawing of an incorrect predicted trajectory may be annoying for the user.
Therefore, when it is determined that the prediction accuracy is low, it is preferable not to reflect the prediction result in the drawing.
Hereinafter, this processing example will be described.

First, the reliability of the predicted trajectory will be described with reference to FIG.
FIG. 9 shows a drawn trajectory 90 and a previous trajectory 91 that is the tip portion thereof. A predicted locus is set ahead of the immediately preceding locus 91.
Based on the kNN method described above, k auxiliary prediction trajectories 92 are calculated at an arbitrary time point. Since the k predicted trajectories represent k similar trajectories in the past, in the case of a simple trajectory such as a straight line, the positions of the k trajectories converge, and conversely, When there is a change or the like, the positions of the k trajectories are scattered.

The fact that a plurality of auxiliary predicted trajectories 92 are “disjoint” means that “the accuracy of an optimal past trajectory that is close to the current trajectory is low. Therefore, the standard deviation σ at each coordinate position of the k auxiliary prediction trajectories is calculated, and processing corresponding to the calculated standard deviation σ is performed.
Specifically, only when the calculated standard deviation is small, the average of k auxiliary predicted trajectories is calculated, and the coordinate position of the final determined predicted trajectory is determined and drawn. Thereby, the mistake of the prediction perceived by the user can be reduced.

The circles (standard deviations 95U1 to 3 of the auxiliary prediction trajectory) shown in FIG. 9 are coordinate points of a plurality (k) of auxiliary prediction trajectories in the future frames U = t + 1, t + 2, and t + 3 after the display frame t of the previous trajectory 91. This is a circle conceptually showing the size of the standard deviation.
The larger the size of the circle, the larger the standard deviation, that is, the greater the variation in the auxiliary predicted trajectory, indicating that the final determined predicted trajectory to be calculated as an average value is uncertain.

The final position of the immediately preceding trajectory 91 corresponds to the coordinate position of the trajectory displayed in the frame t, and the subsequent small circle 95U1 is a standard of a plurality of predicted coordinates calculated by a plurality of auxiliary predicted trajectories in the future frame U = t + 1. The deviation is shown.
The next circle 95U2 shows the standard deviation of the plurality of predicted coordinates calculated by the plurality of auxiliary prediction trajectories of the future frame U = t + 2.
The next circle 95U3 shows the standard deviation of the plurality of predicted coordinates calculated by the plurality of auxiliary prediction trajectories of the future frame U = t + 3.

* Only when the standard deviation is small, that is, when there is little variation in prediction, the average of k auxiliary prediction trajectories is calculated and the coordinate position of the final determined prediction trajectory is determined and drawn. Thereby, the mistake of the prediction perceived by the user can be reduced.

The sequence of this standard deviation calculation process and the calculation / drawing process of the predicted trajectory accompanied by the process change according to the process result will be described with reference to the flowchart shown in FIG.

FIG. 10 is a diagram illustrating a flowchart for explaining a prediction trajectory estimation and drawing process sequence to which the kNN method is applied, similarly to the flow of FIG. 5 described above.
Note that the processing shown in this flowchart is executed under the control of a data processing unit of the information processing apparatus according to the present disclosure, specifically, a data processing unit including, for example, a CPU having a program execution function. The program is stored, for example, in the memory of the information processing apparatus.

(Step S201)
Steps S201 to S202 are the same processing as the processing of steps S101 to S102 in the flow shown in FIG.
First, in step S201, a plurality (k) of trajectory regions (similar trajectories) similar to the latest drawing trajectory (immediate previous trajectory) are searched from the drawn trajectories.

The drawn trajectory is a trajectory region in which the trajectory analysis of the input device is completed and the line drawing processing corresponding to the trajectory, that is, the output display processing for the display unit is completed.
The latest drawing trajectory (immediately preceding trajectory) is the latest trajectory region in the drawn trajectory, and is a trajectory region in contact with the drawing delay unit.
Specifically, the immediately preceding locus is, for example, a drawn area of a predetermined length including the latest drawing position 72 shown in FIG. In step S201, a plurality (k) of trajectory regions (similar trajectories) similar to the latest drawing trajectory (immediate previous trajectory) are searched from the drawn trajectories. For example, k is a predetermined number such as 3, 5, 10, 20, 30 or the like.

(Step S202)
Next, in step S202, subsequent trajectories of a plurality (k) of trajectory regions (similar trajectories) searched in step S201 are estimated or selected, and the plurality of subsequent trajectories are connected to the tip of the latest drawing position.
The latest drawing position corresponds to the latest drawing position 72 shown in the example shown in FIG.

(Step S203)
After step S203, the process takes into account the standard deviation of the predicted trajectory.
For a plurality of prediction trajectories corresponding to a plurality of similar trajectories, that is, k auxiliary prediction trajectories shown in FIG. 9, standard deviation calculation is executed for each future frame, and processing is performed for each frame according to the calculation result.
First, in step S203, the first future frame U is set as U = t + 1.
Frame t is the frame number of the frame that displays the latest drawing position at the tip of the immediately preceding locus. U = t + 1 corresponds to the frame next to the display frame t at the latest drawing position.

(Step S204)
In step S204, the standard deviation σu of the coordinate position in the frame U of the k auxiliary prediction trajectories is calculated.
Note that the coordinate positions corresponding to the k auxiliary predicted trajectories are executed as the same process as described with reference to the flow of FIG.
In step S204, the standard deviation σu of k coordinate positions corresponding to k frames U is further calculated.

(Step S205)
Next, in step S205, it is determined whether or not drawing of the predicted trajectory is appropriate based on the standard deviation σu corresponding to the frame U, that is, whether or not a predicted trajectory with reliability can be drawn.

This reliability determination is executed by applying the following determination formula.
Judgment formula σu <αVt

In the above formula,
αVt corresponds to the drawing determination threshold value. In addition,
α: Pre-set coefficient Vt: Drawing speed in the previous trajectory.
Vt corresponds to the length s of the immediately preceding locus in FIG. s corresponds to the distance of the trajectory advanced between one frame t-1 and frame t.
Vt is the length of the trajectory advanced in the immediately preceding frame, and corresponds to the speed of the immediately preceding trajectory 81.
The coefficient α can be changed according to the situation, for example, if the setting is to allow drawing only when the reliability is high, the value is reduced, and if the drawing is allowed even at low reliability, the value is increased. For example, a value that can be set by the user.

If the above determination formula is satisfied, that is, if the standard deviation σu is smaller than the threshold value, it is determined that a relatively reliable prediction trajectory can be determined, and the process proceeds to step S206.
On the other hand, if the determination formula is not satisfied, that is, if the standard deviation σu is equal to or greater than the threshold, it is determined that it is difficult to draw a highly reliable predicted trajectory, and the process proceeds to step S211.

(Step S206)
In step S206, an average coordinate of k coordinates corresponding to the frame U of a plurality (k) of auxiliary prediction trajectories is calculated.

(Step S207)
In step S207, the average coordinates calculated in step S206 are determined as the coordinates of the final determined predicted trajectory corresponding to the frame U, and the predicted trajectory drawing process is executed.

(Step S208)
In step S208, it is determined whether or not there is an unprocessed frame in which the drawing process of the freem U is to be executed.
If there is, the process proceeds to step S209, and if not, the process ends.

(Step S209)
In step S209, the frame number U is updated. That is,
U = U + 1
The frame number that has been set is updated, and the processing of the next frame is started in step S204 and subsequent steps.
When all the unprocessed frames have been processed, the process ends.

(Step S211)
Step S211 is a process executed when No is determined in the determination process of step S205.
In other words, the process is executed when it is determined in step S205 that the standard deviation σu is equal to or greater than the threshold value and it is difficult to draw a highly reliable predicted trajectory.
In this case, in step S211, a determination is made to end the prediction process. Or the prediction system change process switched to the conventional static prediction process is performed.

As described above, by executing the processing according to the flow shown in FIG. 10, it becomes possible to draw a predicted trajectory only when the reliability is high, and it is erroneous due to drawing a predictive trajectory with low reliability. The display of the predicted trajectory can be suppressed.

[5. About processing using pressure detection information of input device]
For example, when a dedicated pen is applied as an input device, it is possible to detect the pressure on the display unit of the pen, calculate reliability according to the pressure, and perform drawing control of the predicted trajectory according to the calculated reliability.

The pressure value for the display unit of the input device is detected by, for example, the input device or a pressure detection sensor set on the surface of the display unit, and the detected value is input to the control unit and used for reliability calculation. Is done.

When a dedicated pen is used as an input device, as a feature of pen input, when the pen leaves the screen, the pressure value gradually decreases from several frames before the pen leaves (releases) the display unit surface.
In particular, the tendency of the pressure value to gradually decrease becomes noticeable when the character “Hara” of the character is drawn.

In the embodiment described above, the following feature amounts are used as the feature amounts used for the determination process of the similar trajectory.
(1) Speed: Z1
(2) Acceleration: Z2
(3) Angle: Z3
(4) Angle difference: Z4
In addition to each of these feature amounts, a pressure value: Z5 and a pressure value change amount: Z6 corresponding to the writing pressure applied to the display unit of the input device are added.

Specifically, the pressure value in the frame t (or time t) as p _t, is defined as follows a feature amount Z5, Z6.
Pressure value of frame t: Z5 (t) = _pt
Pressure value change amount of frame t: Z6 (t) = p _t −p _t−1
These feature amounts Z5 and Z6 are added to determine a similar locus.

In addition, when performing the pen pressure prediction in the predicted trajectory after the similarity determination, the predicted pen pressure (pressure value) pu in the future frame u (or future time t) on the predicted trajectory can be calculated by the following equation.
Predictive writing pressure: p _u = (1 / k) Σ ^k _{n = 1} p _{sn + u−t}

U is the frame number (or time)
k is the number of extracted similar trajectories.
n is a variable of 1 to k.
sn is the frame number (or time) of each of k similar trajectories with n = 1 to k,
It is.

導入 By introducing a pressure value corresponding to writing pressure as a feature value of the learning process, for example, the accuracy of prediction at the time of pen release is improved.

[6. Calculation of predicted trajectory and modification of drawing process]
Next, some modified examples that can be expected to further improve the effect will be described below using the above-described embodiment as a basic process.

(6-1) Process of changing learning pattern according to type of drawing object For example, the learning pattern is changed according to the type of drawing object input by the user. Specifically, if the user's drawing object is a picture, a character, or even a character, the type, for example, alphabet, Japanese, kanji or hiragana, etc. Change the learning pattern according to the type.

The prediction process is executed by applying the fact that the shape features differ for each drawing object such as a picture or text. For example, compared to alphabets, Kanji has a straight line or sharp angle switch, has many short lines, etc., and the characteristics are greatly different from alphabets that use a lot of long curves as a whole.
As described above, by changing the feature amount used for the extraction of the similar trajectory and the determination of the predicted trajectory according to the type of the drawing object, it is possible to perform processing with higher accuracy.

(6-2) Process for Changing Predicted Trajectory Drawing Mode Depending on Relative Position of Line of Sight and Pen Tip When drawing a predicted trajectory on the display unit, an example of processing for changing the drawing mode of the predicted trajectory depending on the relative position of the line of sight and the pen tip Will be described.

For example, when a line is drawn using a dedicated pen as an input device, a right-handed user holds the pen in the right hand, and when drawing a line from left to right, the space on the left side of the right hand can be observed well. However, if you draw a line from right to left, the right side of the space will be hidden by hand.

Even if the predicted trajectory is drawn in such a portion that cannot be seen, the user cannot observe it well. Therefore, such a prediction process is not executed. Alternatively, measures such as reducing the number of predicted frames are taken.
That is, if the line segment is drawn from the left to the right, the number of predicted frames is increased, and if the line segment is drawn from the right to the left, control is performed to stop the prediction or decrease the number of predicted frames. Do,
In the case of a left-handed user, the process opposite to that for a right-handed user is performed.

(6-3. Example of processing using a combination with other prediction processing)
By using other prediction processes together with the above-described trajectory prediction process, it is possible to further improve the prediction accuracy.

((A) Combined use with word prediction processing)
For example, when the user's drawing object is a document, word prediction can be performed, and the prediction accuracy can be improved using the word prediction result.
For example, the next character to be written is estimated by word prediction, the locus corresponding to the estimated character is estimated, and the weight of the auxiliary prediction locus similar to the estimation result is increased.

((B) Processing for switching between dynamic prediction and static prediction)
The process of searching for similar trajectories from the past trajectories and applying the learning process for determining the predicted trajectory is a dynamic prediction process. However, the trajectory estimation process by linear approximation using only the data of the previous trajectory is performed. Conventional static prediction processing performed may be effective.

When learning in the environment is not sufficiently advanced, such as when the drawing application is activated or when the user changes, the accuracy of the dynamic prediction method decreases. In such a case, a prediction process using a static prediction method may be more effective. Thereafter, when a drawn trajectory sufficient for searching for a similar trajectory is obtained, the dynamic prediction is switched.
In this way, by switching between dynamic prediction and static prediction according to the situation, optimum prediction according to the situation becomes possible.

((C) Combined with prediction considering character database)
Generally, in a device capable of inputting characters, a character database (font database) is stored in a memory. Many of such devices have a function of performing a process of searching a character database for characters similar to a user's drawing trajectory.
Using this function, a character similar to the character written by the user is selected from the database, a trajectory corresponding to the selected character is estimated, and the weight of the auxiliary predicted trajectory similar to the estimation result is increased. .

((D) Example of prediction processing independent of character scale)
For example, when a user draws a character, the size (scale) differs depending on the time and the case. In the above-described embodiment, in the determination of the similar trajectory, when the scale is different, it may be difficult to determine the personal similarity.
In order to solve this problem, for example, a character size of one character drawn by the user is determined, a process of converting the determined character size into an absolute scale having a preset standard size is performed, and a locus having the absolute scale is executed. A comparison process using is performed.
Note that the scale conversion processing is executed by combining, for example, enlargement processing, reduction processing, interpolation processing, and thinning processing.
By performing such processing, it is possible to reduce erroneous determinations based on scale differences.

(6-4. Other modifications)
Various settings can be made as to how far the predicted trajectory is drawn. For example, it may be set to perform maximum processing according to the processing capability of the information processing apparatus, set to a predetermined distance from the actual pen position specified in advance, to the position in front, or set to a user's desired range It is good also as a possible structure.
Note that the processing capability (performance) of the information processing apparatus may be benchmarked and calculated, and the prediction trajectory drawing range may be set in accordance with the calculation result.

Also, the drawing type of the predicted trajectory is determined by determining the device type of the information processing terminal. Or it is good also as a structure which determines the drawing range of a prediction locus | trajectory according to execution applications, such as an application which draws a character, or an application which performs drawing of a picture.

Further, when similarity determination processing using a past drawing trajectory or the like is performed, the past similar trajectory is stored in the memory in association with the user ID that is the identifier of the user who has drawn the trajectory, and the similarity determination or predicted trajectory is stored. When drawing the above, it is possible to select and use past trajectory information of the same user.
Further, when using the device, the setting for determining whether or not the user is right-handed or left-handed or changing the drawing setting of the predicted trajectory according to the dominant hand information may be adopted.

As a method for determining the delay time from the hardware configuration and determining the number of predicted frames, for example, the following processing can be applied.
First, information such as the product number of the information processing apparatus terminal, the ID of the touch panel used in the terminal, the ID of the touch panel driver, and the ID of the graphic chip is acquired.
Next, the delay time is estimated by collating with a database storing correspondence data of these IDs and delay times, and the number of predicted frames is determined.
The database referred to at this time may be stored in the information processing terminal, or may be configured to be placed on a server on the network. When the server is used, the acquired ID information is transmitted from the terminal to the server, and the estimated delay time or the predicted number of frames is received from the server.
Further, the configuration may be such that the number of predicted frames is determined by actually measuring the delay time. That is, the time from input detection by the input device to the completion of drawing may be measured internally, and the number of predicted frames may be determined based on the measured time. Note that input detection and detection of drawing completion by the input device may be estimated based on signals from the touch detection driver and the graphic driver, or may be estimated by recognizing an image based on an image captured by the camera.

In addition, it is good also as a structure which can be switched according to an input device whether a locus | trajectory prediction process is performed.
For example, when the input device is a dedicated pen, trajectory prediction is executed, and when the input device is a finger (touch input), the trajectory prediction may not be executed.
Other input devices include, for example, a mouse, and can be configured to switch whether or not to perform trajectory prediction according to various input devices.

Moreover, it is good also as a structure which can set whether a locus | trajectory prediction process is performed by user setting.
Furthermore, the difference between the predicted trajectory and the actual trajectory may be sequentially calculated, and when the difference is equal to or greater than a predetermined threshold, control for stopping the trajectory prediction (OFF) may be performed.

Also, when the trajectory prediction is executed and the drawing application is terminated, the learning result data applied to the trajectory prediction is stored in the memory as log data associated with the user ID of the application. By performing such processing, when the application is started up again, the user-corresponding learning data stored in the memory can be used by inputting the user ID, and the user-corresponding learning result is used. Trajectory prediction can be executed immediately.

[7. Example of controlling drawing mode of predicted trajectory]
Next, an embodiment for controlling the drawing mode of the predicted trajectory will be described.
As described above, in the process of the present disclosure, a process of drawing a predicted trajectory is performed by predicting a future trajectory through learning from a drawn trajectory (actual trajectory).
However, the predicted trajectory displayed on the display unit may deviate from the actual trajectory. As described above, if the “predicted trajectory” that may be shifted from the actual trajectory is displayed in the same manner as the “drawn trajectory” corresponding to the actual trajectory, it may be annoying for the user.
In the following, an embodiment will be described in which the display mode of the “predicted trajectory” having such a possibility of error is displayed in a mode different from the “drawn trajectory” that is the actual trajectory.

The predicted trajectory display control processing method will be described sequentially in the following five processing examples.
(Processing Example 1) The “drawn locus” corresponding to the actual trajectory is displayed as a solid line, and the “predicted trajectory” is displayed by changing at least one of the color, the transmittance, and the thickness so as not to stand out from the solid line.
(Processing Example 2) The predicted trajectory is updated every frame. That is, the displayed predicted trajectory is erased in the next drawing frame, and a new predicted trajectory is drawn.
(Processing Example 3) At least one of length, color, transmittance, and thickness is changed according to the reliability index value of the predicted trajectory.
(Processing Example 4) Depending on the situation, drawing of the predicted trajectory is turned on / off, that is, switching between display and non-display is performed.
(Processing Example 5) When the predicted trajectory deviates from the actual trajectory, control is performed to make the predicted trajectory inconspicuous by changing the display mode of the predicted trajectory.

Note that the process of determining the predicted trajectory is the same as the process described above. In the display control processing example described below, processing for controlling the display mode of the predicted trajectory determined in accordance with the predicted trajectory estimation processing described above is performed.

FIG. 11 is a diagram similar to FIG. 9 described above, and shows an example of a plurality of (k) auxiliary prediction trajectories applied to the prediction trajectory determination process and the standard deviation of each coordinate position of the auxiliary prediction trajectory. It is.
FIG. 11 shows a drawn trajectory 100 displayed according to an actual trajectory that is an actual trajectory of a dedicated pen as an input device, and auxiliary prediction calculated by the above-described embodiment, for example, processing according to the flow shown in FIG. A trajectory 103 and a predicted trajectory 105 calculated as an average trajectory of the auxiliary predicted trajectory 103 are shown.

The tip of the drawn trajectory 100 is the latest drawing position 102 of coordinates (x, y), and the latest drawing position 102 is the latest drawing position 102 displayed in the frame t.
The drawing locus from frame t-1 to frame t is the locus immediately before the length s.

As described above, k auxiliary prediction trajectories 103 are calculated at an arbitrary time point based on the kNN method. The k prediction trajectories have different variations depending on the situation. The large variation in the plurality of auxiliary prediction trajectories 103 means that “the accuracy of the optimum past trajectory that is close to the current trajectory is low. That is, the prediction trajectory 105 set by taking k average values. Is less likely to be equal to the actual trajectory.

The standard deviation σ of each coordinate position of the k auxiliary prediction trajectories is calculated as an index indicating the degree of variation. The circles (standard deviations 104-U1 to U3 of the auxiliary prediction trajectory) shown in FIG. 11 indicate a plurality (k) of auxiliary prediction trajectories in the future frames U = t + 1, t + 2, and t + 3 after the display frame t of the immediately preceding trajectory 101. A circle conceptually showing the size of the standard deviation of coordinate points.
The larger the size of the circle, the larger the standard deviation, that is, the greater the variation in the auxiliary predicted trajectory, indicating that the final determined predicted trajectory to be calculated as an average value is uncertain.

The final position of the previous trajectory 101 corresponds to the coordinate position of the trajectory displayed in the frame t, and the subsequent small circle 104-U1 has a plurality of predicted coordinates calculated by a plurality of auxiliary predicted trajectories in the future frame U = t + 1. The standard deviation is shown.
The next circle 104-U2 shows the standard deviation of the plurality of predicted coordinates calculated by the plurality of auxiliary prediction trajectories of the future frame U = t + 2.
The next circle 104-U3 shows the standard deviation of the plurality of predicted coordinates calculated by the plurality of auxiliary prediction trajectories of the future frame U = t + 3.
In this embodiment, this standard deviation is used as a reliability index value corresponding to the predicted trajectory, and display control corresponding to the reliability index value is executed, for example.

FIG. 12 is a diagram illustrating a calculation example of the reliability of each future frame U = t + 1, t + 2, t + 3.
As shown in FIG. 12, the predicted trajectory 105 calculated by the process according to the flow shown in FIG. 5 is drawn ahead of the latest drawing position (x, y) 102 displayed in the frame t. The latest drawing position 102 is the latest drawing position 102 displayed in the frame t. The drawing locus from frame t-1 to frame t is the locus immediately before the length s.

The predicted trajectory 105 calculated by the process according to the flow shown in FIG. 5 is drawn ahead of the latest drawing position (x, y) 102 displayed in the frame t.
The predicted trajectory is set by setting to connect the following points.
A predicted point 111 consisting of predicted coordinates (px1, py1) of the future frame u = t + 1,
A predicted point 112 consisting of predicted coordinates (px2, py2) of the future frame u = t + 2,
A predicted point 113 composed of predicted coordinates (px3, py3) of the future frame u = t + 3,
The predicted trajectory 105 is set and displayed with the setting connecting these points.

The predicted trajectory 105 is a trajectory calculated as an average trajectory of auxiliary predicted trajectories calculated according to a plurality of similar trajectories as described above with reference to FIGS.
That is, as described above with reference to FIG. 6, a plurality (k) of similar trajectories similar to the region including the previous trajectory up to the latest drawing position 102 are extracted from the drawn trajectory 100, and this similarity is extracted. The average value of the k auxiliary predicted trajectories set based on the trajectory is set as the predicted trajectory 105.

As described with reference to FIG. 11, the extracted k auxiliary prediction trajectories have a predetermined variation. The degree of variation of the k auxiliary prediction trajectories is calculated as a standard deviation σ, and this is used as a reliability index value.
For example, the reliability index value (= standard deviation) of each point on the predicted trajectory 105 ahead of the latest drawing position (x, y) displayed in the frame t is set as shown in FIG.

The standard deviation calculated based on the coordinate positions corresponding to the k auxiliary prediction trajectories, that is, the reliability index value is set as follows.
Reliability index value of prediction point 111 consisting of prediction coordinates (px1, py1) of future frame u = t + 1: SD [1] = 0.08,
Reliability index value of the prediction point 112 composed of the predicted coordinates (px2, py2) of the future frame u = t + 2: SD [2] = 3.75
Reliability index value of prediction point 113 consisting of prediction coordinates (px3, py3) of future frame u = t + 3: SD [3] = 8.52.
Thus, the reliability index value corresponding to each frame is calculated.
The reliability index value corresponds to the standard deviation indicating the degree of variation of the auxiliary prediction trajectory as described above, and is a value indicating that the smaller the value is, the higher the reliability is, and the larger the value is, the lower the reliability is.

The display mode of the predicted trajectory is controlled according to the reliability index value SD [u] of the predicted trajectory corresponding to each future frame u.
Although u is described as a frame number, it is also possible to perform processing in which u is set as time.

Specifically, the display control according to the reliability index value can be executed as any one or a combination of the five processing modes described above, that is, the following five processing examples.
(Processing Example 1) The “drawn locus” corresponding to the actual trajectory is displayed as a solid line, and the “predicted trajectory” is displayed by changing at least one of the color, the transmittance, and the thickness so as not to stand out from the solid line.
(Processing Example 2) The predicted trajectory is updated every frame. That is, the displayed predicted trajectory is erased in the next drawing frame, and a new predicted trajectory is drawn.
(Processing Example 3) At least one of length, color, transmittance, and thickness is changed according to the reliability index value of the predicted trajectory.
(Processing Example 4) Depending on the situation, drawing of the predicted trajectory is turned on / off, that is, switching between display and non-display is performed.
(Processing Example 5) When the predicted trajectory deviates from the actual trajectory, control is performed to make the predicted trajectory inconspicuous by changing the display mode of the predicted trajectory.

Hereinafter, specific modes and effects of each processing example will be described.
(Processing example 1)
Processing example 1 displays the “drawn locus” corresponding to the actual locus as a solid line, and displays the “predicted locus” by changing at least one of the color, the transmittance, and the thickness so as not to stand out from the solid line. It is.

A specific display control example will be described with reference to FIG.
FIG. 13 shows examples of display modes of the following trajectories.
(1) Trajectory already drawn (corresponding to actual trajectory)
(2) Predicted trajectory The following display control is performed for these two trajectories.
(A) The color to be displayed is controlled, and (1) the drawn trajectory (corresponding to the actual trajectory) is displayed as a black solid line, and (2) the predicted trajectory is displayed as a solid line other than black (for example, red).
(B) The transmittance of the line to be displayed is controlled, and (1) the drawn locus (corresponding to the actual locus) = displayed as a black solid line that does not transmit (transmittance 0%), and (2) the predicted locus = transparent black Displayed as a solid line (for example, transmittance 50%).
(C) Control the thickness of the line to be displayed, (1) Display as a drawn trajectory (corresponding to a real trajectory) = black solid line (thick line), (2) Display as a predicted trajectory = black solid line (thin line) .
For example, display control of any one of (A) to (C) or a combination thereof is performed.

Fig. 14 shows a specific display example. The example shown in FIG. 14 is an example corresponding to the display control example of FIG. In other words, this is an example of controlling the thickness of a line to be displayed. (1) Displayed as a drawn trajectory (corresponding to a real trajectory) = black solid line (thick line), (2) As a predicted trajectory = black solid line (thin line) It is the displayed control example.

In this way, by performing display control in which (1) a drawn trajectory (corresponding to an actual trajectory) and (2) a predicted trajectory are displayed in different modes, the user (render) can draw the corresponding trajectory. The trajectory and the predicted trajectory can be clearly distinguished and recognized, and drawing processing that is not confused by the predicted trajectory is possible.

Further, the display control of this (Processing Example 1) may be developed to set the display mode of the predicted trajectory according to the reliability.

A specific display control example will be described with reference to FIG.
FIG. 15 shows a display example of the following trajectories as in FIG.
(1) Trajectory already drawn (corresponding to actual trajectory)
(2) Predicted trajectory The following display control is performed for these two trajectories.

(A) The color to be displayed is controlled, (1) the drawn locus (corresponding to the actual locus) = displayed as a black solid line, and (2) the predicted locus = displayed as a solid line other than black.
Further, the color of the predicted trajectory is changed according to the reliability.
For example, the prediction trajectory with high reliability is red, and the prediction trajectory with low reliability is yellow, and the color is changed according to the reliability.

(B) The transmittance of the line to be displayed is controlled, and (1) the drawn locus (corresponding to the actual locus) = displayed as a black solid line that does not transmit (transmittance 0%), and (2) the predicted locus = transparent black Display as a solid line.
Further, for the predicted trajectory, the transmittance is changed according to the reliability.
For example, the predicted trajectory with high reliability sets the transmittance low, and the predicted trajectory with low reliability sets the transmittance high.

(C) Control the thickness of the line to be displayed, (1) Display as a drawn trajectory (corresponding to a real trajectory) = black solid line (thick line), (2) Prediction trajectory = A black solid line thinner than the drawn trajectory indicate.
Further, the thickness of the predicted trajectory is changed according to the reliability.
For example, the prediction trajectory with high reliability is a medium thick line, the prediction trajectory with low reliability is a thin line, etc., and the line thickness is changed according to the reliability.
For example, display control of any one of (A) to (C) or a combination thereof is performed.

A specific display example is shown in FIG. The example shown in FIG. 16 is an example corresponding to the display control example of FIG. In other words, this is an example of controlling the thickness of a line to be displayed. (1) Displayed as a drawn locus (corresponding to a real locus) = black solid line (thick line), and (2) a predicted locus = black solid line. This is a control example in which a predicted trajectory with a high degree is displayed as a medium thick line, and a predicted trajectory with a low degree of reliability is displayed as a thin line.

As described above, (1) the drawn trajectory (corresponding to the actual trajectory) and (2) the predicted trajectory are different from each other, and the display control is performed so that the predicted trajectory is displayed differently depending on the reliability. Thus, the user (drawer) can clearly distinguish the drawn trajectory corresponding to the actual trajectory from the predicted trajectory, and further confirm the reliability of the predicted trajectory.

(Processing example 2)
Next, processing example 2 will be described. Processing example 2 updates the predicted trajectory for each frame. That is, in this example, the displayed predicted trajectory is erased in the next drawing frame to draw a new predicted trajectory.

A specific display example will be described with reference to FIG.
FIG. 17 shows a display example of the following two consecutive frames.
(A) Display example of frame n (b) Display example of frame n + 1

In the display example of the frame n shown in (a), the predicted trajectory connecting the predicted

points

1, 111, the predicted

points

2, 112, and the predicted

points

3, 113 first from the latest drawing position 102 of the drawn trajectory 100 corresponding to the actual trajectory. 105 is displayed. It is assumed that the actual locus at this time is an actual locus 121 shown in the figure. This real locus 121 is not displayed.

In the display example of (b) frame n + 1 shown in the figure which is the next frame, the latest drawing position is updated to become the updated latest drawing position 122. This position is a position that substantially coincides with the prediction points 1 and 111 in the frame n.

Furthermore, each prediction point of the prediction trajectory 105 is also updated, and each prediction point is set as update prediction points 1 to 3 and 131 to 133 at positions ahead of the frame n, and the prediction trajectory is connected so as to connect these update prediction points. 105 is displayed.

In this way, in (Processing Example 2), the prediction line is erased every frame and redrawn as a new point. As a result, the prediction line is updated with a length corresponding to the refresh rate of the screen, and a display in which the prediction line is replaced with a highly accurate prediction line for each frame is realized.

(Processing example 3)
Next, processing example 3 will be described. Processing example 3 is a processing example in which at least one of length, color, transmittance, and thickness is changed according to the reliability index value of the predicted trajectory.
This (Processing Example 3) is an extension of (Processing Example 1) and performs display control for changing the display length according to the reliability index value of the predicted trajectory.
The reliability index value is a value corresponding to the standard deviation of the auxiliary prediction trajectory described above with reference to FIGS. 11 and 12, and the smaller the value, the higher the reliability of the prediction trajectory, and the larger the value, the prediction trajectory. The reliability of is low.

When the value of the reliability index value is large, there is a high possibility that it is out of the actual trajectory. In such a case, the predicted trajectory is not displayed.
Specifically, the reliability index value (standard deviation) of each prediction point described above with reference to FIGS. 11 and 12 is compared with a preset threshold value (reliability threshold value), and reliability is determined. If the value index value does not fall below the predetermined threshold value, a response is made not to draw the prediction trajectory immediately before reaching the prediction point.

An example of this display control process will be described with reference to FIG.
In FIG. 18, it is assumed that the reliability index values of the three prediction points constituting the predicted trajectory 105 scheduled to be displayed are calculated as follows.
(A) Reliability SD [1] of prediction point 1,111 = 0.08
(B) Reliability SD [2] of prediction point 2,112 = 3.75
(C) Reliability SD [3] of predicted point 3,113 = 8.52
The reliability index value is the standard deviation σ of the coordinate position of the auxiliary prediction trajectory corresponding to the similar trajectory used to calculate each prediction point, as described above with reference to FIGS. The smaller the value, the higher the reliability.

Here, the threshold value for determining whether or not to display the predicted trajectory is a value that varies according to the speed of the previous trajectory in the drawn trajectory, that is, the moving speed of the previous trajectory 161 shown in FIG.
Specifically, a value obtained by multiplying a predetermined constant α by the moving speed Vt of the immediately preceding locus 161, that is,
α × Vt
Is used as a threshold value.
α is a preset coefficient.

For Vt, for example, the moving distance s between one frame of the immediately preceding locus can be applied. That is, the moving speed is defined as the moving distance between one frame, and the locus moving distance s between frames shown in FIG. 18 is applied.
α × s
May be used as a threshold value.

For example, the above threshold value α × s is compared with the reliability index value (standard deviation) SD [u] of the predicted point on the predicted trajectory. Note that. u means a future frame number after the display frame of the display frame t at the latest drawing position. In FIG. 18, u represents the display frame t of the latest drawing position as t = 0, and the reliability index values of the prediction points corresponding to the future frames 1, 2, and 3 are SD [1], SD [2], SD [ 3].

A comparison judgment formula between the reliability index value of each prediction point and the threshold value α × s, that is,
SD [u] <α × s
Whether or not to display the predicted trajectory after each prediction point is determined using the determination formula.
When the above judgment formula is satisfied, that is,
When the reliability index value SD [u] of a prediction point is a value smaller than the threshold value α × s, it is determined that the reliability of the prediction point is high, and a prediction trajectory up to the prediction point is drawn and displayed. .
On the other hand, when the reliability index value SD [u] of the prediction point is a value not smaller than the threshold value α × s, it is determined that the reliability of the prediction point is low, and the prediction trajectory up to the prediction point is drawn. Cancel the display.

For example, the prediction trajectory drawing determination using the reliability index value of each prediction point shown in FIG. 18 is set as follows.
As an example,
Coefficient α = 1.5
Movement distance s = 4.0 between one frame of the immediately preceding locus
Set as above. When this setting is made, a comparison judgment formula between the reliability index value SD [u] of each prediction point shown in FIG.
SD [u] <α × s
The reliability index value of each prediction point is substituted into the comparison determination formula to determine whether or not the above formula is satisfied.

The prediction point 1 has a reliability index value: SD [1] = 0.08.
That is,
SD [1] = 0.08,
α × s = 1.5 × 4.0 = 6
And
SD [1] = 0.08 <6.0
The above equation holds,
SD [u] <α × s
The above judgment formula is satisfied.

Further, the prediction point 2 has a reliability index value: SD [2] = 3.75.
That is,
SD [2] = 3.75,
α × s = 1.5 × 4.0 = 6
And
SD [2] = 3.75 <6.0
The above equation holds,
SD [u] <α × s
The above judgment formula is satisfied.

Further, the prediction point 3 has a reliability index value: SD [3] = 8.52.
That is,
SD [3] = 8.52
α × s = 1.5 × 4.0 = 6
And
SD [3] = 8.52 <6.0
The above formula does not hold. That is,
SD [u] <α × s
The above judgment formula is not satisfied.

Thus, the reliability index values SD [1] and SD [2] of the prediction point 1 and the prediction point 2 are smaller than the threshold value αs = 1.5 × 4.0 = 6.0,
SD [u] <α × s
It is determined that the determination formula is satisfied and the reliability is high, and drawing and display are executed for the predicted trajectory up to these points.
However, the reliability index value SD [3] of the prediction point 3 is not smaller than the threshold value αs = 1.5 × 4.0 = 6.0,
SD [u] <α × s
It is determined that the determination formula is not satisfied and the reliability is low, and it is determined that drawing and display are not performed for the prediction trajectory immediately before reaching the prediction point.
As shown in FIG. 18, the predicted trajectory from the predicted points 2 112 to the predicted points 3 113 is a non-display predicted trajectory 151.

In the processing of (Processing Example 3), the prediction trajectory with low reliability is not displayed, and only the prediction trajectory with high reliability is displayed. Therefore, the user (render) only displays the prediction trajectory closer to the actual trajectory. Observed processing is possible.

In the above description of (Processing Example 3),
SD [u] <α × s
In the above determination formula, the fixed value such as the coefficient α = 1.5 has been described. When it is determined that the above determination formula is satisfied and the reliability is high, drawing and display are executed for the predicted trajectory up to these points, and the above determination formula is not satisfied and it is determined that the reliability is low. The case has been described as an example of processing for stopping the drawing and display of the predicted trajectory up to these points.
In other words, the above processing example is reliability-dependent display control, but as another processing example, the predicted trajectory becomes thinner (higher transparency) as it goes to the tip of the predicted trajectory regardless of the reliability. It is also possible to adopt a configuration in which processing for displaying, processing for gradually decreasing the color, or control for displaying with thinner lines is possible. Note that this control can also be realized by performing determination processing by changing the coefficient α in the determination formula toward the tip of the predicted trajectory.

(Processing example 4)
Next, processing example 4 will be described. Processing example 4 is a processing example in which drawing of a predicted trajectory is turned on / off, that is, switching between display and non-display is performed according to the situation.

A specific example will be described with reference to FIG.
FIG. 19 shows two detection states as conditions for executing this (processing example 4).
For example, when one of the states (A) and (B) shown in FIG. 19 is detected, the on / off control of the predicted trajectory is executed according to this (processing example 4). That is, this is the case of the following situation detection.
(A) When it is detected that the amount of decrease in the input device pressure value per unit time is equal to or greater than the specified threshold value [Thp]
(B) When it is detected that the input device movement amount per unit time is less than the specified threshold value [Thd],
Even if a state like the above (A) and (B) is detected, processing for turning off the predicted trajectory is performed.

(A) It is predicted that the input device, for example, a dedicated pen is moving away from the display unit. If the predicted trajectory is continuously displayed in such a state, an overshoot in which only the predicted trajectory is drawn may occur even though the user's actual trajectory is not present. To prevent such overshoot. In such a case, the drawing of the predicted trajectory is stopped.

In (B) above, it is expected that the movement of the input device, for example, the dedicated pen, stops. If the predicted trajectory is continuously displayed in such a state, an overshoot in which only the predicted trajectory is drawn may occur despite the absence of the user's actual trajectory, as in (A) above. To prevent such overshoot. In such a case, the drawing of the predicted trajectory is stopped.

Note that the input device is not limited to a dedicated pen, and may be a finger, for example.

A specific change in pressure value corresponding to FIG. 19A will be described with reference to FIG.

The upper graph in FIG. 20 is a graph showing the time transition of the pressure value (P) with respect to the display unit of the input device.
The horizontal axis represents time (t), and the vertical axis represents the pressure value (P) with respect to the display unit of the input device. The time (t) on the horizontal axis may be replaced with the frame number of the display frame.

The transition of the pressure value (P) from time T1 to T5 is as follows.
Time T1: Pressure value = 0.53
Time T2: Pressure value = 0.54
Time T3: Pressure value = 0.42
Time T4: Pressure value = 0.30
Time T5: Pressure value = 0.00
Thus, the pressure value gradually decreases with time. For example, it is estimated that the pen is gradually away from the display unit.

The lower graph in FIG. 20 is a graph created based on the time transition of the pressure value in the upper section, and is a graph showing the time transition of the pressure value difference data, which is the amount of change in the pressure value per unit time. is there.
The horizontal axis represents time (t), and the vertical axis represents the pressure value difference (ΔP), which is the difference value per unit time of the pressure value with respect to the display unit of the input device.
For example, the value of pressure value difference at time T2 = + 0.01 is the difference between the pressure value at time T2 = 0.54 and the pressure value at time T1, which is the previous pressure measurement time = 0.53.
0.54-0.53 = + 0.01
The above difference values are shown.
The difference value at time T3 is the difference between the pressure value at time T3 and the pressure value at time T2, and so on.

The transition of the pressure value difference (ΔP) between times T1 and T5 is as follows.
Time T1: Pressure value difference = 0.0
Time T2: Pressure value difference = + 0.01
Time T3: Pressure value difference = −0.12
Time T4: Pressure value difference = −0.12
Time T5: Pressure value difference = −0.30

Here, the threshold value [THp] of the pressure value difference is set to −0.09.
THp = −0.09
It is.
At each time, control is performed to stop (OFF) the display of the predicted trajectory when the measured pressure value difference becomes a reduction amount larger than the threshold value (−0.09).
In the example shown in FIG. 20, since the pressure value difference at the timing of time T3 = −0.12 indicates a reduction amount larger than the threshold value (−0.09), the predicted trajectory is displayed at this point. Stop.

It is possible to prevent an overshoot that erroneously displays a predicted trajectory after the user's input device leaves the display unit by performing such a predicted trajectory display stop process.

Next, processing according to the movement amount of the input device per unit time shown in FIG. 19B will be described with reference to FIG.

The graph shown in FIG. 21 is a graph showing the time transition of the movement distance (D) per unit time in the display unit of the input device.
The horizontal axis represents time (t), and the vertical axis represents the movement distance (D) per display unit time of the input device. The time (t) on the horizontal axis may be replaced with the frame number of the display frame.
The movement distance (D) per unit time is, for example, a movement distance between one display frame.
For example, the moving distance 15 shown at time T1 corresponds to the distance moved by the input device during the frame interval from the time (T0) that is the previous frame display timing to the time (T1) that is the frame display time of the month. To do.

The transition of the unit time travel distance (D) from time T1 to T5 is as follows.
Time T1: Unit time travel distance = 15
Time T2: Unit time travel distance = 10
Time T3: Unit time travel distance = 3
Time T4: Unit time travel distance = 2
Time T5: Unit time travel distance = 1
Thus, the unit time moving distance gradually decreases with time. For example, it is estimated that the pen is gradually stopped on the display unit.

Here, the threshold value [THd] of the unit time moving distance is set to 4.
THd = 4
It is.
At each time, when the measured unit time moving distance is equal to or less than the threshold value (4), control is performed to stop (OFF) displaying the predicted trajectory.
In the example shown in FIG. 21, since the unit time moving distance = 3 at the timing of the time T3 indicates a value equal to or less than the threshold value (4), the display of the predicted trajectory is stopped at this point.

It is possible to prevent an overshoot that erroneously displays the predicted trajectory after the user input device has stopped on the display unit by performing such a predicted trajectory display stop process.

(Processing example 5)
Next, processing example 5 will be described. Process example 5 is a process example in which when the predicted trajectory deviates from the actual trajectory, control is performed to change the display mode of the predicted trajectory to make the predicted trajectory inconspicuous.

A specific example of the processing example 5 will be described with reference to FIG.
The display example (a) shown in FIG. 22 is a normal display example. A drawn trajectory 100 and a predicted trajectory 105 are displayed. The predicted trajectory 105 is displayed as a line connecting the predicted points 1 to 3. Predicted points 1 to 3 are average positions of constituent coordinates of a plurality of auxiliary predicted trajectories calculated from a plurality of similar trajectories as described above.
In addition, the real locus | trajectory 121 shown to a figure is not displayed on a display part.

FIG. 22B is a display example when the present processing example 5 is applied. In FIG. 22B, the display area of the predicted trajectory 105 is set as the display mode change area 171 and the predicted trajectory 105 is changed to a display mode that is not conspicuous.
This is executed when the reliability of the predicted trajectory is low.
Specifically, for example, the process using the reliability index value described above is performed.

The reliability index value SD [u] is calculated for each of the prediction points 1,111 to 3,113 set as the constituent points of the prediction trajectory 105. This reliability index value corresponds to the standard deviation (σ) of the constituent coordinates of a plurality of auxiliary prediction trajectories (see FIG. 11) applied to the calculation of the prediction points.

When it is determined that the reliability index value is equal to or greater than the predetermined value and the reliability is low, the display area of the predicted trajectory 105 is set as the display mode change area 171 as shown in FIG. The display mode is changed so that the predicted trajectory 105 is not conspicuous.

By performing such display control, an erroneous predicted trajectory away from the actual trajectory can be presented inconspicuously to the user (drawer).

[8. Processing sequence for predictive trajectory display control]
Next, the processing sequence of the display control of the predicted trajectory will be described with reference to the flowcharts in FIG.

The flowchart shown in FIG. 23 is a basic sequence of display control processing, and is a flowchart illustrating a sequence for executing display control according to the above-described (Processing Example 1) to (Processing Example 3).
The flowchart shown in FIG. 24 is a flowchart for explaining a display control sequence for executing the drawing control of the predicted trajectory according to the pressure value of the input device in addition to the above (processing example 1) to (processing example 3).
Note that the processes shown in these flowcharts are executed under the control of a data processing unit of the information processing apparatus according to the present disclosure, specifically, a data processing unit including a CPU having a program execution function, for example. The program is stored, for example, in the memory of the information processing apparatus.

First, a sequence for executing display control according to the above (Processing Example 1) to (Processing Example 3) will be described with reference to the flowchart shown in FIG.

(Step S301)
First, in step S301, an input event based on an input device is detected. Specifically, for example, a touch position detection process for the input display unit of the dedicated pen is performed.
This process is performed as a process of detecting time (or frame) = t and pen contact position coordinate (x, y) information corresponding to the time.

(Step S302)
The information processing apparatus draws a trajectory for the display unit by applying the event information input in step S301, but as described above, an area where the drawing process corresponding to the actual trajectory of the pen is not in time, that is, The drawing delay area 23 described with reference to FIG. 1 is generated.

In order to draw a predicted trajectory in the drawing delay area, a learning process using trajectory information of a past drawn trajectory is performed.
That is, a learning process for estimating a predicted trajectory is performed according to the process described with reference to FIGS. Specifically, as shown in FIG. 6, processing for detecting a similar locus similar to the immediately preceding locus 82 including the latest drawing position 81 of the drawn locus 80 from the drawn locus 80 is performed.
In step S302, learning processing such as detection of similar trajectories is executed.

(Step S303)
In step S303, a prediction trajectory estimation process using the similar region detected in step S302 is executed.
In addition, as a procedure of the estimation process of the predicted trajectory, k coordinates corresponding to the future frame u estimated according to a plurality (k) of similar trajectories are calculated, and an average position of these k coordinates is determined as a frame. The process of setting the coordinate of the predicted trajectory at u is performed.
When the predicted future frame is up to n points ahead, the coordinates constituting the predicted trajectory and the reliability index value SD of each predicted coordinate are calculated individually. That is, (x ₀ , y _0, sd ₀ ) to (x _n , y _n , sd _n ) are calculated. Here, the variable corresponding to the predicted future frame number is set to i, and n + 1 coordinate positions of i = 0 to n and the reliability index value are calculated.

(Step S304)
Steps S304 to S308 are loop processing that repeatedly executes processing according to the reliability index value of each coordinate position for each coordinate position of i = 0 to n corresponding to the constituent coordinates of the predicted trajectory.
The reliability index value SD is a value corresponding to the standard deviation σ of a plurality of estimated coordinates of each future frame calculated based on a plurality of similar trajectories, as described above with reference to FIG. is there.

(Step S305)
In step S305, the reliability index value SD [i] of the constituent coordinates (x _i , y _i ) of the predicted trajectory is compared with the threshold value (α × s).
As described above with reference to FIG. 18 and the like, α is a preset coefficient, and s is a trajectory moving distance s between one frame in the immediately preceding trajectory 161 as shown in FIG.

In step S305,
SD [i] <α × s
It is determined whether or not the determination formula is satisfied.
When the determination formula is satisfied, the constituent coordinates (x _i , y _i ) of the predicted trajectory have small standard deviations σ of a plurality of predicted coordinate positions calculated according to the plurality of similar trajectories from which the calculation is made. It means that the variation is small. In this case, it is determined that the reliability of the constituent coordinates (x _i , y _i ) of the predicted trajectory is high, and the process proceeds to step S306.
On the other hand, when the determination formula is not satisfied, the constituent coordinates (x _i , y _i ) of the predicted trajectory have large standard deviations σ of a plurality of predicted coordinate positions calculated according to the plurality of similar trajectories from which the calculation is made. That is, the variation is large. In this case, it is determined that the reliability of the constituent coordinates (x _i , y _i ) of the predicted trajectory is low, and the process proceeds to step S307.

(Step S306)
If it is determined in step S305 that the reliability of the constituent coordinates (x _i , y _i ) of the predicted trajectory is high, a predicted trajectory drawing process is executed in step S306. Here, the predicted trajectory is drawn in a manner different from the eyeless trajectory to be drawn corresponding to the actual trajectory.
That is, as described with reference to FIGS. 13 and 14, a predicted trajectory having at least one of color, transmittance, and thickness set differently from the drawn actual trajectory is drawn and displayed.
Further, as described with reference to FIGS. 15 and 16, it may be configured to display by changing at least one of color, transmittance, and thickness according to the reliability.
The predicted trajectory is updated and displayed sequentially every time a new event occurs.

(Step S307)
On the other hand, if it is determined in step S305 that the reliability of the constituent coordinates (x _i , y _i ) of the predicted trajectory is low, the predicted trajectory drawing process is stopped in step S307.
In this case, the predicted trajectory is not drawn.

(Step S308)
Steps S304 to S308 are loop processing that repeatedly executes processing according to the reliability index value of each coordinate position for each coordinate position of i = 0 to n corresponding to the constituent coordinates of the predicted trajectory.
When all the processes for the coordinate positions i = 0 to n are completed, the process proceeds to step S3409.

(Step S309)
In step S309, it is determined whether or not there is a next event input.
If the next event input is not detected, the process ends.
If the next event input is detected, the process proceeds to step S310.

(Step S310)
In step S310, coordinate information corresponding to a new input event is acquired.
Similar to step S301, this process is performed as a process of detecting time (or frame) = t and pen contact position coordinate (x, y) information corresponding to the time.
After that, based on the coordinate information corresponding to the new input event, the processing from step S302 is repeated.

Next, with reference to the flowchart shown in FIG. 24, in addition to the above (Processing Example 1) to (Processing Example 3), a display control sequence for executing drawing control of a predicted locus according to the pressure value of the input device and the like. explain.

(Step S401)
First, in step S401, an input event based on an input device is detected. Specifically, for example, a touch position detection process for the input display unit of the dedicated pen is performed.
This process is performed as a process for detecting time (or frame) = t, the touch position coordinate (x, y) information of the pen corresponding to the time, and a process for detecting the pressure value p.

(Step S402)
The information processing apparatus draws a trajectory for the display unit by applying the event information input in step S401, but as described above, an area where the drawing process corresponding to the actual trajectory of the pen is not in time, that is, The drawing delay area 23 described with reference to FIG. 1 is generated.

In order to draw a predicted trajectory in the drawing delay area, a learning process using trajectory information of a past drawn trajectory is performed.
That is, a learning process for estimating a predicted trajectory is performed according to the process described with reference to FIGS. Specifically, as shown in FIG. 6, processing for detecting a similar locus similar to the immediately preceding locus 82 including the latest drawing position 81 of the drawn locus 80 from the drawn locus 80 is performed.
In step S402, a learning process such as similar locus detection is executed.

(Step S403)
In step S403, a prediction trajectory estimation process using the similar region detected in step S402 is executed.
In addition, as a procedure of the estimation process of the predicted trajectory, k coordinates corresponding to the future frame u estimated according to a plurality (k) of similar trajectories are calculated, and an average position of these k coordinates is determined as a frame. The process of setting the coordinate of the predicted trajectory at u is performed.
When the predicted future frame is up to n points ahead, the coordinates constituting the predicted trajectory and the reliability index value SD of each predicted coordinate are calculated individually. That is, (x ₀ , y _0, sd ₀ ) to (x _n , y _n , sd _n ) are calculated. Here, the variable corresponding to the predicted future frame number is set to i, and n + 1 coordinate positions of i = 0 to n and the reliability index value are calculated.

(Step S404)
Steps S404 to S408 are loop processing for repeatedly executing processing according to the reliability index value of each coordinate position for each coordinate position of i = 0 to n corresponding to the constituent coordinates of the predicted trajectory.
The reliability index value SD is a value corresponding to the standard deviation σ of a plurality of estimated coordinates of each future frame calculated based on a plurality of similar trajectories, as described above with reference to FIG. is there.

(Step S405)
In step S405, the value of the reliability index value SD [i] of the constituent coordinates (x _i , y _i ) of the predicted trajectory is compared with the threshold value (α × s).
As described above with reference to FIG. 18 and the like, α is a preset coefficient, and s is a trajectory moving distance s between one frame in the immediately preceding trajectory 161 as shown in FIG.

In step S405,
SD [i] <α × s
It is determined whether or not the determination formula is satisfied.
When the determination formula is satisfied, the constituent coordinates (x _i , y _i ) of the predicted trajectory have small standard deviations σ of a plurality of predicted coordinate positions calculated according to the plurality of similar trajectories from which the calculation is made. It means that the variation is small. In this case, it is determined that the reliability of the constituent coordinates (x _i , y _i ) of the predicted trajectory is high, and the process proceeds to step S406.
On the other hand, when the determination formula is not satisfied, the constituent coordinates (x _i , y _i ) of the predicted trajectory have large standard deviations σ of a plurality of predicted coordinate positions calculated according to the plurality of similar trajectories from which the calculation is made. That is, the variation is large. In this case, it is determined that the reliability of the constituent coordinates (x _i , y _i ) of the predicted trajectory is low, and the process proceeds to step S407.

(Step S406)
If it is determined in step S405 that the reliability of the constituent coordinates (x _i , y _i ) of the predicted trajectory is high, a predicted trajectory drawing process is executed in step S406. Here, the predicted trajectory is drawn in a manner different from the eyeless trajectory to be drawn corresponding to the actual trajectory.
That is, as described with reference to FIGS. 13 and 14, a predicted trajectory having at least one of color, transmittance, and thickness set differently from the drawn actual trajectory is drawn and displayed.
Further, as described with reference to FIGS. 15 and 16, it may be configured to display by changing at least one of color, transmittance, and thickness according to the reliability.
The predicted trajectory is updated and displayed sequentially every time a new event occurs.

(Step S407)
On the other hand, if it is determined in step S405 that the reliability of the constituent coordinates (x _i , y _i ) of the predicted trajectory is low, the drawing process of the predicted trajectory is stopped in step S407.
In this case, the predicted trajectory is not drawn.
Alternatively, as described above with reference to FIG. 23, display control such as blurring processing is executed so that the predicted trajectory is not noticeable.

(Step S408)
Steps S404 to S408 are loop processing for repeatedly executing processing according to the reliability index value of each coordinate position for each coordinate position of i = 0 to n corresponding to the constituent coordinates of the predicted trajectory.
When all the processes for the coordinate positions i = 0 to n are completed, the process proceeds to step S3409.

(Step S409)
In step S409, it is determined whether a condition for stopping the drawing process of the predicted trajectory has been detected.
This is a process for determining whether or not the detection states (A) and (B) described above with reference to FIG. 19 have been detected.
That is, the following state is detected.
(A) Detection that the amount of decrease in the input device pressure value per unit time is equal to or greater than a specified threshold value [Thp].
(B) Detection that the input device movement amount per unit time is less than the specified threshold value [Thd].

In step S409, for example, it is determined whether any of the above (A) and (B) is detected.
When it determines with having detected, it progresses to S410 in step S409.
If not detected, the process proceeds to step S411.

(Step S410)
If it is determined in step S409 that the situation (A) or (B) has been detected, the drawing of the predicted trajectory is stopped in step S410.
This process corresponds to the process described above with reference to FIGS.
After this process, the process proceeds to step S411.

(Step S411)
In step S411, it is determined whether or not there is a next event input.
If the next event input is not detected, the process ends.
If the next event input is detected, the process proceeds to step S412.

(Step S412)
In step S412, coordinate information corresponding to a new input event is acquired.
Similar to step S401, this processing is performed as detection processing of time (or frame) = t, pen contact position coordinate (x, y) information corresponding to the time, and pressure value p.
Then, based on the coordinate information corresponding to the new input event, the processing from step S402 is repeated.

[9. An example of the effect of displaying a highly accurate predicted trajectory]
By applying the processing of the present disclosure described above, it is possible to display the locus of an input device such as a dedicated pen with high accuracy.
An example of the effect of executing such processing will be described with reference to FIG.

For example, as shown in FIG. 25A, consider a case where a process of drawing some characters is performed according to steps S1 and S2.
In step S1, the horizontal line L1 is written.
Thereafter, in step S2, a vertical line L2 is written.
It is assumed that the line L2 needs to be written as a line that substantially passes through the center position of the line L1.

However, for example, as shown in FIG. 25B, when the trace drawing is delayed, the user writes the horizontal line L1 as shown in FIG. The line segment at the end will be hidden.

In this state, even if the user tries to start writing the vertical line L2, the center position of the horizontal line L1 cannot be accurately grasped. Therefore, as shown in (P2) of FIG. 25B, there arises a problem that the write start position of L2 cannot be determined.

Execute the drawing of the predicted trajectory with high accuracy by applying the processing of the present disclosure to solve such a problem, and the user can perform the processing smoothly.

[10. Example of hardware configuration of information processing apparatus of present disclosure]
Next, a hardware configuration example of the information processing apparatus according to the present disclosure will be described with reference to FIG.

As illustrated in FIG. 26, the information processing apparatus includes an input unit 301, an output unit (display unit) 302, a sensor 303, a control unit (CPU or the like) 304, a memory (RAM) 305, and a memory (nonvolatile memory) 306. .

The input unit 301 is configured as an input unit that also serves as a display unit having a touch panel function, for example. However, the touch panel function is not indispensable, and any touch panel function may be used as long as it has a configuration for inputting motion detection information of other input devices.

In addition, as an input device, it is also possible to use a dedicated pen and a user's finger as an input device. In addition, it is good also as a structure which inputs the motion information of devices, such as a gyro pointer. Further, the input unit 301 may be used as a camera to detect user movement (such as a gesture) and use it as input information.
In addition, a mouse provided in a general PC may be set as an input device.
Note that the input unit 301 is not limited to the function of inputting the movement information of the input device, and includes an input unit that performs various settings such as brightness adjustment and mode setting of the display unit 301.

The output unit 302 includes a display unit that displays trajectory information corresponding to movement information of the input device detected by the input unit 301.
For example, a touch panel display.

The sensor 303 is a sensor that inputs information to be applied to the processing of the present disclosure, such as a sensor that detects the pressure on the touch pen or a sensor that detects the pressing area of the finger.

The control unit 304 includes, for example, a CPU configured by an electronic circuit, and functions as a data processing unit that executes processing according to the flowchart described in the above-described embodiment, for example.

The memory 305 is, for example, a RAM, and serves as a work area for executing processing according to the flowchart described in the embodiment, as well as a storage area for input device location information used by the user, various parameters applied to data processing, and the like. Used.

The memory 306 is a non-volatile memory, and is used as a storage area for storing, for example, a program for executing processing according to the flowchart described in the above-described embodiment, and a user's drawing trajectory.

In the above-described embodiment, a processing example in which the k-nearest neighbors (kNN: kNearest Neighbors) method is applied as the learning process for calculating the predicted trajectory has been described. In addition to kNN, other methods may be applied. For example, the following method can be applied.
Linear regression method,
Support Vector Regression (SVR) method,
Relevant Vector Regression (RVR) method,
Hidden Markov Model (HMM) method,
Further, for example, a prediction technique described in Japanese Patent Application Laid-Open No. 2011-118786, which is a previous patent application of the present applicant, may be applied.

In addition, the information processing apparatus according to the present disclosure may be configured to be integrated with a display device that performs trajectory display, but may be configured to be able to communicate with the display device. For example, it is good also as a server which can communicate data via a network. In this case, the input information of the input device operated by the user is transmitted to the server, and the server calculates the predicted trajectory based on the learning process described above. Further, the server executes a process of transmitting the calculation result to a display device on the user side, for example, a tablet terminal, and displaying a predicted locus on the display unit of the tablet terminal.

[11. Summary of composition of the present disclosure]
As described above, the embodiments of the present disclosure have been described in detail with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present disclosure, the claims should be taken into consideration.

The technology disclosed in this specification can take the following configurations.
(1) having a data processing unit that performs display control processing of a locus according to input position information generated by user operation input;
The data processing unit
An actual trajectory identified based on the input position information;
A trajectory of an area where the identification of the real trajectory is not completed, and a predicted trajectory identified by a predetermined prediction process;
An information processing apparatus that executes display control for displaying the image in a different manner.

(2) The information processing apparatus according to (1), wherein the predicted trajectory is a trajectory predicted based on an actual trajectory specified in the past.

(3) The information processing according to (1) or (2), wherein the data processing unit displays at least one of a color, a transparency, and a thickness of the predicted trajectory in a mode different from the actual trajectory. apparatus.

(4) The data processing unit displays at least one of the color, the transparency, or the thickness of the predicted locus different from the actual locus, and displays the predicted locus in a less conspicuous manner than the actual locus. The information processing apparatus according to any one of 1) to (3).

(5) The information processing apparatus according to any one of (1) to (4), wherein the data processing unit determines a display mode of the predicted trajectory according to a reliability of the predicted trajectory.

(6) The information according to (5), wherein the data processing unit determines and displays at least one of a color, a transparency, or a thickness of the predicted trajectory according to the reliability of the predicted trajectory. Processing equipment.

(7) The data processing unit changes at least one of the color, transparency, or thickness of the predicted trajectory according to the reliability of the predicted trajectory, and the predicted trajectory is reduced as the reliability decreases. The information processing apparatus according to (5), wherein the information processing apparatus is displayed as a less conspicuous aspect.

(8) The information processing apparatus according to (5), wherein the data processing unit controls the display unit so that the predicted trajectory is not displayed when the reliability of the predicted trajectory is determined to be lower than a predetermined threshold.

(9) The information processing apparatus according to (5), wherein when the reliability of the predicted trajectory is determined to be lower than a predetermined threshold, the data processing unit changes the display of the predicted trajectory to an inconspicuous display mode. .

(10) The input position information is input position information obtained by detecting a contact position of the input object with respect to the touch panel, and the data processing unit acquires a pressure value representing a pressure at which the input object contacts the touch panel, and the pressure The information processing apparatus according to any one of (1) to (9), wherein a display mode of the predicted trajectory is determined according to a value.

(11) When the data processing unit detects that the decrease in the pressure value is larger than a specified threshold, the data processing unit controls the display unit so that the predicted locus is not displayed. The information processing apparatus described.

(12) The data processing unit acquires a movement amount per unit time of the input position information, and determines a display mode of the predicted trajectory according to the movement amount according to any one of (1) to (11). The information processing apparatus described.

(13) The information processing apparatus according to (12), wherein the data processing unit stops displaying the predicted trajectory when detecting that the movement amount is less than a predetermined threshold.

The information processing apparatus according to (10), wherein the input object is a stylus.

(15) The data processing unit detects, as the predicted trajectory calculation process, a plurality of similar trajectories similar to the trajectory immediately before the calculation area of the predicted trajectory from the past trajectory, and performs each similarity based on the detected plurality of similar trajectories. The information processing apparatus according to any one of (1) to (14), wherein a subsequent trajectory of the trajectory is estimated, and a predicted trajectory is calculated by an averaging process or a weighted addition process of the estimated plurality of subsequent trajectories.

(16) An information processing method executed in the information processing apparatus,
The data processing unit performs a trajectory display control process according to the input position information generated by the user's operation input,
The data processing unit, in the display control process,
An actual trajectory identified based on the input position information;
A trajectory of an area where the identification of the real trajectory is not completed, and a predicted trajectory identified by a predetermined prediction process;
An information processing method for executing display control for displaying in a different manner.

(17) A program for executing information processing in an information processing device,
Causing the data processing unit to perform display control processing of the locus according to the input position information generated by the user's operation input;
In the display control process,
An actual trajectory identified based on the input position information;
A trajectory of an area where the identification of the real trajectory is not completed, and a predicted trajectory identified by a predetermined prediction process;
A program for executing display control for displaying the image in a different manner.

Further, the series of processes described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run. For example, the program can be recorded in advance on a recording medium. In addition to being installed on a computer from a recording medium, the program can be received via a network such as a LAN (Local Area Network) or the Internet and installed on a recording medium such as a built-in hard disk.

In addition, the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

As described above, according to the configuration of an embodiment of the present disclosure, display control of a predicted trajectory predicted according to past trajectory information is realized.
Specifically, it has a data processing unit that performs a display control process of the trajectory according to the input position information, and the data processing unit displays the trajectory after the trajectory calculation process according to the input position information is completed as an actual trajectory, A trajectory in an area where the trajectory calculation process according to the input position information is not completed is estimated as a predicted trajectory, and the estimated predicted trajectory is displayed in a manner different from the displayed actual trajectory. For example, at least one of the color, transparency, and thickness of the predicted trajectory is displayed in a manner different from the actual trajectory, and the predicted trajectory is displayed in a manner that is less conspicuous than the actual trajectory. Further, a process for hiding the predicted trajectory is performed according to the reliability.
With this configuration, display control of a predicted trajectory predicted according to past trajectory information is realized.

DESCRIPTION OF SYMBOLS 10 Input display apparatus 11 Input device 15 Pen tip position 21 Drawn locus | trajectory 22 Latest drawing position 23 Drawing delay area 31 Drawn locus 32 Latest drawing position 33 Approximation application locus 34 Pen locus 35 Predicted locus 41 Drawn locus 42 Latest drawing position 43 Approximate application locus 51 Drawn locus 52 Latest drawing position 53 Predicted locus 54 Pen position 71 Drawn entry 72 Latest drawing position 73 Predicted locus 80 Drawn locus 81 Latest drawing position 82 Immediate locus 83 Similar locus 84 Subsequent locus of similar locus 85 Auxiliary predicted trajectory 86 Final determined predicted trajectory 90 Drawn trajectory 91 Previous trajectory 92 Auxiliary predicted trajectory 93 Final determined predicted trajectory 100 Drawn trajectory 101 Immediate trajectory 102 Latest drawing position 103 Auxiliary predicted trajectory 104 Standard deviation of auxiliary predicted trajectory 105 Prediction Measurement locus 111 to 113 Prediction point 121 Actual locus 151 Non-display prediction locus 171 Display mode change area 301 Input unit 302 Output unit 303 Sensor 304 Control unit 305 Memory (RAM)
306 memory (non-volatile memory)

Claims

A data processing unit that performs display control processing of a locus according to input position information generated by user operation input;
The data processing unit
An actual trajectory identified based on the input position information;
A trajectory of an area where the identification of the real trajectory is not completed, and a predicted trajectory identified by a predetermined prediction process;
An information processing apparatus that executes display control for displaying the image in a different manner.
The information processing apparatus according to claim 1, wherein the predicted trajectory is a trajectory predicted based on an actual trajectory specified in the past.
The data processing unit
The information processing apparatus according to claim 1, wherein at least one of a color, a transparency, and a thickness of the predicted trajectory is displayed in a mode different from the actual trajectory.
The data processing unit
The information processing apparatus according to claim 1, wherein at least one of a color, transparency, or thickness of the predicted trajectory is different from the actual trajectory, and the predicted trajectory is displayed in a less conspicuous manner than the actual trajectory.
The data processing unit
The information processing apparatus according to claim 1, wherein a display mode of the predicted trajectory is determined according to the reliability of the predicted trajectory.
The data processing unit
The information processing apparatus according to claim 5, wherein at least one of a color, a transparency, and a thickness of the predicted trajectory is determined and displayed according to the reliability of the predicted trajectory.
The data processing unit
The color, transparency, or thickness of the predicted trajectory is changed according to the reliability of the predicted trajectory, and the predicted trajectory is displayed in a less conspicuous manner as the reliability decreases. 5. The information processing apparatus according to 5.
The data processing unit
The information processing apparatus according to claim 5, wherein when the reliability of the predicted trajectory is determined to be lower than a predetermined threshold, the information processing apparatus controls the display unit so that the predicted trajectory is not displayed.
The data processing unit
The information processing apparatus according to claim 5, wherein when it is determined that the reliability of the predicted trajectory is lower than a predetermined threshold, the display of the predicted trajectory is changed to an inconspicuous display mode.
The input position information is input position information obtained by detecting the contact position of the input object with respect to the touch panel,
The data processing unit
Obtaining a pressure value representing the pressure at which the input object touches the touch panel;
The information processing apparatus according to claim 1, wherein a display mode of the predicted trajectory is determined according to the pressure value.
The data processing unit
The information processing apparatus according to claim 10, wherein when the decrease in the pressure value is detected as a decrease amount greater than a specified threshold, the display unit is controlled so that the predicted locus is not displayed.
The data processing unit
Obtain the amount of movement per unit time of the input position information,
The information processing apparatus according to claim 1, wherein a display mode of the predicted trajectory is determined according to the movement amount.
The data processing unit
The information processing apparatus according to claim 12, wherein when it is detected that the movement amount is less than a predetermined threshold value, the display of the predicted trajectory is stopped.
The information processing apparatus according to claim 10, wherein the input object is a stylus.
The data processing unit
As the predicted trajectory calculation process,
A plurality of similar trajectories similar to the previous trajectory of the predicted trajectory calculation area are detected from past trajectories,
Estimating the subsequent trajectory of each similar trajectory based on the detected multiple similar trajectories,
The information processing apparatus according to claim 1, wherein a predicted trajectory is calculated by an averaging process or a weighted addition process of a plurality of estimated subsequent trajectories.
An information processing method executed in an information processing apparatus,
The data processing unit performs a trajectory display control process according to the input position information generated by the user's operation input,
The data processing unit, in the display control process,
An actual trajectory identified based on the input position information;
A trajectory of an area where the identification of the real trajectory is not completed, and a predicted trajectory identified by a predetermined prediction process;
An information processing method for executing display control for displaying in a different manner.
A program for executing information processing in an information processing apparatus;
Causing the data processing unit to perform display control processing of the locus according to the input position information generated by the user's operation input;
In the display control process,
An actual trajectory identified based on the input position information;
A trajectory of an area where the identification of the real trajectory is not completed, and a predicted trajectory identified by a predetermined prediction process;
A program for executing display control for displaying the image in a different manner.