JPWO2016046953A1 - Electronic device, method and program - Google Patents

Abstract

According to the embodiment, the method includes inputting first stroke data corresponding to one or more first strokes input by handwriting in a state where the hand contacts the first contact point on the screen; After leaving from one contact point, a user interface used for an operation for designating an input stroke is displayed at a position separated by a first distance in the stroke direction from a first area where one or more first strokes are displayed. It comprises.

Description

  Embodiments described herein relate generally to a technique for inputting a character string by handwriting.

  In recent years, various electronic devices such as tablet computers, notebook personal computers, smartphones, and PDAs have been developed that can input documents by handwriting.

JP 2005-025566 A

  An object of one embodiment of the present invention is to provide an electronic device, a method, and a program capable of efficiently creating a handwritten document.

  According to the embodiment, the method includes inputting first stroke data corresponding to one or more first strokes input by handwriting in a state where the hand is in contact with the first contact point on the screen; After leaving one contact point, a user interface used for an operation for designating an input stroke is displayed at a position away from the first area where the one or more first strokes are displayed in the stroke direction by a first distance. Comprising.

FIG. 1 is a perspective view illustrating an example of an external appearance of an electronic apparatus according to an embodiment. FIG. 2 is a block diagram illustrating an example of a cooperative operation between the electronic device and another device. FIG. 3 is a diagram illustrating an example of a handwritten document handwritten on the touch screen display. FIG. 4 is a diagram illustrating an example of time-series information that is a set of stroke data. FIG. 5 is a diagram illustrating an example of a system configuration of an electronic device. FIG. 6 is a diagram illustrating an example of a home screen displayed by the electronic device. FIG. 7 is a diagram illustrating an example of a note preview screen displayed by the electronic device. FIG. 8 is a diagram illustrating an example of a setting screen displayed by the electronic device. FIG. 9 is a diagram illustrating an example of a page editing screen displayed by the electronic device. FIG. 10 is a diagram illustrating an example of a search dialog displayed by the electronic device. FIG. 11 is a block diagram illustrating an example of a functional configuration of a handwritten note application program executed by the electronic device. FIG. 12 is a diagram for explaining an example of a user operation area when a handwriting input operation is performed on the screen of the touch screen display using a pen. FIG. 13 is a flowchart illustrating an example of the processing procedure of the input auxiliary UI display processing. FIG. 14 is a diagram for describing an example of the first determination process. FIG. 15 is a diagram for describing a first display example of the input auxiliary UI. FIG. 16 is a diagram for describing a second display example of the input auxiliary UI. FIG. 17 is a diagram for describing an example of the second determination process.

  Hereinafter, embodiments will be described with reference to the drawings.

  FIG. 1 is a perspective view illustrating an example of an external appearance of an electronic apparatus according to an embodiment. This electronic device is, for example, a pen-based portable electronic device that can be handwritten with a pen or a finger. This electronic device can be realized as a tablet computer, a notebook personal computer, a smartphone, a PDA, or the like. FIG. 1 shows an example in which this electronic device is realized as a tablet computer. In the following description, it is assumed that the electronic device according to the present embodiment is realized as a tablet computer. A tablet computer is a portable electronic device also called a tablet or a slate computer.

  A tablet computer 10 shown in FIG. 1 includes a main body 11 and a touch screen display 12. The main body 11 has a thin box-shaped housing, and the touch screen display 12 is attached so as to overlap the upper surface of the main body.

  The touch screen display 12 incorporates a flat panel display and a sensor configured to detect a contact position of a pen or a finger on the screen of the flat panel display. The flat panel display may be, for example, a liquid crystal display (LCD). As the sensor, for example, a capacitive touch panel, an electromagnetic induction digitizer, or the like can be used. Hereinafter, a case where both of the two types of sensors, the touch panel and the digitizer, are incorporated in the touch screen display 12 will be described. For this reason, the touch screen display 12 can detect not only a touch operation on the screen using a finger but also a touch operation on the screen using the pen 100.

  The pen 100 may be, for example, a digitizer pen (electromagnetic induction pen). The user can perform a handwriting input operation on the touch screen display 12 using the pen 100 (pen input mode). In the pen input mode, a trajectory of the movement of the pen 100 on the screen, that is, a stroke handwritten by a handwriting input operation is obtained, and thereby a plurality of strokes input by handwriting are displayed on the screen. The locus of movement of the pen 100 while the pen 100 is in contact with the screen corresponds to one stroke. A plurality of strokes constitute characters, symbols, and the like. A set of a large number of strokes corresponding to handwritten characters, handwritten graphics, handwritten tables and the like constitutes a handwritten document.

  In the present embodiment, this handwritten document is stored in the storage medium as time series information (handwritten document data) indicating the coordinate sequence of the trajectory of each stroke and the order relationship between the strokes, not image data. However, this handwritten document may be generated based on the image data. Although details of the time-series information will be described later, the time-series information indicates the order in which a plurality of strokes are handwritten, and includes a plurality of stroke data respectively corresponding to the plurality of strokes. In other words, the time series information means a set of time series stroke data respectively corresponding to a plurality of strokes. Each stroke data corresponds to a certain stroke, and includes a coordinate data series (time series coordinates) corresponding to each point on the locus of this stroke. The order of arrangement of the stroke data corresponds to the order in which the strokes are handwritten.

  The tablet computer 10 can read existing arbitrary time-series information from the storage medium and display a handwritten document corresponding to the time-series information, that is, a plurality of strokes indicated by the time-series information on the screen. The plurality of strokes indicated by the time series information are also a plurality of strokes input by handwriting.

  Furthermore, the tablet computer 10 according to the present embodiment also has a touch input mode for performing a handwriting input operation with a finger without using the pen 100. When the touch input mode is valid, the user can perform a handwriting input operation on the touch screen display 12 using a finger. In the touch input mode, a trajectory of finger movement on the screen, that is, a stroke handwritten by a handwriting input operation is obtained, and thereby a plurality of strokes input by handwriting are displayed on the screen.

  The tablet computer 10 has an editing function. This editing function is an arbitrary handwritten portion in a displayed handwritten document selected by the range selection tool in response to an editing operation by the user using the “eraser” tool, the range selection tool, and other various tools ( Handwritten characters, handwritten marks, handwritten graphics, handwritten tables, etc.) can be deleted or moved. Also, an arbitrary handwritten part in the handwritten document selected by the range selection tool can be designated as a search key for searching for a handwritten document. In addition, recognition processing such as handwritten character recognition / handwritten figure recognition / handwritten table recognition can be performed on an arbitrary handwritten portion in a handwritten document selected by the range selection tool.

  In the present embodiment, the handwritten document can be managed as one or a plurality of pages. In this case, a group of time-series information that fits on one screen may be recorded as one page by dividing time-series information (handwritten document data) by area units that fit on one screen. Alternatively, the page size may be variable. In this case, since the page size can be expanded to an area larger than the size of one screen, a handwritten document having an area larger than the screen size can be handled as one page. When one entire page cannot be simultaneously displayed on the display, the page may be reduced and displayed, or the display target portion in the page may be moved by vertical and horizontal scrolling.

  FIG. 2 shows an example of the linking operation between the tablet computer 10 and an external device. The tablet computer 10 includes a wireless communication device such as a wireless LAN, and can execute wireless communication with the personal computer 1. Furthermore, the tablet computer 10 can execute communication with the server 2 on the Internet using a wireless communication device. The server 2 may be a server that executes an online storage service and other various cloud computing services.

  The personal computer 1 includes a storage device such as a hard disk drive (HDD). The tablet computer 10 can transmit time series information (handwritten document data) to the personal computer 1 and record it in the HDD of the personal computer 1 (upload). In order to secure secure communication between the tablet computer 10 and the personal computer 1, the personal computer 1 may authenticate the tablet computer 10 at the start of communication. In this case, a dialog prompting the user to input an ID or password may be displayed on the screen of the tablet computer 10, and the ID of the tablet computer 10 is automatically transmitted from the tablet computer 10 to the personal computer 1. May be.

  As a result, even when the storage capacity of the tablet computer 10 is small, the tablet computer 10 can handle a large number or a large amount of time-series information.

  Further, the tablet computer 10 reads (downloads) any one or more time series information recorded in the HDD of the personal computer 1, and displays the stroke indicated by the read time series information on the touch screen display 12 of the tablet computer 10. Can be displayed on the screen. In this case, a list of thumbnails obtained by reducing each page of a plurality of time-series information may be displayed on the screen of the touch screen display 12, or one page selected from these thumbnails may be displayed on the touch screen display 12. You may display in normal size on the screen.

  Furthermore, the destination to which the tablet computer 10 communicates may be the server 2 on the cloud that provides a storage service or the like, as described above, instead of the personal computer 1. The tablet computer 10 can transmit time series information (handwritten document data) to the server 2 via the Internet and record it in the storage device 2A of the server 2 (upload). Furthermore, the tablet computer 10 reads (downloads) arbitrary time-series information recorded in the storage device 2A of the server 2, and displays the trajectory of each stroke indicated by the time-series information on the touch screen display 12 of the tablet computer 10. Can be displayed on the screen.

  As described above, in this embodiment, the storage medium in which the time series information is stored may be any one of the storage device in the tablet computer 10, the storage device in the personal computer 1, and the storage device in the server 2.

  Next, with reference to FIG. 3 and FIG. 4, the relationship between strokes (characters, figures, tables, etc.) handwritten by the user and time-series information will be described. FIG. 3 shows an example of a handwritten document (handwritten character string) handwritten on the touch screen display 12 using the pen 100 or the like.

  In a handwritten document, in many cases, another character or figure is input by handwriting on the character or figure once input by handwriting. In FIG. 3, the handwritten characters “A”, “B”, and “C” are input by handwriting in this order, and then a handwritten arrow is input by handwriting near the handwritten character “A”. Yes.

  The handwritten character “A” is expressed by two strokes (“∧” -shaped trajectory, “−”-shaped trajectory) handwritten using the pen 100 or the like, that is, by two trajectories. The trajectory of the first “∧” -shaped pen 100 handwritten is sampled in real time, for example, at equal time intervals, thereby obtaining the time-series coordinates SD11, SD12,. Similarly, the trajectory of the “−” shaped pen 100 to be handwritten next is also sampled in real time at equal time intervals, thereby obtaining the time series coordinates SD21, SD22,..., SD2n of the “−” shaped stroke.

  The handwritten character “B” is represented by two strokes handwritten using the pen 100 or the like, that is, two trajectories. The handwritten character “C” is represented by one stroke handwritten using the pen 100 or the like, that is, one locus. The handwritten “arrow” is expressed by two strokes handwritten using the pen 100 or the like, that is, two trajectories.

  FIG. 4 shows time-series information 200 corresponding to the handwritten document of FIG. The time series information includes a plurality of stroke data SD1, SD2,. In the time series information 200, these stroke data SD1, SD2,..., SD7 are arranged in time series in the order in which these strokes are handwritten.

  In the time series information 200, the first two stroke data SD1 and SD2 indicate two strokes of the handwritten character “A”, respectively. The third and fourth stroke data SD3 and SD4 indicate two strokes constituting the handwritten character “B”, respectively. The fifth stroke data SD5 indicates one stroke constituting the handwritten character “C”. The sixth and seventh stroke data SD6 and SD7 indicate two strokes constituting the handwritten “arrow”, respectively.

  Each stroke data includes a coordinate data series (time series coordinates) corresponding to one stroke, that is, a plurality of coordinates corresponding to each of a plurality of sampling points on one stroke locus. In each stroke data, the coordinates of a plurality of sampling points are arranged in chronological order in the order in which the strokes are written (sampled order). For example, for the handwritten character “A”, the stroke data SD1 includes coordinate data series (time series coordinates) corresponding to each point on the locus of the stroke of the “∧” shape of the handwritten character “A”, that is, n pieces of data. Coordinate data SD11, SD12,..., SD1n are included. The stroke data SD2 includes coordinate data series corresponding to each point on the trajectory of the stroke of the “−” shape of the handwritten character “A”, that is, n coordinate data SD21, SD22,. Note that the number of coordinate data may be different for each stroke data. If the strokes are sampled at equal time intervals, the lengths of the strokes are different, so the number of sampling points is also different.

  Each coordinate data indicates the X coordinate and Y coordinate of a certain point in the corresponding locus. For example, the coordinate data SD11 indicates the X coordinate (X11) and the Y coordinate (Y11) of the start point of the “∧” -shaped stroke. SD1n indicates the X coordinate (X1n) and Y coordinate (Y1n) of the end point of the “∧” -shaped stroke.

  Each coordinate data may include time stamp information T corresponding to the time point (sampling timing) when the point corresponding to the coordinates is handwritten. The handwritten time may be either absolute time (for example, year / month / day / hour / minute / second) or relative time based on a certain time. For example, the absolute time when the stroke is started to be written is added as time stamp information to each stroke data, and the relative time indicating the difference from the absolute time is added as time stamp information T to each coordinate data in the stroke data. It may be added.

  As described above, by using the time series information in which the time stamp information T is added to each coordinate data, the temporal relationship between the strokes can be expressed more accurately. Although not shown in FIG. 4, information (Z) indicating writing pressure may be added to each coordinate data.

  The time series information 200 having the structure as described in FIG. 4 can represent not only the handwriting of each stroke but also the temporal relationship between the strokes. Therefore, by using this time-series information 200, as shown in FIG. 3, the tip of the handwritten “arrow” is written over the handwritten character “A” or close to the handwritten character “A”. However, the handwritten character “A” and the tip of the handwritten “arrow” can be handled as different characters or figures.

  Further, in the present embodiment, as described above, handwritten document data is not stored as an image or character recognition result, but is stored as time-series information 200 composed of a set of time-series stroke data. It can handle handwritten characters without depending on. Therefore, the structure of the time-series information 200 in this embodiment can be commonly used in various countries around the world with different languages.

  FIG. 5 is a diagram showing a system configuration of the tablet computer 10. As shown in FIG. 5, the tablet computer 10 includes a CPU 101, a nonvolatile memory 102, a main memory 103, a BIOS-ROM 104, a system controller 105, a graphics controller 106, a wireless communication device 107, an EC, and the like. In the tablet computer 10, the touch screen display 12 shown in FIG. 1 includes an LCD 12A, a touch panel 12B, and a digitizer 12C.

  The CPU 101 is a processor that controls the operation of various modules in the tablet computer 10. The CPU 101 executes various software loaded into the main memory 103 from the nonvolatile memory 102 which is a straw register device. The software includes an operating system (OS) 103a and various application programs. The various application programs include a handwritten note application program 103b. Hereinafter, the handwritten document data is also referred to as a handwritten note.

  This handwritten note application program 103b has a function for creating and displaying the above-mentioned handwritten document data, a function for editing handwritten document data, handwritten document data including a desired handwritten part, and a desired handwritten part in certain handwritten document data. It has a handwritten document search function for searching.

  Further, it is assumed that the handwritten note application program 103b has a function of displaying a user interface used for an operation for designating a stroke to be input on the screen of the tablet computer 10, for example. The user interface is used for, for example, an operation for specifying a font (color), color, style (thickness), size, and the like of a stroke (handwritten character string) and an operation for specifying a candidate for handwriting input to be described later. User interface etc. are included. Furthermore, this user interface is, for example, a user interface used for an operation for designating confirmation of input of a handwritten stroke (handwritten character string), which will be described later, and an operation for designating cancellation of input of the stroke. May be. That is, the user interface displayed in the present embodiment is a user interface (hereinafter referred to as input assist UI) used for an operation for using a function for assisting handwriting input operation by the user (hereinafter referred to as input assist function). )including. In the following description, the user interface displayed in the present embodiment will be described as an input auxiliary UI, but the user interface may be displayed as long as the user performs a handwriting input operation.

  The CPU 101 also executes a basic input / output system (BIOS) stored in the BIOS-ROM 104. The BIOS is a program for hardware control.

  The system controller 105 is a device for connecting between the local bus of the CPU 101 and various component modules. The system controller 105 also includes a memory controller that controls access to the main memory 103. The system controller 105 also has a function of executing communication with the graphics controller 106 via a PCI Express standard serial bus or the like.

  The graphics controller 106 is a display controller that controls the LCD 12 </ b> A used as a display monitor of the tablet computer 10. A display signal generated by the graphics controller 106 is sent to the LCD 12A. The LCD 12A displays a screen image based on the display signal. The LCD 12A, the touch panel 12B, and the digitizer 12C are overlaid on each other. The touch panel 12B is a capacitance-type pointing device for inputting on the screen of the LCD 12A. The touch position on the screen where the finger is touched and the movement of the touch position are detected by the touch panel 12B. In addition, according to the touch panel 12B, for example, contact of a user's hand on the screen when a handwriting input operation is performed on the screen can be detected. The digitizer 12C is an electromagnetic induction type pointing device for inputting on the screen of the LCD 12A. The contact position on the screen where the pen (digitizer pen) 100 is touched, the movement of the contact position, and the like are detected by the digitizer 12C.

  The wireless communication device 107 is a device configured to execute wireless communication such as wireless LAN or 3G mobile communication.

  The EC 108 is a one-chip microcomputer including an embedded controller for power management. The EC 108 has a function of turning on or off the tablet computer 10 in accordance with the operation of the power button by the user.

  Next, examples of some typical screens presented to the user by the above-described handwritten note application program 103b will be described.

  FIG. 6 shows an example of the home screen of the handwritten note application program 103b. The home screen is a basic screen for handling a plurality of handwritten document data, and can manage notes and set the entire application.

  The home screen includes a desktop screen area 70 and a drawer screen area 71. The desktop screen area 70 is a temporary area for displaying a plurality of note icons 801 to 805 corresponding to a plurality of handwritten notes being worked. Each of the note icons 801 to 805 displays a thumbnail of a page in the corresponding handwritten note. The desktop screen area 70 further displays a pen icon 771, a calendar icon 772, a scrap note (gallery) icon 773, and a tag (label) icon 774.

  The pen icon 771 is a graphical user interface (GUI) for switching the display screen from the home screen to the page editing screen. The calendar icon 772 is an icon indicating the current date. The scrap note icon 773 is a GUI for browsing data (scrap data or gallery data) imported from another application program or from an external file. The tag icon 774 is a GUI for attaching a label (tag) to an arbitrary page in an arbitrary handwritten note.

  The drawer screen area 71 is a display area for browsing a storage area for storing all created handwritten notes. The drawer screen area 71 displays note icons 80A, 80B and 80C corresponding to some handwritten notes in all handwritten notes. Each of the note icons 80A, 80B, and 80C displays a thumbnail of a page in the corresponding handwritten note. The handwritten note application program 103b can detect a certain gesture (for example, a swipe gesture) on the drawer screen area 71 performed by the user using the pen 100 or a finger. In response to detection of this gesture (for example, a swipe gesture), the handwritten note application program 103b scrolls the screen image on the drawer screen area 71 leftward or rightward. Thereby, the note icon corresponding to each arbitrary handwritten note can be displayed in the drawer screen area 71.

  The handwritten note application program 103b can detect another gesture (for example, a tap gesture) on the note icon in the drawer screen area 71 performed by the user using the pen 100 or a finger. In response to detecting a gesture (for example, a tap gesture) on a note icon on the drawer screen area 71, the handwritten note application program 103b moves the note icon to the center of the desktop screen area 70. The handwritten note application program 103b selects a handwritten note corresponding to the note icon, and displays a note preview screen shown in FIG. 7 instead of the desktop screen. The note preview screen shown in FIG. 7 is a screen on which an arbitrary page in the selected handwritten note can be viewed.

  Furthermore, the handwritten note application program 103b can also detect a gesture (for example, a tap gesture) on the desktop screen area 70 performed by the user using the pen 100 or a finger. In response to detecting a gesture (for example, a tap gesture) on a note icon located at the center of the desktop screen area 70, the handwritten note application program 103b selects a handwritten note corresponding to the note icon located at the center. Then, a note preview screen shown in FIG. 7 is displayed instead of the desktop screen.

  Further, the home screen can display a menu. This menu includes a note list button 81A, a note creation button 81B, a note deletion button 81C, a search button 81D, and a setting button 81E displayed in the lower part of the screen, for example, the drawer screen area 71. The note list button 81A is a button for displaying a list of handwritten notes. The note creation button 81B is a button for creating (adding) a new handwritten note. The note deletion button 81C is a button for deleting a handwritten note. The search button 81D is a button for opening a search screen (search dialog). The setting button 81E is a button for opening an application setting screen.

  Although not shown, a return button, a home button, and a recent application button are also displayed below the drawer screen area 71.

  FIG. 8 shows an example of a setting screen that is opened when the setting button 81E is tapped with the pen 100 or a finger.

  Various setting items are displayed on this setting screen. These setting items include “backup and restoration”, “input mode (pen or touch input mode)”, “license information”, “help”, and the like.

  When note creation button 81B is tapped with pen 100 or a finger on the home screen, a note creation screen is displayed. Here, the name of the note is input by handwriting in the title field. Note that a notebook cover and paper can be selected. When the creation button is pressed, a new note is created, and the created note is placed in the drawer screen area 71.

  FIG. 7 shows an example of the above-described note preview screen. The note preview screen is a screen capable of browsing an arbitrary page in the selected handwritten note. Here, a case where a handwritten note corresponding to the note icon 801 in the desktop screen area 70 of the home screen is selected will be described. In this case, the handwritten note application program 103b can visually recognize a plurality of pages 901, 902, 903, 904 and 905 included in the handwritten note, at least a part of each of these pages 901, 902, 903, 904 and 905, In addition, these pages 901, 902, 903, 904, and 905 are displayed in an overlapping form.

  The note preview screen further displays the pen icon 771, the calendar icon 772, and the scrap note icon 773 described above.

  The note preview screen can further display a menu at the bottom of the screen. This menu includes a home button 82A, a page list button 82B, a page add button 82C, a page edit button 82D, a page delete button 82E, a label button 82F, a search button 82G, and a property display button 82H. The home button 82A is a button for closing the preview of the note and displaying the home screen. The page list button 82B is a button for displaying a list of pages in the currently selected handwritten note. The page addition button 82C is a button for creating (adding) a new page. The page edit button 82D is a button for displaying a page edit screen. The page deletion button 82E is a button for deleting a page. The label button 82F is a button for displaying a list of types of labels that can be used. The search button 82G is a button for displaying a search screen. The property display button 82H is a button for displaying the property of this note.

  The handwritten note application program 103b can detect various gestures on the note preview screen performed by the user. For example, in response to detection of a certain gesture, the handwritten note application program 103b changes the page to be displayed at the top to an arbitrary page (page advance, page return). Also, in response to detection of a certain gesture (for example, tap gesture) performed on the top page, or in response to detection of a gesture (for example, tap gesture) performed on the pen icon 771 or editing. In response to detecting a gesture (eg, a tap gesture) made on button 82D, handwritten note application program 103b selects the top page and instead of the note preview screen, the page shown in FIG. Display the edit screen.

  The page editing screen shown in FIG. 9 is a screen that allows new creation of a page (handwritten page) in a handwritten note and browsing and editing of an existing page. When the page 901 on the note preview screen shown in FIG. 7 is selected, the page editing screen displays the contents of the page 901 as shown in FIG.

  In this page editing screen, a rectangular area 500 surrounded by a broken line is a handwritten input area that can be handwritten. In the handwriting input area 500, an input event from the digitizer 12C is used for displaying (drawing) a handwritten stroke, and is not used as an event indicating a gesture such as a tap. On the other hand, in an area other than the handwriting input area 500 on the page editing screen, an input event from the digitizer 12C can be used as an event indicating a gesture such as a tap.

  An input event from the touch panel 12B is not used for displaying (drawing) a handwritten stroke, but is used as an event indicating a gesture such as a tap and swipe.

  The page editing screen further displays a quick select menu including three types of pens 501 to 503 registered in advance by the user, a range selection pen 504, and an eraser pen 505 at the top of the screen outside the handwriting input area 500. Here, a case where the black pen 501, red pen 502, and marker 503 are registered in advance by the user is described. The user can switch the type of pen to be used by tapping a pen (button) in the quick select menu with the pen 100 or a finger. For example, when the handwriting input operation using the pen 100 is performed on the page editing screen in a state where the black pen 501 is selected by the user using the pen 100 or a tap gesture using a finger, the handwriting note application program 103b is A black stroke (trajectory) is displayed on the page edit screen in accordance with 100 movements.

  The above-mentioned three types of pens in the quick select menu can be switched by operating a side button (not shown) of the pen 100. For each of the above-described three types of pens in the quick select menu, combinations of frequently used pen colors and pen thicknesses can be set.

  The page editing screen further displays a menu button 511, a page return (return to note preview screen) button 512, and a new page addition button 513 at the bottom of the screen outside the handwriting input area 500. The menu button 511 is a button for displaying a menu.

  This menu includes, for example, putting this page in the trash, pasting a part of the copied or cut page, opening the search screen, displaying the export submenu, displaying the import submenu, converting the page to text Buttons for sending an e-mail and displaying a pen case may be displayed. The export submenu is, for example, a function for recognizing a handwritten page displayed on the page editing screen and converting it into an electronic document file, presentation file, image file, etc., or converting a page into an image file and other applications. Have users select the features to share. For example, the import submenu allows the user to select a function for importing a memo from the memo gallery or a function for importing an image from the gallery. The pen case is a button for calling up the pen setting screen that allows you to change the color (color of the line to be drawn) and thickness (thickness of the line to be drawn) of each of the three types of pens in the quick select menu. is there.

  FIG. 10 shows an example of a search screen (search dialog). FIG. 10 illustrates a case where the search button 82G is selected on the note preview screen shown in FIG. 7 and the search screen (search dialog) is opened on the note preview screen.

  The search screen displays a search key input area 530, a handwriting search button 531, a text search button 532, a delete button 533, and a search execution button 534. The handwriting search button 531 is a button for selecting handwriting search. The text search button 532 is a button for selecting text search. The search execution button 534 is a button for requesting execution of search processing.

  In handwriting search, the search key input area 530 is used as an input area for handwriting a character string, a figure, a table, and the like to be used as a search key. In FIG. 10, the handwritten character string “Determine” is input as a search key in the search key input area 530. The user can handwrite not only the handwritten character string but also a handwritten figure, a handwritten table, and the like in the search key input area 530 with the pen 100. When the search execution button 534 is selected by the user while the handwritten character string “Determine” is input as a search key in the search key input area 530, a stroke set (query stroke set) constituting the handwritten character string “Determine” is selected. The handwriting search for searching for a handwritten document (note) including a stroke set corresponding to the query stroke set is executed. In handwriting search, a stroke set similar to the query stroke set is searched by matching between strokes. In calculating the similarity between the query stroke set and another stroke set, DP (Dynamic Programming) matching may be used.

  In text search, for example, a software keyboard is displayed on the screen. The user can input an arbitrary text (character string) to the search key input area 530 as a search key by operating the software keyboard. When the search execution button 534 is selected by the user in a state where text is input as a search key in the search key input area 530, text search for searching for a handwritten note including a stroke set representing the text (query text) is performed. Executed.

  The handwriting search / text search can be executed for all handwritten documents, or can be executed only for selected handwritten documents. When handwriting search / text search is executed, a search result screen is displayed. On the search result screen, a list of handwritten documents (pages) including a stroke set corresponding to the query stroke set (or query text) is displayed. Note that hit words (a stroke set corresponding to a query stroke set or a query text) are highlighted.

  Next, the functional configuration of the handwritten note application program 103b executed by the tablet computer 10 will be described with reference to FIG.

  The handwritten note application program 103b is a WYSIWYG application capable of handling handwritten document data. The handwritten note application program 103b includes, for example, a display processing unit 301, a time-series information generation unit 302, an editing processing unit 303, a page storage processing unit 304, a page acquisition processing unit 305, a work memory 401, and the like. The display processing unit 301 includes a handwritten data input unit 301A, a handwriting drawing unit 301B, and a UI display processing unit 301C.

  The touch panel 12B described above is configured to detect occurrence of events such as “touch (contact)”, “movement (slide)”, and “release”. “Touch (contact)” is an event indicating that an object (finger) on the screen has touched. “Move (slide)” is an event indicating that the contact position has been moved while the object (finger) is in contact with the screen. “Release” is an event indicating that an object (finger) has been released from the screen.

  The above-described digitizer 12C is also configured to detect occurrence of events such as “touch (contact)”, “movement (slide)”, and “release”. “Touch (contact)” is an event indicating that the object (pen 100) has touched the screen. “Move (slide)” is an event indicating that the contact position has been moved while the object (pen 100) is in contact with the screen. “Release” is an event indicating that the object (pen 100) has been released from the screen.

  The handwritten note application program 103b displays on the touch screen display 12 a page editing screen for creating, browsing and editing handwritten page data.

  The display processing unit 301 and the time-series information generation unit 302 receive a “touch (contact)”, “move (slide)”, or “release” event generated by the digitizer 12C, and thereby detect a handwriting input operation. . The “touch (contact)” event includes the coordinates of the contact position. The “movement (slide)” event includes the coordinates of the contact position of the movement destination. Therefore, the display processing unit 301 and the time-series information generating unit 302 can receive a coordinate sequence corresponding to the movement locus of the contact position from the digitizer 12C.

  The display processing unit 301 displays a handwritten stroke on the screen according to the movement of the object (pen 100) on the screen detected using the digitizer 12C. The display processing unit 301 displays the trajectory of the pen 100 while the pen 100 is in contact with the screen, that is, the trajectory of each stroke on the page editing screen.

  The time series information generation unit 302 receives the above-described coordinate sequence output from the digitizer 12C, and includes time series information (coordinate data sequence) having a structure as described in detail in FIG. 4 based on the coordinate sequence. Generate handwritten data. The time series information generation unit 302 temporarily stores the generated handwritten data in the work memory 401.

  The edit processing unit 303 executes a process for editing the handwritten page currently being displayed. That is, the edit processing unit 303 adds a new stroke (new handwritten character, new handwritten mark, etc.) to the currently displayed handwritten page in response to an editing operation and a handwriting input operation performed by the user on the touch screen display 12. Edit processing including processing for adding a character, processing for deleting or moving one or more of the displayed strokes, and the like are executed. Further, the edit processing unit 303 updates the time series information in the work memory 401 in order to reflect the result of the edit process in the time series information being displayed.

  The page storage processing unit 304 stores handwritten page data including a plurality of stroke data corresponding to a plurality of handwritten strokes on the handwritten page being created in the storage medium 402. The storage medium 402 may be, for example, a storage device in the tablet computer 10 or may be a storage device of the server (computer) 2, for example.

  The page acquisition processing unit 305 acquires arbitrary handwritten page data from the storage medium 402. The acquired handwritten page data is sent to the display processing unit 301. The display processing unit 301 displays a plurality of strokes corresponding to the plurality of stroke data included in the handwritten page data on the screen.

Next, details of the display processing unit 301 illustrated in FIG. 11 will be described.
As described above, the touch screen display 12 detects a touch operation on the screen by the touch panel 12B or the digitizer 12C. The handwritten data input unit 301A is a module that inputs a detection signal output from the touch panel 12B or the digitizer 12C. The detection signal includes coordinate information (X, Y) of the touch position. By inputting such detection signals in chronological order, the handwritten data input unit 301A inputs stroke data corresponding to a stroke corresponding to a handwriting input operation (that is, written in handwriting). The stroke data (detection signal) input by the handwritten data input unit 301A is supplied to the handwriting drawing unit 301B.

  The handwriting drawing unit 301 </ b> B is a module that draws a handwriting input locus (handwriting) and displays it on the LCD 12 </ b> A of the touch screen display 12. The handwriting drawing unit 301B draws a line segment corresponding to the locus (handwriting) of handwriting input based on the stroke data (detection signal) from the handwriting data input unit 301A.

  When the stroke data input by the handwriting data input unit 301A corresponds to a stroke corresponding to a handwriting input operation on the page editing screen (upper handwriting input area 500) described above, for example, the UI display processing unit 301C This is a module for displaying the input auxiliary UI described above.

  Here, with reference to FIG. 12, an example of a user operation area when a pen 100 is used to perform a handwriting input operation on the screen of the touch screen display 12 (that is, one or more strokes are written by handwriting). Will be described.

  FIG. 12 shows an example in which the user writes a handwritten character string in horizontal writing using the pen 100. Here, the handwriting direction when the handwritten character string is written horizontally is the left-to-right direction as viewed from the user. Depending on the language, the stroke direction may be from the right to the left as viewed from the user.

  An area 1001 illustrated in FIG. 12 is an area in which the user's hand contacts on the touch screen display 12 when a handwriting input operation is performed on the touch screen display 12.

  The area 1002 is the maximum area (a handwritten input possible area) in which the user can move the pen tip of the pen 100 in a state where the user's hand is in contact with the above-described area 1001. Note that the user can perform a handwriting input operation within the range of the area 1002, but in the vicinity of the outer periphery of the area 1002 (for example, the upper part and the lower part), the maximum distance that the pen tip can be moved is Therefore, the handwriting input operation becomes relatively difficult.

  On the other hand, the region 1003 is a region excluding the region in which the handwriting input operation is relatively difficult (that is, a region where the handwriting input operation can be easily performed) from the region 1002 described above. That is, the region 1003 is an optimal region for writing a stroke by handwriting in a state where the user's hand is in contact with the region 1001 described above, and is a region where the user feels easy to operate.

  Here, in the present embodiment, a function for assisting a handwriting input operation can be used via the input assist UI described above. This input auxiliary UI is preferably displayed in the above-described region 1003 in consideration of user operability.

  However, when there is no need to perform an operation on the input auxiliary UI, the input auxiliary UI displayed in the area 1003 is likely to hinder a user's handwriting input operation.

  On the other hand, when the input auxiliary UI is displayed in, for example, the area 1002 other than the area 1003 in order to avoid the hindrance to the handwriting input operation, the operation on the input auxiliary UI becomes difficult.

  Therefore, in the present embodiment, the UI display processing unit 301C has a predetermined distance (first distance) from the area 1003 (first area) where the handwriting input operation is performed in a state where the user's hand is in contact with the area 1001 in the handwriting direction. ) The input auxiliary UI is displayed at a distant position. A specific display example of the input auxiliary UI will be described later.

  Although not shown in FIG. 11, the handwritten note application program 103b may use, for example, a stroke set specified based on one or more strokes according to a handwriting input operation as a candidate for handwriting input by the user other than the above. A candidate display processing unit displayed on the page editing screen and a search processing unit for executing the above-described handwriting search and text search are provided.

  Hereinafter, the operation of the tablet computer 10 according to the present embodiment will be described. Here, among the processes executed by the tablet computer 10 according to the present embodiment, a process (hereinafter referred to as input assist) for displaying the input assist UI during the above-described handwriting input operation (that is, when strokes are written by handwriting). The UI display process and notation) will be mainly described.

  With reference to the flowchart of FIG. 13, an example of the processing procedure of the input auxiliary UI display processing will be described. The input auxiliary UI display process is executed when stroke data corresponding to a stroke corresponding to a handwriting input operation is input in the handwriting input area 500 on the page editing screen.

  First, the handwritten data input unit 301A inputs stroke data corresponding to one or more strokes according to a handwriting input operation by the user (block B1). When stroke data is input by the handwriting data input unit 301A in this way, the handwriting drawing unit 301B draws a stroke (a line segment corresponding to a handwriting input locus) based on the stroke data. Execute the process.

  When stroke data is input by performing the handwriting input operation in this way, it is assumed that the user's hand is in contact with the screen of the touch screen display 12. In the following description, a point on the screen that is touched by the user's hand when stroke data is input in block B1 is referred to as a first contact point. The first contact point (coordinates thereof) can be detected by the touch panel 12B provided in the touch screen display 12 based on the above-described region 1001 shown in FIG.

  Next, the UI display processing unit 301C determines whether or not the user's hand has left the screen of the touch screen display 12 based on the input event from the touch panel 12B (block B2).

  When it is determined that the user's hand is not separated from the screen of the touch screen display 12 (NO in block B2), the handwriting input operation is continued while the user's hand is in contact with the first contact point As described above, the processing is repeated after returning to the above-described block B1. That is, in the present embodiment, the input auxiliary UI is not displayed (that is, not displayed) while the state in which the user's hand is in contact with the screen of the touch screen display 12 continues.

  Here, when the user describes a relatively long character string (stroke set) on the screen of the touch screen display 12 using the pen 100, the user once removes his / her hand from the screen for handwriting, and next In many cases, an operation of bringing a hand into contact with the screen again (hereinafter referred to as a stroke operation) is performed to describe the characters (columns).

  In this embodiment, when it is determined that the user's hand has been removed from the screen of the touch screen display 12 in the above-described block B2 in order to determine such a stroke operation (YES in block B2), UI display processing The unit 301C determines whether or not the user's hand has come into contact with the screen of the touch screen display 12 again (block B3).

  When it is determined that the user's hand has come into contact with the screen of the touch screen display 12 again (YES in block B3), the UI display processing unit 301C can determine that the above-described brushing operation has been performed. In the following description, a point on the screen where the user's hand is in contact after such a stroke operation is performed is referred to as a second contact point. The second contact point (coordinates) can be detected by the touch panel 12B in the same manner as the first contact point described above.

  Even if the user's hand once touches the screen after the user's hand has left the screen, the movement of the user's hand may not be due to the above-described handwriting action. Therefore, in order to improve the discrimination accuracy of the stroke operation, for example, the stroke operation is performed only when the position of the second contact point is within a predetermined distance from the first contact point and in the stroke direction, for example. You may make it discriminate | determine that it was broken. The stroke direction can be determined from the stroke data input in the block B1.

  When it is determined that the stroke operation has been performed as described above, the UI display processing unit 301C determines a position (display position of the input auxiliary UI) for displaying the input auxiliary UI described above (block B4). In this case, for example, the UI display processing unit 301C moves from the region where the handwriting input operation is performed in the state where the user's hand is in contact with the first contact point on the screen described above (for example, the region 1003 shown in FIG. 12) in the direction of handwriting A position separated by a predetermined distance is determined as the display position of the input auxiliary UI.

  Here, with reference to FIG. 14, the determination process of the display position of the input auxiliary UI (hereinafter referred to as the first determination process) will be described in detail. Here, the same parts as those in FIG. 12 described above are denoted by the same reference numerals, and detailed description thereof is omitted.

  An area 1004 shown in FIG. 14 is displayed when the user's hand is in contact with the area 1001 on the screen and then the user's hand is moved when the hand is moved by a handwriting operation to perform the handwriting input operation. This is the area that touches. In FIG. 14, a point 1001a and a point 1004a indicate the first contact point and the second contact point described above, respectively. The first contact point 1001a and the second contact point 1004a are points located at the leftmost ends of the region 1001 and the region 1004, respectively. It doesn't matter.

  In the process of determining the display position of the input auxiliary UI, the UI display processing unit 301C acquires, for example, the X coordinate (hereinafter referred to as X1 coordinate) of the point located at the rightmost end of the region 1002 (and the region 1003). The areas 1002 and 1003 (and the X1 coordinates) are calculated from the stroke data (time series coordinates) input in the block B1 and the position (coordinates) of the first contact point 1001a detected by the touch panel 12B. Shall be.

  Also, the UI display processing unit 301C acquires the X coordinate (hereinafter referred to as X2 coordinate) of the second contact point 1004a.

  Next, the UI display processing unit 301C includes a Y coordinate (hereinafter referred to as Y1 coordinate) of a point located at the lowermost end of the area 1003 and a Y coordinate (hereinafter referred to as Y2 coordinate) of a point located at the uppermost end of the area 1003. (Notation).

  Thereby, the UI display processing unit 301C calculates the region 1005 based on the X1, X2, Y1, and Y2 coordinates as shown in FIG.

  In this case, the UI display processing unit 301C determines the calculated region 1005 (inside position) as a position for displaying the input auxiliary UI. The display position of the input auxiliary UI may be within the area 1005, but the user determines the relationship or relationship between the input auxiliary UI and the input auxiliary function used via the input auxiliary UI. It shall be within a range where Specifically, when the input assist function used via the input assist UI is a function for changing the color of the handwritten character string described above, the input assist UI is, for example, the handwritten character described in the area 1003 It shall be displayed near the column.

  Here, as indicated by regions 1001 to 1003, handwriting input operations cannot be performed near the region 1001. For this reason, there is a certain interval between the area 1001 on the screen where the user's hand contacts and the handwritten input possible areas 1002 and 1003. According to this, it is presumed that the handwriting input operation cannot be performed even in the vicinity of the region 1004. Therefore, in order to display the input auxiliary UI at a position (area) where the user can easily perform a handwriting input operation, the X coordinate of the second contact point described above is set to, for example, the X coordinate of the first contact point and X1. It is preferable that the position where the input auxiliary UI is displayed is determined using the X coordinate of the position shifted in the negative direction of the X axis by the difference from the coordinate as the X2 coordinate.

  Further, here, the Y coordinate (Y1 coordinate) of the point located at the lowermost end of the region 1003 and the Y coordinate (Y2 coordinate) of the point located at the uppermost end of the region 1003 have been described, but the Y1 coordinate and Instead of the Y2 coordinate, for example, the Y coordinate of the point located at the lowermost end of the region 1002 and the Y coordinate of the point located at the uppermost end of the region 1002 may be used.

  Note that the process of determining the display position of the input auxiliary UI described here is merely an example, and the display position of the input auxiliary UI may be determined by other processes. Specifically, for example, based on the positional relationship between the region 1001 and the region 1003, a region where a handwriting input operation can be easily performed when the user's hand is in contact with the region 1004 is calculated. The area (inside position) may be determined as the display position of the input auxiliary UI.

  Returning to FIG. 13 again, the UI display processing unit 301C displays the input auxiliary UI at the display position of the input auxiliary UI determined as described above (block B5). The user can use the input assist function described above by performing a touch operation using the pen 100, for example, on the input assist UI displayed in this way. It is preferable that the input auxiliary UI displayed in this way is quickly hidden when it is determined that the input auxiliary UI is unnecessary for smooth handwriting input operation by the user. Specifically, for example, when an operation with respect to the input auxiliary UI is not performed for a certain period of time, it is determined that the input auxiliary UI is unnecessary and can be hidden.

  When it is determined that the user's hand is not in contact with the screen of the touch screen display 12 in block B3 described above (NO in block B3), for example, the input assist UI display indicates that the handwriting input operation has been completed. Processing is terminated.

  Next, the display of the input auxiliary UI described above will be specifically described. Here, a first display example and a second display example of the input auxiliary UI will be described.

  First, a first display example of the input auxiliary UI will be described with reference to FIG. FIG. 15 shows a case where the above-described stroke operation is performed after stroke data corresponding to one or more strokes (here, handwritten character string “ABC”) corresponding to a handwriting input operation by the user is input. .

  As shown in FIG. 15, when a stroke operation is performed, the input auxiliary UI 1100 is displayed at a position away from the region 1003 where the handwriting input operation has been performed before the stroke operation is performed by a predetermined distance in the stroke direction. . According to this, when the user finishes writing the handwritten character string “ABC” in the area 1003 and performs the handwriting operation, the user can easily perform an operation (touch operation or the like) on the input auxiliary UI 1100, and Handwriting input operations (such as writing a handwritten character string) can be performed.

  The input assistance UI 1100 shown in FIG. 15 is a user interface for using an input assistance function for changing the color of a handwritten character string (ie, a user used for an operation for designating the color of a handwritten character string). Interface).

  In the example illustrated in FIG. 15, the input auxiliary UI 1100 is provided with, for example, an “R (red)” button 1101, a “G (green)” button 1102, and a “B (blue)” button 1103. The user can change the color of the handwritten character string “ABC” to a color corresponding to the button 1101 (that is, red) by performing a touch operation on the “R” button 1101 using the pen 100, for example. Although the “R” button 1101 has been described here, the same applies to a case where a touch operation is performed on the other buttons 1102 and 1103.

  Note that the input auxiliary UI 1100 may be provided with buttons other than the buttons 1101 to 1103 corresponding to red, green, and blue. It is assumed that the type (color corresponding to the button), the number, and the like of each button provided in the input auxiliary UI 1100 can be appropriately changed (set) by the user.

  In addition, when a handwritten note (handwritten document) is created in the present embodiment, each time a handwritten character string is written on the screen of the touch screen display 12, a predetermined operation (hereinafter referred to as a confirming operation) is performed on the screen. An operation of confirming (input) the handwritten character string is performed. In this case, the function (in this case, the function for designating the color of the handwritten character string) used via the input auxiliary UI 1100 shown in FIG. 15 has not been confirmed by the above-described confirming operation, that is, not confirmed (during editing). ) Handwritten character string (that is, one or more strokes). In other words, in the present embodiment, an undetermined stroke is an operation target using the input auxiliary UI 1100.

  Next, a second display example of the input auxiliary UI will be described with reference to FIG. Here, the tablet computer 10 according to the present embodiment (the handwritten note application program 103b operating above) displays the above-described handwriting input candidates (hereinafter referred to as recommended handwritten character strings) as input assist functions (hereinafter referred to as recommended handwritten character strings). It is assumed that it has an input suggest function. According to this input suggestion function, a stroke set specified based on one or more strokes corresponding to a handwriting input operation (for example, composed of strokes similar to the one or more strokes among handwritten character strings described in the past) Handwritten character string) is displayed as a recommended handwritten character string.

  FIG. 16 shows a case where the above-described handwriting operation is performed after such a recommended handwritten character string is displayed.

  Here, in the example illustrated in FIG. 16, for example, when the handwritten character “a” is written by handwriting by a handwriting input operation, the recommended handwriting specified based on the handwritten character “a” (that is, one or more strokes). The character strings “app”, “apple”, and “air” are displayed.

  In FIG. 16, the above-described areas 1001 to 1003 are omitted, but the recommended handwritten character string “app” is, for example, an area 1201 (second area) located above the area 1003 (area where the handwriting input operation is performed). Area). Also, the recommended handwritten character string “apple” is displayed in, for example, the area 1003 (overlapping area 1202). Further, the recommended handwritten character string “air” is displayed in, for example, a region 1203 (third region) located below the region 1003. Of the recommended handwritten character strings to be displayed, for example, the recommended handwritten character string having the highest priority is displayed in the area 1202. This priority can be calculated based on, for example, the degree of coincidence with a handwritten character string written by handwriting by a handwriting input operation, the number of times selected by the user when displayed as a recommended handwritten character string in the past, and the like. However, it may be calculated by other methods. In the example illustrated in FIG. 16, the recommended handwritten character string is displayed in each of the areas 1201 to 1203, but the recommended handwritten character string is displayed in at least one of the areas 1201 to 1203. Good.

  When the recommended handwritten character string is displayed in this way, the user can select one of the recommended handwritten character strings as a character string to be displayed (in the handwriting input area 500). That is, when one of the recommended handwritten character strings is selected by the user, the handwriting drawing unit 301 </ b> B displays (draws) the recommended handwritten character string in the handwritten input area 500.

  In the example illustrated in FIG. 16, when a handwriting action is performed, an input auxiliary UI 1205 that can select (specify) one of recommended handwritten character strings by moving the cursor 1204 is displayed. For example, when the recommended handwritten character string “apple” displayed in the area 1202 is selected, the input auxiliary UI 1205 is in the vicinity of the position where the next handwriting input operation is estimated to be performed (that is, the recommended handwritten character string “apple”). Displayed after the handwritten character string).

  In this input auxiliary UI 1205, for example, an “up” button 1205 a for moving the cursor 1204 described above, a “down” button 1205 b for moving the cursor 1204 down, and a recommended handwriting combined with the cursor 1204. An “OK” button 1205c for designating a character string is provided.

  According to this, it is possible to easily perform an operation (that is, an operation of selecting a recommended handwritten character string) on the input auxiliary UI 1205 after the handwriting operation, and it is possible to easily perform the next handwriting input operation.

  The input assistance UIs 1100 and 1205 described with reference to FIGS. 15 and 16 are examples, and the input assistance UI in the present embodiment changes, for example, the thickness of the handwritten character string, or enlarges or reduces the handwritten character string. For using a function such as deleting the handwritten character string, specifying the handwritten character string as a search key, or executing a character recognition process on the handwritten character string, or a function combining these functions It may be. That is, the input assistance UI in the present embodiment may be any one that uses a function that assists the user's handwriting input operation.

  As described above, in the present embodiment, one or more strokes (first strokes) input by handwriting in a state where a hand is in contact with a contact point (first contact point) on the screen of the touch screen display 12 are supported. After inputting stroke data (first stroke data) and the hand leaving the contact point, a predetermined distance (first distance) is left in the stroke direction from the area (first area) where the one or more strokes are displayed. An input auxiliary UI (a user interface used for an operation for designating an input stroke) is displayed at the position. Specifically, a hand movement operation is performed after one or more strokes (first strokes) are input with a hand touching a contact point (first contact point) on the screen, and different contact points ( When one or more strokes (second stroke) are input in a state where the hand is in contact with the second contact point), the input auxiliary UI displays the region where the one or more strokes are displayed before the stroke operation and the stroke operation. It is displayed at a position between the subsequent contact points.

  In the present embodiment, the input auxiliary UI is displayed at a position where the handwriting input operation is not hindered or interrupted (that is, in the vicinity of the position where the next handwriting input operation is performed after the handwriting operation) according to such a configuration. Can easily operate the input auxiliary UI, and can efficiently create a handwritten document (handwritten note).

  In the present embodiment, an area 1202 (first area) in which one or more strokes are written by hand, an area 1201 (second area) located above the area, and an area located below the area A recommended handwritten character string (candidate for handwriting input) is displayed in at least one area of 1203 (third area), and an input auxiliary UI 1205 used for an operation for designating the recommended handwritten character string is displayed. In this embodiment, with such a configuration, when a function (display suggestion function) for displaying (presenting) a recommended handwritten character string is provided, an operation for specifying the recommended handwritten character string (shown in FIG. 16). It is possible to display an input auxiliary UI that can easily perform an operation on the input auxiliary UI 1205.

  Furthermore, in this embodiment, while the state in which the user's hand is in contact with the screen of the touch screen display 12 continues, the input auxiliary UI is not displayed, so that, for example, a handwriting input operation is performed on the screen. When the operation is continuously performed, it is possible to prevent an operation on the input auxiliary UI unintended by the user from being performed.

  Further, in the present embodiment, an undetermined stroke is set as an operation target using the input auxiliary UI (that is, the input auxiliary function used via the input auxiliary UI is applied to the undetermined stroke. Thus, it is possible to prevent the input assist function from being applied even to a stroke (handwritten character string) in a range not intended by the user (that is, the effect of the input assist function is generated).

  In the present embodiment, the input auxiliary UI is displayed at the position determined by executing the first determination process described in FIG. 13 and FIG. 14 (that is, the input auxiliary UI is displayed after the handwriting operation). The input auxiliary UI is displayed before the user's hand touches the screen again (that is, while the hand is moving to the next handwriting input position). Also good. In this case, the display position of the input auxiliary UI is determined by predicting the position on the screen where the user next touches the hand.

  Hereinafter, with reference to FIG. 17, a process for determining the display position of the input auxiliary UI in this case (hereinafter referred to as a second determination process) will be described. Here, the same parts as those in FIGS. 12 and 14 described above are denoted by the same reference numerals, and detailed description thereof is omitted.

  In the case of the second determination process, the UI display processing unit 301C acquires, for example, the X coordinate (hereinafter referred to as the X0 coordinate) of a point located at the leftmost end of the area 1002 (and the area 1003).

  In addition, the UI display processing unit 301C acquires the X1 coordinates described in FIG.

  Note that the area 1002 and the area 1003 (X0 coordinate and X1 coordinate) are calculated from the stroke data (time series coordinates) input in the block B1, the position (coordinates) of the first contact point 1001a detected by the touch panel 12B, and the like. Shall.

  Here, as shown in FIG. 17, the UI display processing unit 301C calculates a difference d between the acquired X1 coordinate and the X0 coordinate, and adds the calculated difference d to the X1 coordinate (hereinafter referred to as X2). ('Coordinate and notation) are calculated.

  Next, the UI display processing unit 301C acquires the Y1 coordinate and the Y2 coordinate described with reference to FIG.

  Accordingly, the UI display processing unit 301C calculates the region 1006 based on the X1, X2 ′, Y1, and Y2 coordinates as shown in FIG.

  In this case, the UI display processing unit 301C determines the calculated area 1006 (inside position) as a position for displaying the input auxiliary UI.

  That is, in the second determination process, as described above, the region 1006 where the next handwriting input operation is estimated to be performed is predicted based on the region 1002 and the region 1003, and the position in the region 1006 is set as the input assist UI. The display position is determined.

  According to the second determination process, it is possible to display the input auxiliary UI before the user's hand is placed on (contacted with) the screen by the user's handwriting operation. It can be created intentionally.

  However, assuming that the handwritten character string is written in horizontal writing, when the input auxiliary UI is displayed at the position determined by the second determination process, the hand movement is in progress (that is, the user is moving the hand). In the middle), the input auxiliary UI is displayed under the user's hand. That is, there is a possibility that a touch operation unintended by the user (that is, an erroneous operation) is performed until the user's stroke operation is completed (that is, until the user places his / her hand on the screen).

  On the other hand, according to the first determination process described above, the input assist UI is displayed after the user's stroke operation (that is, after the user's hand is placed on the screen). There is a low possibility that an erroneous operation is performed as compared with the case where the process is executed.

  Whether the first or second determination process described above is executed (that is, whether the input auxiliary UI is displayed after the user's hand is placed on the screen, or the user's hand is on the screen) Whether the auxiliary input UI is displayed before being placed) is determined by the use case.

  In the present embodiment, the case where the user writes the handwritten character string in horizontal writing using the pen 100 has been mainly described. However, the present embodiment is also applied to the case where the handwritten character string is written in vertical writing. Is possible. Here, the handwriting direction when the handwritten character string is written vertically is different from the case where the handwritten character string is written horizontally, as viewed from the top. In addition, the handwriting input available area may be different from the area 1002 in the horizontal writing depending on how to hold and move the pen 100 that is optimized in the writing direction and the handwriting habit of the user. Furthermore, unlike the region 1003 shown in FIG. 12 described above, the shape of the region where the handwriting input operation can be easily performed in the case where the handwritten character string is written vertically is also a vertically long elliptical shape. In this way, there is a difference in the direction of stroke as described above between the case where the handwritten character string is written in horizontal writing and the case where the handwritten character string is written in vertical writing, but even when the handwritten character string is written in vertical writing, By executing the processing as shown in FIG. 13, it is possible to display the input auxiliary UI at a position away from the region where the user performs a handwriting input operation in the stroke direction. Thereby, even when a handwritten character string is described in vertical writing, the same effect as in the case of writing the above-described handwritten character string in horizontal writing can be realized.

  Since the processing of the present embodiment can be realized by a computer program, the computer program is installed and executed on a computer through a computer-readable storage medium that stores the computer program, as in the present embodiment. The effect of can be easily realized.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalents thereof.

  DESCRIPTION OF SYMBOLS 10 ... Tablet computer (electronic device), 11 ... Main body, 12 ... Touch screen display, 12A ... LCD, 12B ... Touch panel, 12C ... Digitizer, 100 ... Pen, 101 ... CPU, 102 ... Non-volatile memory, 103 ... Main memory, DESCRIPTION OF SYMBOLS 103a ... Operating system, 103b ... Handwritten note application program, 104 ... BIOS-ROM, 105 ... System controller, 106 ... Graphics controller, 107 ... Wireless communication device, 108 ... EC, 301 ... Display processing part, 301A ... Handwritten data input part , 301B: handwriting drawing unit, 301C: UI display processing unit, 302 ... time-series information generation unit, 303 ... editing processing unit, 304 ... page storage processing unit, 305 ... page acquisition processing unit, 401 ... working memory, 402 ... storage Medium .

Claims (15)

Inputting first stroke data corresponding to one or more first strokes input by handwriting in a state where the hand is in contact with the first contact point on the screen;
After the hand leaves the first contact point, it is used for an operation for designating an input stroke at a position away from the first area where the one or more first strokes are displayed by a first distance in the stroke direction. Displaying a user interface.   After one or more first strokes are input with the hand touching the first contact point on the screen, one or more second strokes are input with the hand touching the second contact point on the screen. 2. The method of claim 1, wherein the user interface is displayed at a location between the first region and the second contact point. Further comprising displaying a stroke set identified based on the one or more strokes as a candidate for handwriting input,
The handwriting input candidates are displayed in at least one of the first region, the second region located above the first region, and the third region located below the first region,
The method according to claim 1, wherein the user interface is used for an operation for designating a candidate for the handwriting input.   The method according to claim 1, wherein the user interface is hidden while a state in which a hand touches the first contact point on the screen continues.   The method according to claim 1, wherein an undetermined stroke is a target of an operation using the user interface. Input means for inputting first stroke data corresponding to one or more first strokes input by handwriting in a state where the hand is in contact with the first contact point on the screen;
After the hand leaves the first contact point, it is used for an operation for designating an input stroke at a position away from the first area where the one or more first strokes are displayed by a first distance in the stroke direction. An electronic device comprising display processing means for displaying a user interface.   After one or more first strokes are input with the hand touching the first contact point on the screen, one or more second strokes are input with the hand touching the second contact point on the screen. The electronic device according to claim 6, wherein the user interface is displayed at a position between the first region and the second contact point. The display processing means displays a stroke set specified based on the one or more strokes as a candidate for handwriting input,
The handwriting input candidate is displayed in at least one area of the first area, the second area located above the first area, and the third area located below the first area,
The electronic device according to claim 6, wherein the user interface is used for an operation for designating a candidate for the handwriting input.   The electronic device according to claim 6, wherein the user interface is not displayed while a state in which a hand contacts the first contact point on the screen is continued.   The electronic apparatus according to claim 6, wherein an undetermined stroke is an operation target using the user interface. A program executed by a computer,
In the computer,
Inputting first stroke data corresponding to one or more first strokes input by handwriting in a state where the hand is in contact with the first contact point on the screen;
After the hand leaves the first contact point, it is used for an operation for designating an input stroke at a position away from the first area where the one or more first strokes are displayed by a first distance in the stroke direction. A program for displaying and executing the user interface.   After one or more first strokes are input with the hand touching the first contact point on the screen, one or more second strokes are input with the hand touching the second contact point on the screen. In this case, the user interface is displayed at a position between the first region and the second contact point. Causing the computer to further display a stroke set identified based on the one or more strokes as a candidate for handwriting input;
The handwriting input candidates are displayed in at least one of the first region, the second region located above the first region, and the third region located below the first region,
The program according to claim 11, wherein the user interface is used for an operation for designating a candidate for the handwriting input.   The program according to claim 11, wherein the user interface is not displayed while a state in which a hand contacts the first contact point on the screen continues.   The program according to claim 11, wherein an undetermined stroke is an operation target using the user interface.

Images