Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T11/00—2D [Two Dimensional] image generation
  • G06T11/60—Editing figures and text; Combining figures or text
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
  • G06F3/0304—Detection arrangements using opto-electronic means
  • G06F3/0317—Detection arrangements using opto-electronic means in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
  • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
  • G06F3/0354—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
  • G06F3/03545—Pens or stylus
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T11/00—2D [Two Dimensional] image generation

Definitions

• the present invention relates to editing of data generated by handwriting.
• Such a prior-art technique implies that the person uses a specific electronic device which is provided with a screen on which the person can write directly using a handwriting device in the form of an elongate instrument.
• the electronic device is constructed so as to be able to detect and store, as said data, the characters that the person is writing on the screen.
• the device is also capable of reproducing the characters on the screen so that the person may see what she is writing.
• Another prior-art technique implies that the person uses a handwriting device in the form of an ordinary pen and writes on an ordinary sheet of paper.
• the sheet of paper is specific insofar as it is provided with a position code coding a plurality of positions on the surface. The positions are located close to each other.
• the pen is also specific insofar as it comprises a device which detects the position code while the person is writing with the pen. The detected positions are stored, as said data, and can then be processed as any electronic data and can, for instance, be transferred to a computer or some other electronic device, for instance via short range radio, and be further processed as desired.
• the person who is writing sometimes wants to edit data directly and with the aid of the handwriting device.
• the object of the invention is to provide an accurate and efficient method of editing data generated by handwriting, said method affording a great freedom of choice as to how the user marks what is to be edited.
• the object is achieved by a method of editing data generated by handwriting according to the appended claim 1.
• the object is achieved by a handheld device and a computer program for editing data generated by handwriting according to the appended claims 23 and 24, respectively.
• a method of editing data generated by handwriting comprising the steps of - detecting an initiation of an editing; identifying a marking stroke which marks data that is to be edited; determining an essentially minimal convex envelope which surrounds the marking stroke; identifying the remaining data, in addition to the data of the actual marking stroke, which is positioned within the convex envelope; and performing an operation on said data positioned within the convex envelope.
• the method according to the invention is applicable to many different techniques of handling the handwriting and the generating of data, including the two techniques mentioned above.
• Each data set that relates to handwritten characters and that has a connection to position so that it can be determined to which data a marking which is also handwritten relates, can in principle be processed according to the method.
• written characters is besides here meant anything that a person may write and draw by hand.
• a character can thus be, for instance, a letter, an indefinite curve, a symbol or a freely drawn line included in a drawing.
• a "convex envelope” is understood by a person skilled in the art as a convex polygon which surrounds a quantity of dots. The convex polygon is such that a line between two arbitrary dots on the edge of the polygon is located completely inside the edge.
• minimal convex envelope is meant precisely the smallest possible convex polygon surrounding a quantity of dots. According to the invention, an approximation is provided in the form of an essentially minimal convex envelope.
• stroke is the expert term for a continuous curve formed between a putting down of a pen and a subsequent raising of the pen.
• penndrag is the Swedish term, which however is normally not used by those skilled in the art.
• An essential advantage of the above method according to the invention is that the use of the convex envelope to surround the marking stroke makes the method relatively insensitive to how the marking is made. For instance, a closed ring is not necessary, like in the above-discussed known document , but the marking can be made in the person' s usual way when writing on a sheet of paper with a pen, for instance an extended zigzag line or a spiral.
• the convex envelope is adapted according to the method so as to be essentially minimal. In other words, it is not made larger than necessary. In combination with the basic definition of the convex envelope, this adaptation means that the envelope essentially connects to the extreme points of the marking.
• the actual determination of the envelope implies that the convex envelope has an extent in a starting position and that the extent is then adapted to the marking as made, so that the marking stroke, or, if desired, the data representing the marking stroke, is precisely surrounded by the envelope, i.e. according to some rule belongs to the envelope, or, if desired, is located within the envelope.
• the envelope is illustrated on a writing surface by drawing the edge of the envelope round the marking as made.
• the envelope has in the starting position a minimum extent which is then extended according to a suitable algorithm until the marking is surrounded.
• the envelope can, in terms of definition, have the extent zero in the starting position.
• the minimal convex envelope is approximated with a convex envelope consisting of a plurality of intersecting planes.
• the time of execution for determining the convex envelope can be controlled particularly well. Besides, for different devices and different applications, different numbers of planes and, thus, easier or more complicated calculations, but also a more or less approximative restriction of the marking, can be selected according to means and needs .
• the representation in the x-y plane will then be a line which is given a direction represented by the normal (A,B) of the plane/line.
• the line can also be regarded as a vector. Consequently, it should be possible to use, in a two-dimensional case, the equation of the straight line, but in order not to confuse the reader, the plane equation is maintained as the form of the definition.
• the plane equation is advantageous for computerized calculations.
• the marking stroke is seen as a formation which comprises one or more positions.
• a quantity of positions is usually obtained in connection with the transfer to the electronic representation by the detection taking place in discrete moments at a certain detecting frequency, or sampling frequency.
• the extent of the convex envelope is changed for each position by relating the position to the location of each plane, one plane at a time, and moving the plane if the position according to a location criterion is positioned on the outside of the plane and, thus, outside the convex envelope, so that the position is located in or on the inside of the plane, and thus within the convex envelope.
• the expression "within the convex envelope” thus comprises according to this embodiment the limit position where a position is located in a plane. Since the number of planes is constant, a constant execution time for the processing of each position is obtained, which is advantageous. According to the invention, the remaining data that is positioned within the envelope is identified, i.e.
• This remaining data is to be found among the characters written before the marking stroke . More specifically, data is searched for one or more strokes performed before the marking stroke, and it is determined for each stroke whether at least a part thereof is positioned within the envelope. The operation is then carried out either on precisely the remaining data that is positioned within the envelope or on whole strokes, even if only a part of one or more strokes is represented among said remaining data.
• the last -mentioned embodiment i.e. a whole stroke being edited even if only a part thereof is positioned within the envelope, is advantageous, for instance, if the writing person wants to make a marking that is a single stroke, for instance an inclined stroke over a character that the writing person wants to delete.
• a marking that is a single stroke, for instance an inclined stroke over a character that the writing person wants to delete.
• the minimal convex envelope for such a marking is determined, probably only a part of the character to be deleted will be positioned within the envelope. Consequently it does not matter in this embodiment, and the entire character is edited, provided that it is a single stroke or that all the strokes involved are touched by the marking stroke .
• the marking alternative that is applicable can be conditioned, for instance by how the marking is performed.
• a favorable variant being that an essentially straight stroke indicates that the operation is to be carried out on the whole stroke whereas all other forms of marking indicate that exactly what is positioned within the envelope is to be edited.
• the above described method for determining the envelope can in many respects be used also to determine what data is positioned within the envelope.
• each stroke according to an embodiment being represented by one or more positions, the location of each position in relation to the plane of the envelope can, as stated above, be determined, and if it is established that the position is located within the envelope, it is identified to be included in data that is to be edited. Then the operation is carried out starting from said data that is to be edited.
• a closely related operation is to delete a marked character, i.e. to delete corresponding data.
• Another example is choice of colors, i.e. that data when presented on a display is shown in a selected color.
• Further examples are copying the marked character and formatting, i.e. choice of e.g. line thickness, hatching, font etc.
• the method further comprises detection of an editing being initiated.
• an initiation can be carried out in different ways depending on the technique that is used for the writing and the generating of data.
• it is detected that the writing person places a handwriting device within an initiating area on a surface.
• a particular initiating area is marked on or beside the writable surface.
• a small box may be 5 involved, marked with "Edit" for editing.
• the detected initiation results in either the immediately preceding or immediately following stroke 0 with the writing device being assumed to be the marking stroke.
• Other variants are, of course, feasible, but less natural to the writing person.
• FIG. 1 is a schematic view of a convex envelope in a starting position according to an embodiment of the 0 method according to the invention
• Figs 2a- f illustrate schematically an example of how the method according to the invention is used
• Fig. 3 shows an application of the invention where a pen is used for writing on a sheet of paper which is provided 5 with a position code
• Fig. 4 illustrates an alternative way of marking text that is to be edited
• Fig. 5 shows examples of operation boxes for various editing operations
• 0 Figs 6a-c illustrate a simple basic example of the application of the method step by step
• Fig. 7 is a flow chart of an embodiment of the inventive method.
• the handheld device for editing data generated by handwriting in accordance with the invention is here embodied by the writing device 31.
• the writing surface 33 is provided with a position code in the form of e.g. groups of dots of the type as disclosed, for example, in Applicant's International Patent Application WO 01/16691.
• the dots in the groups of dots are displaced in relation to a virtual raster in such a manner that each group of dots uniquely codes the position where it is located.
• the position code is not appreciably seen with the naked eye but is rather perceived as a gray shading of the writing surface 33.
• the writing surface 33 is, for instance, a sheet of paper.
• the writing device 31 is in the form of a pen and is used to write ordinary text on the writing surface 33.
• the pen thus is provided with an ink cartridge or the like which is not shown in more detail.
• the pen 31 is also provided with a reading unit 35, such as a camera, which reads the position code on the writing surface 33, and a processor (not shown) which is connected to the reading unit 35 and which is adapted to store and analyze data generated by the reading unit 35.
• the operation of the processor is, at least in part, controlled by a computer program for editing data generated by handwriting in accordance with the invention.
• the data generated represents what is being written, i.e. different characters such as letters, figures etc, on the writing surface 33.
• data indicates the positions that are read by the pen 31 when the user is writing with it.
• the writing device 31 simultaneously detecting the beginning and the end of each stroke, i.e. putting down and raising of the pen 31, the stored data simultaneously constitutes a representation of the characters that have been written.
• the writing device 31 further includes a transceiver 37 for wireless transfer of information across short distances. In addition to radio, for instance IR light and wire connection are conceivable alternative transferring techniques.
• the writing device is adapted to transfer position data for the read positions, preferably wirelessly by means of short range radio, to another device 39, for instance a server which preferably is reached via a PC, PDA or the like, or via a mobile phone.
• a server which preferably is reached via a PC, PDA or the like, or via a mobile phone.
• the written characters can easily be sent to, for example, an electronic device which comprises a display on which they can be shown.
• data is edited, it is thus possible to edit data that has been stored locally in the pen 31 and/or edit data that has already been sent to another device.
• the writing surface 51 is provided with a number of operation boxes 53, 55, 57, 59 as shown in Fig. 5.
• the operations indicated are only examples and many other operations are conceivable.
• One of the operation boxes, 53, is marked with "delete”. The user now puts the pen in the delete box 53 to initiate the editing.
• the process of the pen 31 detects that the pen 31 is placed in an operation box and identifies the actual marking stroke 21 as the stroke that follows the initiation. It is preferred for the user first to indicate that an editing is to be carried out and then to perform the marking, but the reverse order and other variants are within the scope of the invention.
• the next measure of the user is to mark the letter that is to be deleted by drawing an undulating stroke 21, see Fig. 2b, across the letter. It is the user's personal way of marking previously written characters. Basically, the inventor is in no way dependent on how this marking is made. It is the invention's strong point that the user does not have to follow stringent rules of how the marking is to be made. In practice, there are however only a few basic conditions that must be known to the user to ensure that an editing should touch precisely the desired part of what has been written, as will be explained in more detail below. However, these conditions are very easy to understand and easy to apply.
• the convex envelope which below is also referred to as the envelope only, is approximated with a number of planes forming a polygon.
• the number of planes varies significantly with the application, but, as an example, it may be mentioned that 50D60 planes function well in the application involving writing on a position-coded sheet of paper that has been described above.
• the planes are preferably rotated round a common axis so that the angle between two arbitrary planes, which are located close to each other in terms of angle, is the same for all planes. In other words, the angles for the normals of the planes are uniformly distributed over an entire revolution.
• the natural electronic representation of an inputted character is a sequence of positions, as illustrated with dots in the Figure, since some kind of sampling takes place and the sampling frequency is not infinite.
• the term dot will below be used as an equivalent of position.
• the method of determining the convex envelope, or, if desirable, the extent of the convex envelope operates on the marking stroke 21 dot by dot. In accordance with that described above, the location of each dot is compared with the location of each of the planes 11. If the dot is positioned on the outside of the plane, the plane is moved in parallel so that the dot will get into the plane.
• Fig. 2c shows the result after processing of all dots in the marking stroke 21.
• the area surrounded by the envelope has been elucidated with a check pattern, that does not exist in actual use of the inventive method.
• Fig. 6 there are four planes which define the convex envelope, and three positions which are assumed to be samples of a marking stroke. The positions are illustrated for the sake of clarity with well visible dots 61, 62 and 63. In reality, the dots 61-63 have no extent in terms of definition. It is assumed that the dots 61, 62, 63 have been formed in the order as mentioned when the marking stroke was performed.
• the determination of the convex envelope is initiated, box 701, by the four planes P0, PI, P2 , P3 being initiated.
• the planes P0-P3 are distributed in terms of angle, equidistantly along a revolution round the z axis.
• the planes have been arranged at the angles PO : 45 degrees, PI: 135 degrees, P2 : 225 degrees and P3 : 315 degrees.
• the corresponding angles of the normals N0DN3 of the planes are NO: 315 degrees, Nl : 45 degrees, N2 : 135 degrees and N3 : 225 degrees.
• the normals are interesting since it is on their values that the calculations are carried out.
• the mutually parallel planes PO and P2 and respectively PI and P3 are slightly relatively offset in the normal direction. It goes without saying that in reality the planes PO and P2 and respectively the planes PI and P3 coincide. Then the first position in the marking stroke, i.e. dot 61, is identified, box 703, and all planes P0-P3 are moved, box 705, so that they include the first dot 61, see Fig. 6a. More precisely, this movement will be carried out as follows.
• a check pattern has been inserted in order to easily define and show the positions of the dots 61- 63.
• the next step in the determination of the convex envelope means that the next position, if available, is selected for processing, box 707 and box 709. In this case there is a next position in the form of the dot 62.
• a calculator, K is set to zero, box 711.
• the location of the dot 62 is the next to be related to each plane.
• box 713 use is made of the definition that a dot is located to the left of a plane and, thus, outside the convex envelope which is made up by the planes P0-P3 if the coordinates of the dot, which are inserted into the equation of the plane, give a negative result.
• the plane is then moved so that the result will no longer be negative, box 715. In this simple example, the plane is moved so that the result will be zero.
• the third dot 63 is processed, whose coordinates are (4, -1) .
• the locations of the remaining planes are unchanged.
• the resulting convex envelope is a further extended rectangle where a new corner point instead of the second dot 62 is the third dot 63, as shown in Fig. 6c.
• the next step will be to identify the remaining data, in addition to the data of the actual marking stroke 21, so that it can be established to which data the editing is to relate. Since the edge of the envelope, i.e. the polygon, is now defined, said remaining data can be determined by the processor comparing the positions on the writing surface for the stored strokes with the positions for the sides of the polygon. The comparison preferably proceeds in a way similar to the way in which the dots of the marking stroke 21 were compared with the locations of the planes 11, as was also described in more detail above in connection with Figs 6a-c.
• the previously read and stored strokes that have been performed on the writing surface are thus processed position by position and compared with the locations of the planes.
• the data set that must be processed is thus advantageously restricted to the page on which the marking is made. Further restrictions of the data set may be involved. For such a restriction, methods that are per se known are available. As an example, so called divide and conquer algorithms can be mentioned.
• the planes are, of course, not moved. If a position is located inside all planes, the dot is located within the envelope, i.e. in the check patterned area in Fig. 2d.
• Fig. 5 illustrates examples of some operations.
• the striped elongate operation box 59 illustrates a color scale which can be preprinted on the surface 51.
• the operation of selecting color could be implemented either by putting the pen 31 in box 59, or in 0 two steps, first putting the pen in the box "color" 55 to indicate the type of operation and then putting the pen in box 59 to select the shade of color.
• This operation of selecting color is intended for selecting display color for marked characters when they are shown on the screen 5 of said second device 39. Additional examples are other formattings of the marked characters, and the operation of sending data, i.e. sending precisely the data selected by the marking.
• One example of an alternative embodiment is an application to a different type of equipment for handwriting, viz. a hand-held computer or like electronic device where the user writes directly on a screen, for instance an LCD screen or a pressure-sensitive screen, 0 and what is written is stored merely electronically. In this case there is a possibility of feedback which makes it possible for the user to directly see the result of an editing.
• Further examples of other types of writing equipment are digital white boards, digital flipcharts 5 and digital notebooks. There are also other options as regards position determination of the writing device.
• the method according to the invention is, however, carried out in the same way as when using pen and paper.
• the invention also concerns three- dimensional applications.
• the method is carried out in an essentially equivalent manner.
• An example of application is in a device for virtual reality where a hand-held unit can be used to create three-dimensional products as well as edit the same by moving one's hand in the air.
• the plane equation contains a variable z normal.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• Human Computer Interaction (AREA)
• Processing Or Creating Images (AREA)
• User Interface Of Digital Computer (AREA)
• Document Processing Apparatus (AREA)
• Character Discrimination (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=20284838

Family Applications (1)

Country Status (2)

Cited By (2)

Citations (7)

• 2001
  • 2001-07-13 SE SE0102502A patent/SE519621C2/sv not_active IP Right Cessation
• 2002
  • 2002-07-05 WO PCT/SE2002/001350 patent/WO2003007145A1/en not_active Application Discontinuation

Patent Citations (7)

Also Published As

Similar Documents

Legal Events