JP6059113B2 - Communication terminal, program - Google Patents

Description

  The present invention relates to a communication terminal and a program for transmitting handwritten display by touch sensor input to another terminal.

  In recent years, systems capable of recognizing handwritten characters online (hereinafter referred to as online handwritten character recognition systems) are known. The online handwritten character recognition system collects the movement of the stylus tip (fingertip) and the raising / lowering of the stylus (finger) as stroke order information. The online handwritten character recognition system applies a recognition algorithm, performs pattern matching with a character database including a stroke order, and converts written characters into text information.

  As a method for recognizing handwritten characters to which a recognition algorithm (Hidden Markov Model) is applied, there is, for example, Patent Document 1.

JP 2002-183661 A

  An online communication is assumed in which a handwritten character bulletin board on-line is shared among a plurality of terminals, and handwritten characters input from a plurality of terminals are reflected and displayed on the bulletin board. In this case, when creating a conversation record (minutes), an application that implements a recognition algorithm is installed on one of the terminals, and stroke order information for converting handwritten characters into text on the terminal is provided. It is desirable to provide. On the other hand, in the case where a conversation record (minutes) is not created only by writing comments on the bulletin board, the above-mentioned stroke order information and application are unnecessary. As described above, the stroke order information and the necessity of character recognition usually vary depending on the situation. However, in the above-described online handwritten character recognition system, image information of handwritten characters and corresponding stroke order information must always be handled as a set, and there is a problem that the amount of information increases. In the above-mentioned online handwritten character bulletin board, it is important to be able to transmit information in real time. However, if the stroke order information is always transmitted and received in correspondence with the image information of handwritten characters, traffic increases and the real-time property is impaired. there is a possibility.

  Therefore, an object of the present invention is to provide a communication terminal capable of compressing the data size of handwritten characters to which stroke order information is added.

  The communication terminal of the present invention includes a touch sensor, an image information generation unit, a data communication unit, and a display control unit.

  The touch sensor accepts a user touch sensor input. The image information generation unit acquires the coordinates at which the touch sensor input is detected and the time information indicating the time at which the input is detected, and generates the image information by substituting the time information into the color information of the corresponding coordinates. . The data communication unit transmits image information every predetermined time to a server device that is a relay device for sharing image information among a plurality of communication terminals, and receives updated image information every predetermined time from the server device. To do. The display control unit binarizes and displays the coordinates of the transmitted and received image information.

  According to the communication terminal of the present invention, it is possible to compress the data size of handwritten characters to which the stroke order information is added.

The figure which shows the outline | summary of the handwritten display transmission system of Example 1 of this invention. The block diagram which shows the structure of the communication terminal of Example 1 of this invention. The flowchart which shows operation | movement of the communication terminal of Example 1 of this invention. The block diagram which shows the structure of the image information generation part of the communication terminal of Example 1 of this invention. The flowchart which shows operation | movement of the image information generation part of the communication terminal of Example 1 of this invention. The figure which shows the example of the handwritten character which substituted time information to RGB value. The figure which shows the outline | summary of the handwritten display transmission system of Example 2 of this invention. The block diagram which shows the structure of the communication terminal of Example 2 of this invention. The flowchart which shows operation | movement of the communication terminal of Example 2 of this invention.

  Hereinafter, embodiments of the present invention will be described in detail. In addition, the same number is attached | subjected to the structure part which has the same function, and duplication description is abbreviate | omitted.

<Definition of terms: Handwritten display>
In this specification, characters, figures, symbols and the like handwritten with a stylus, a finger or the like are collectively referred to as handwritten display.

<Definition of terms: Communication terminal>
The communication terminal is a general term for a telephone, particularly a mobile phone or a PHS telephone, or a terminal or device having a communication function.

  The outline of the handwritten display transmission system according to the first embodiment of the present invention will be described below with reference to FIG. FIG. 1 is a diagram illustrating an outline of a handwritten display transmission system according to the present embodiment. As shown in FIG. 1, the handwritten display transmission system 50 of a present Example is the structure containing the server apparatus 9 and n communication terminals 1-1, 1-2, ..., 1-n. n is an integer of 2 or more. The server device 9 and the n communication terminals 1-1, 1-2,..., 1-n can communicate data with each other through the NW8. The virtual data communication path is shown as a dashed arrow in FIG. The server device 9 is a relay device for sharing image information including handwritten display among the n communication terminals 1-1, 1-2, ..., 1-n. Since the n communication terminals 1-1, 1-2,..., 1-n all include the same functional configuration, the same functions as the n communication terminals 1-1, 1-2,. The communication terminal 1 provided will be described as a representative of these terminals.

Hereinafter, the configuration and operation of the communication terminal 1 of this embodiment will be described with reference to FIGS. FIG. 2 is a block diagram showing the configuration of the communication terminal 1 of the present embodiment. FIG. 3 is a flowchart showing the operation of the communication terminal 1 of this embodiment. As shown in FIG. 2, the communication terminal 1 according to the present embodiment includes a touch sensor 11, an image information generation unit 12, a data communication unit 13, a display control unit 14, and a character recognition unit 15. As shown in FIG. 3, the touch sensor 11 receives a user's touch sensor input. The image information generation unit 12 acquires the coordinates at which the touch sensor input is detected and the time information indicating the time at which the input is detected, and generates the image information by substituting the time information into the color information of the corresponding coordinates. (S12). The image information generation unit 12 can use RGB values as color information. Further, the image information generation unit 12 may generate image information by substituting time information for CMYK values instead of RGB values. The image information generation unit 12 can use not only RGB values and CMYK values but also other color information (color space), for example, (L ^* , a ^* , b ^* ) values. Hereinafter, in this specification, an example using RGB values as color information will be described. The data communication unit 13 transmits image information to the server device 9 every predetermined time, and receives updated image information from the server device 9 by adding handwritten display by another terminal every predetermined time (S13). The display control unit 14 binarizes and displays the coordinates of the transmitted and received image information (S14). The character recognition unit 15 acquires stroke order information based on the RGB values of the coordinates of the transmitted and received image information, and performs character recognition based on the stroke order information (S15). The character recognition unit 15 may be included in the server device 9. In this case, the character recognition unit of the server device 9 performs the stroke order information based on the RGB values of the coordinates of the image information including the handwritten display received from the n communication terminals 1-1, 1-2,. And character recognition is executed based on the stroke order information to obtain text information.

  Hereinafter, the configuration and operation of the image information generation unit 12 of the communication terminal 1 according to this embodiment will be described with reference to FIGS. 4 and 5. FIG. 4 is a block diagram illustrating a configuration of the image information generation unit 12 of the communication terminal 1 according to the present embodiment. FIG. 5 is a flowchart showing the operation of the image information generation unit 12 of the communication terminal 1 of this embodiment. As shown in FIG. 4, the image information generation unit 12 of the communication terminal 1 of this embodiment includes a stroke timer control unit 121, a stroke timer 122, a coordinate acquisition unit 123, an image information update unit 124, and an initialization unit. 125, a release timer control unit 126, and a release timer 127. As shown in FIG. 5, when a touch sensor input is detected (S12AY), the stroke timer control unit 121 starts the stroke timer 122 (S121A). When the touch sensor input is not detected (S12AN), the touch sensor input is continuously monitored. When the touch sensor input is detected (S12AY), the coordinate acquisition unit 123 acquires the coordinates at which the touch sensor input is detected (S123). The image information update unit 124 updates the image information by substituting time information obtained by converting the timer value of the stroke timer 122 into the RGB values of the coordinates at which the touch sensor input is detected (S124). When the stroke timer 122 expires (S122Y), the initialization unit 125 initializes the stroke timer 122 (S125A). In step S125A, the timer value of the stroke timer 122 returns to the initial value. Thereafter, Step S12B is executed. On the other hand, when the stroke timer 122 does not expire (S122N), the initialization unit 125 does not operate and step S12B is executed.

  When a release indicating interruption of touch sensor input is detected (S12BY), the stroke timer control unit 121 stops the stroke timer 122 (S121B). On the other hand, if no release is detected (S12BN), the process returns to step S123, and the operations after step S123 are executed again.

  When release is detected (S12BY), the release timer control unit 126 activates the release timer 127 (S126A). When the release timer 127 expires (S127Y), the initialization unit 125 initializes both the stroke timer 122 and the release timer 127 (S125B). In step S125B, the timer values of the stroke timer 122 and the release timer 127 are returned to the initial values. Thereafter, step S12C is executed. On the other hand, when the release timer 127 does not expire (S127N), the initialization unit 125 does not operate and step S12C is executed.

  When the touch sensor input is detected (S12CY), the release timer control unit 126 stops the release timer 127 (S126B). In this case, the process returns to step S121A, and the stroke timer control unit 121 activates the stroke timer 122 (S121A). Thereafter, step S123 and subsequent steps are executed again. On the other hand, when the touch sensor input is not detected (S12CN), the process returns to step S127, and the operations after step S127 are executed again.

  Hereinafter, the operation of the image information generation unit 12 will be described according to a specific example with reference to FIG. FIG. 6 is a diagram showing an example of handwritten characters in which time information is substituted into RGB values. FIG. 6 shows a Hiragana sentence “Nice to meet you” as an example of handwritten characters. It is assumed that the handwritten characters in FIG. 6 are input by handwriting according to the correct stroke order of the sentence “Nice to meet you” in Hiragana. In FIG. 6, the start point and the end point of each stroke of the handwritten character “Nice to meet you” are numbered. For example, numbers 101S and 101E are assigned to the start point and end point of the first drawing of the handwritten character “ha”, respectively. S means start and E means end. Similarly, the numbers 102S and 102E are assigned to the start and end points of the second stroke “ha”, and the numbers 103S and 103E are assigned to the start and end points of the third stroke “ha”, respectively. Similarly, numbers 104S and 104E are assigned to the start and end points of the first stroke of “Ji”, and numbers 105S and 105E are assigned to the start and end points of the second stroke of “Ji”, respectively. The numbers 106S and 106E are assigned to the start point and end point of the third stroke of “Ji”, respectively. Similarly, 107S, 107E, 108S, 108E, 109S, 109E, 110S, 110E, 111S, 111E, 112S, 112E, 113S, and 113E are numbered according to the stroke order of Hiragana “Nice to meet you”.

  In the example of FIG. 6, first, the user makes the fingertip (stylus tip) contact the coordinates of the point 101S on the touch panel (touch input detection, S12AY). In this case, the stroke timer control unit 121 activates the stroke timer 122 (S121A). The coordinate acquisition unit 123 acquires the coordinates of the point 101S where the touch sensor input is detected (S123). The image information update unit 124 is time information obtained by converting the timer value (0 because it is an initial value) of the stroke timer 122 with respect to the RGB values (R, G, B) of the coordinates of the point 101S (0, 0, 0) is substituted to update the image information (S124). Next, after confirming that the stroke timer has not expired in step S122 (S122N) and that release is not detected in step S12B (S12BN), the coordinate acquisition unit 123 again determines the coordinates from which the touch sensor input has been detected. Obtain (S123). In this example, the coordinate acquisition unit 123 continuously acquires the coordinates from the point 101S to the point 101E without detecting the release. Similarly, the image information update unit 124 updates the image information by substituting time information obtained by converting the timer value of the stroke timer 122 into the RGB values of the coordinates from the point 101S to the point 101E (S124). . For example, when the user's fingertip (tip of the stylus) is in contact with the coordinates of the point 101E on the touch panel, the coordinate acquisition unit 123 acquires the coordinates of the point 101E where the touch sensor input is detected (S123). The image information updating unit 124 substitutes (0, 0, 9), which is time information obtained by converting the timer value of the stroke timer 122 with respect to the RGB values (R, G, B) of the coordinates of the point 101E. The image information is updated (S124). Next, since the user releases the fingertip (tip of the stylus) from the touch panel from the end point 101E of the first screen of “ha” to the start point 102S of the second screen of “ha” (S12BY), the stroke timer The control unit 121 stops the stroke timer 122 (S121B), and the release timer control unit 126 starts the release timer 127 (S126A). After confirming that the release timer has not expired (S127N) in step S127, the process proceeds to step S12C.

  When the user again brings the fingertip (stylus tip) into contact with the coordinates of the point 102S on the touch panel (touch input detection, S12CY), the release timer control unit 126 stops the release timer 127 (S126B). The process returns to step S121A, the stroke timer control unit 121 starts the stroke timer 122 (S121A), and the coordinate acquisition unit 123 acquires the coordinates of the point 102S where the touch sensor input is detected (S123).

  Since the stroke timer 122 is stopped during release detection, the time information is not updated. Accordingly, the image information updating unit 124 determines the coordinates of the end point 101E of the first image of “ha” with respect to the RGB values (R, G, B) of the coordinates of the start point 102S of the second image of “ha”. The image information is updated by substituting the same time information (0, 0, 9) (S124). In this way, the image information updating unit 124 substitutes time information obtained by converting the timer value of the stroke timer 122 for the RGB values (R, G, B) of the coordinates constituting the handwritten character “Nice to meet you”. Then, the image information is updated (S124). In this way, the image information including the handwritten character “Nice to meet” finally obtained is generated so as to include the stroke order information (time information) of the handwritten character “Nice to meet” as grayscale information (RGB information).

  Note that, depending on the type of characters, such as hiragana “ha” and “ma” appearing in the example of FIG. 6, some strokes intersect, and the old time information is overwritten with new time information at the intersected coordinates. There is a case. In this case, old time information may be preferentially stored and overwriting may be prohibited. Specifically, for coordinates where (R, G, B) = (255, 255, 255), it is possible to write shading information indicating stroke order information (time information), and in all other cases, writing is performed. It may be impossible.

  Since the RGB values (R, G, B) of an image are usually variable from (0, 0, 0) to (255, 255, 255), they can be used as a data area of 8 + 8 + 8 = 24 bits. Of course, white (255, 255, 255) of the RGB values is a data area necessary for distinguishing the blank area from the character area, and cannot be used as an area to which time information is assigned exceptionally. . Accordingly, 16777215 patterns obtained by subtracting 1 from 24 bits = 16777216 patterns can be used as an area for substituting time information. For example, when the timer value of the stroke timer 122 is updated in units of ms, and the timer value (ms) of the stroke timer 122 is used as time information without being compressed, 16777215 (ms) ≈16777 seconds≈280 minutes Therefore, the time information can be recorded practically sufficiently in a system that transmits handwritten display like the present system.

  For example, when the system is implemented to transmit / receive only monochrome data, normally R = G = B is set, and the data amount of image information transmitted / received is compressed. In this case, the RGB values (R, G, B) as the shade information of the image are usually (0, 0, 0), (1, 1, 1), (2, 2, 2),. 255, 255) up to 255, 255), and 254, excluding white (255, 255, 255), can be used as an area for substituting time information. In this case, as described above, when the timer information (ms) of the stroke timer 122 is used as time information without being compressed, the time information can be recorded only for about 254 (ms) = 0.254 seconds. In this case, for example, if the timer value of the stroke timer 122 is 15 fps (updated 15 times per second) and converted into time information, time information of about 254 / 15≈17 seconds can be recorded. For example, if the timer value of the stroke timer 122 is converted to time information with 60 fps (60 updates per second), time information of about 254 / 60≈4.2 seconds can be recorded. Also, for example, if the timer value (ms) of the stroke timer 122 is updated once every 10 ms and converted into time information, time information of about 254 × 10 = 2540 (ms) ≈2.5 seconds can be recorded. it can.

<Initialization of stroke timer 122>
As described above, 24 bits are sufficient as the data area of the RGB values of each coordinate for recording time information, but 8 bits require information compression as appropriate when converting from timer values to time information. Become. Here, if the stroke timer is initialized at an appropriate timing and the timer value of the stroke timer is returned to 0, the time information can be sufficiently obtained even when the RGB value (R, G, B) is 8 bits. It can be stored. For example, consider a case where the expiration value of the release timer 127 is 1000 ms, the expiration value of the stroke timer is 2540 ms, and the timer value (ms) of the stroke timer 122 is updated once every 10 ms and converted to time information. In step S122Y (expiration of stroke timer 122), the initialization unit 125 initializes the timer value of the stroke timer to 0 (S125A). As a result, a data break in which the RGB information becomes (254, 254, 254) → (0, 0, 0) in the middle of the character occurs every time character input continues for 2.5 seconds. Even when such a data break occurs, it is only necessary to modify the application side that reads the stroke order information from the time information. The stroke order information is lost or deteriorated due to the initialization of the stroke timer 122. Absent. Similarly, in the case of step S127Y (expiration of release timer 126), the timer value of the stroke timer is similarly reset to 0 (S125B). The reason why the expiration value of the release timer 127 is set to 1000 ms is that, for example, “Nice to meet you”, “Thank you”, “Now in Shibuya at 19:00”, “What do you do tomorrow?” This is based on an empirical fact that an interval until a next sentence is written after a sentence is written, that is, a release time is often about 1000 ms or more = 10 seconds or more. Therefore, as described above, a data delimitation in which the RGB information becomes (254, 254, 254) → (0, 0, 0) occurs. In many cases, this data delimitation is performed after the short single sentence is written. It often occurs at the beginning of the next sentence written across the release time. Therefore, even in this case, the stroke order information is not lost or deteriorated due to the initialization of the stroke timer 122, and the stroke order information can be acquired for each segment of text, which improves convenience.

  As described above, according to the communication terminal 1 of the present embodiment, the image information can be generated so as to include the stroke order information of the handwritten characters as the shade information, and thus the data size of the handwritten characters including the stroke order information is compressed. be able to.

  The outline of the handwritten display transmission system according to the second embodiment of the present invention will be described below with reference to FIG. FIG. 7 is a diagram showing an outline of the handwritten display transmission system of the present embodiment. As illustrated in FIG. 7, the handwritten display transmission system 250 according to the present embodiment includes the server device 9 and n communication terminals 2-1, 2-2,. The server device 9 and the n communication terminals 2-1, 2-2,..., 2-n can communicate data with each other through the NW 8. The virtual data communication path is shown as a dashed arrow in FIG. The server device 9 is a relay device for sharing image information including handwritten display among the n communication terminals 2-1, 2-2,. Since the n communication terminals 2-1, 2-2,..., 2-n all include the same functional configuration, the same functions as the n communication terminals 2-1, 2-2,. The communication terminal 2 provided will be described as a representative of these terminals.

  Hereinafter, the configuration and operation of the communication terminal 2 according to the present embodiment will be described with reference to FIGS. 8 and 9. FIG. 8 is a block diagram showing the configuration of the communication terminal 2 of this embodiment. FIG. 9 is a flowchart showing the operation of the communication terminal 2 of this embodiment. As shown in FIG. 8, the communication terminal 2 according to the present embodiment includes a touch sensor 11, an image information generation unit 12, a data communication unit 13, a display control unit 24, and a character recognition unit 15. The only difference between the communication terminal 1 of the first embodiment and the communication terminal 2 of the present embodiment is that the display control unit 14 in the first embodiment is changed to the display control unit 24 in the present embodiment. Hereinafter, only step S24, which is the operation of the display control unit 24, will be described.

  In step S24, the display control unit 24 acquires the stroke order information based on the RGB values of the coordinates of the transmitted and received image information, and converts the binarized image information into moving image information that changes according to the stroke order information. Are displayed (S24). Here, the moving image information that changes in accordance with the stroke order information is, for example, moving image information in which binarized handwritten characters (image information) are sequentially added and displayed one by one in accordance with the stroke order information.

  As described above, according to the communication terminal 2 of the present embodiment, the image information binarized by the display control unit 24 is converted into moving image information that changes according to the stroke order information and displayed. By displaying the state of writing each image as a moving image on each user's terminal, it becomes possible to provide a realistic handwritten display transmission service, and the convenience of the user is enhanced.

  The various processes described above are not only executed in time series according to the description, but may also be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes. Needless to say, other modifications are possible without departing from the spirit of the present invention.

  Further, when the above-described configuration is realized by a computer, processing contents of functions that each device should have are described by a program. The processing functions are realized on the computer by executing the program on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing the process, the computer reads a program stored in its own recording medium and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to the computer. Each time, the processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good.

  Note that the program in this embodiment includes information that is used for processing by an electronic computer and that conforms to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer). In this embodiment, the present apparatus is configured by executing a predetermined program on a computer. However, at least a part of these processing contents may be realized by hardware.

Claims (5)

A touch sensor that accepts user touch sensor input;
An image information generation unit that obtains coordinates at which the touch sensor input is detected and time information indicating the time at which the input is detected, and generates image information by substituting the time information into color information of the corresponding coordinates. When,
Data that transmits the image information every predetermined time to a server device that is a relay device for sharing the image information among a plurality of communication terminals, and receives updated image information every predetermined time from the server device A communication department;
A communication terminal including a display control unit that binarizes and displays each coordinate of the image information transmitted and received. The communication terminal according to claim 1,
A communication terminal including a character recognition unit that acquires stroke order information based on color information of each coordinate of the transmitted and received image information and performs character recognition based on the stroke order information. The communication terminal according to claim 1,
The display control unit
A communication terminal that obtains stroke order information based on color information of each coordinate of the transmitted and received image information, converts the binarized image information into moving image information that changes according to the stroke order information, and displays the converted image information. The communication terminal according to claim 1,
The image information generation unit
A stroke timer control unit that starts a stroke timer when the touch sensor input is detected, and stops the stroke timer when a release that means interruption of the touch sensor input is detected;
A coordinate acquisition unit that acquires coordinates at which the touch sensor input is detected;
A communication terminal including an image information update unit that updates image information by substituting time information obtained by converting a timer value of the stroke timer into color information of coordinates at which the touch sensor input is detected.   A program for causing a computer to function as the communication terminal according to claim 1.