JP2012018544A - Audio output device, audio output method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an audio output device, an audio output method and a program to solve a problem in which it is difficult for audio output devices to output recognition results as audio with expression.SOLUTION: An audio output device includes: an input unit for inputting a series, from the beginning to the end, of handwriting; a recognition unit for recognizing the shape and the size of the handwriting inputted by the input unit; a classification unit for classifying the handwriting recognized by the recognition unit into a category based on the numbers of the strokes that constitute the handwriting and the size of the handwriting; a selection unit for selecting a onomatopoeic word or a mimetic word associated with the category specified by the classification unit from onomatopoeic words and mimetic words prestored on a storage unit; and an output unit for outputting the onomatopoetic word or the mimetic word selected by the selection unit as audio.

Description

  The present invention relates to an audio output device, an audio output method, and a program that output audio in accordance with user input.

  An apparatus that outputs sound in accordance with a user's handwriting input is known. Patent Document 1 discloses an apparatus that outputs pseudo sound. This is to calculate the moving speed of the pen input and to output a pseudo sound corresponding to the moving speed. This has the effect of giving the user the feeling of writing something on paper. Patent Document 2 discloses an apparatus that outputs not only a character recognition result but also a word recognition result together with voice output. This has the effect of giving the user a sense of input as words and sentences.

  Also known are devices that output onomatopoeic words instead of letters and words. Patent Document 3 discloses an apparatus that synthesizes an onomatopoeia corresponding to an input sound volume as a character with a captured image. There is an effect that it is possible to visually give a sound to the user.

JP-A-8-190450 JP 2002-288588 A JP 2006-109322 A

  In a device having a function of recognizing characters and gestures and a sound output function, a function of outputting the result recognized by the device is useful. Further, in order to attract the user's interest, it is desirable that the output voice is rich in expression by using onomatopoeia (onomatopoeia or mimicry word).

  However, it is difficult to confirm the recognition result of the device from the sound output by the device disclosed in Patent Document 1. Also, the sound output by the device disclosed in Patent Document 2 is monotonous. The apparatus disclosed in Patent Document 3 does not consider a method for associating handwritten input with onomatopoeia. That is, conventionally, there is a problem that it is difficult to output the recognition result of the voice output device that outputs a handwritten character or the like as voice in an expressive manner.

  In view of the above-described problems, an object of the present invention is to express a recognition result of handwritten characters and the like in an expressive manner.

An audio output device according to the present invention that achieves the above object is as follows.
An input means for inputting a series of handwriting from the beginning of drawing to the end of drawing,
Recognizing means for recognizing the shape and size of the handwriting input by the input means;
Classification means for classifying the handwriting recognized by the recognition means into categories for each number of line segments constituting the handwriting and the size of the handwriting,
A selection means for selecting an onomatopoeia or mimicry word corresponding to the category classified by the classification means from an onomatopoeia or mimicry word stored in advance in the storage means;
Output means for outputting the onomatopoeia or mimicry word selected by the selection means as speech;
It is characterized by providing.

  According to the present invention, it is possible to express a recognition result of handwritten characters and the like in a richly expressive manner.

(A) The block diagram which shows the hardware constitutions which concern on the audio output device, (b) The block diagram which shows the functional constitution of the audio output device. The figure which shows the example of the stroke dictionary 152 (A) The figure which shows the example of the onomatopoeia dictionary 153, (b) The figure which shows the example of the onomatopoeia dictionary 153. The figure which shows the example of the gesture dictionary 161. FIG. The flowchart which shows the process sequence of an audio | voice output apparatus. The figure which shows the operation example of an audio | voice output apparatus. The flowchart which shows the process sequence for recognizing the shape of a stroke. (A) The figure explaining the vertex of a stroke, (b) And (c) The figure which shows the example of the input gesture. The flowchart which shows the process sequence which changes the stroke wording into the other stroke wording simplified. The flowchart which shows the process sequence for carrying out the audio | voice output of how to write the gesture correctly.

(First embodiment)
With reference to FIG. 1A, the hardware configuration of the audio output device according to the present invention will be described. A CPU (Central Processing Unit) 101 controls the operation of the entire apparatus as a system control unit. The ROM 102 stores a control program. Specifically, a program for performing processing to be described later is stored. The RAM 103 provides a work area for the CPU 101 and is used for storing various data. The storage device 104 is used for storing image data and the like. Specifically, it is an SD card or a hard disk (HDD). The touch panel 105 is used for processing an input with a finger or a pen. The speaker 106 is used for outputting sound.

  With reference to FIG.1 (b), the block diagram which shows the function structure of an audio | voice output apparatus is demonstrated. The voice output device includes an input unit 151, a stroke dictionary 152, an onomatopoeia dictionary 153, a stroke processing unit 154, a stroke recognition unit 155, a stroke classification unit 156, an onomatopoeia selection unit 157, and a voice output unit 158. Further, a stroke holding unit 159, a stroke word acquisition unit 160, a gesture dictionary 161, a gesture processing unit 162, a gesture recognition unit 163, a gesture word acquisition unit 164, a gesture execution unit 165, and a help unit 166 are provided.

  The input unit 151 receives input from the user. The stroke dictionary 152 holds information on strokes (handwriting) that can be recognized by the audio output device and is input information from the user. A stroke (handwriting) is a series of operations from when the user touches the touch panel 105 until the user leaves. The onomatopoeia dictionary 153 classifies and records the shape of the input stroke for each category using the number of line segments constituting the stroke. Onomatopoeia is a word that comprehensively represents onomatopoeia and mimicry words. The stroke processing unit 154 supervises processing related to the stroke. The stroke recognition unit 155 recognizes the stroke shape and the stroke size in the input area such as a touch panel from the input coordinate sequence sent from the stroke processing unit 154. The stroke classification unit 156 classifies the input strokes into categories using the shape and size recognized by the stroke recognition unit 155. The onomatopoeia selection unit 157 selects an onomatopoeia corresponding to the classified category from the onomatopoeia dictionary 153. The voice output unit 158 outputs the onomatopoeia, the later-described stroke wording (first wording), the later-described gesture wording (second wording), or any other wording.

  The stroke holding unit 159 holds a column of input coordinates. The stroke word acquisition unit 160 functioning as a first acquisition unit acquires a stroke word (first word) corresponding to the recognized shape from the stroke dictionary 152. The gesture dictionary 161 holds gesture information (graphics) composed of one or more strokes. The gesture processing unit 162 controls processing related to gestures (processing corresponding to graphics). The gesture recognition unit 163 recognizes a gesture (figure) using the input coordinate sequence sent from the gesture processing unit 162. The gesture word acquisition unit 164 functioning as the second acquisition unit acquires a gesture word (second word) corresponding to the gesture recognized by the gesture recognition unit 163 from the gesture dictionary 161. The gesture execution unit 165 executes a command corresponding to the recognized gesture. The help unit 166 acquires the correct way of writing the gesture when the likelihood of the recognized gesture is small.

  Hereinafter, the processing in each of the processing units will be specifically described. The input unit 151 includes a touch panel 105. When the user touches the touch panel 105 with a finger or a pen, the input unit 151 detects the input coordinates of the touched position and generates a pen event. There are three types of pen events to be generated: penDown, penMove, and penUp. First, when the user's finger touches the touch panel 105, the input unit 151 generates penDown. When the finger moves while touching the touch panel 105, penMove is generated. Finally, when the finger is removed from the touch panel 105, penUp is generated. The input coordinates detected in the pen event are held by a stroke holding unit 159 described later. It is also possible to use a mouse instead of a finger or pen for input. In that case, the input unit 151 may not include the touch panel 105.

  The stroke processing unit 154 receives a pen event generated by the input unit 151 and detects a stroke break. A stroke from when the user touches the touch panel 105 until the user leaves is defined as one stroke. That is, one stroke is an event sequence starting with one penDown, followed by one or more penMove, and ending with one penUp. When the stroke processing unit 154 detects a stroke break, the stroke processing unit 154 extracts input coordinates from each pen event, and sends a string of input coordinates to the stroke recognition unit 155.

  An example of the stroke dictionary 152 will be described with reference to FIG. The stroke dictionary 152 holds stroke information that can be recognized by the apparatus. In the stroke dictionary 152 shown in FIG. 2, seven types of strokes are registered for the shape name, the reference stroke, and the corresponding stroke wording. Specifically, there are seven types: horizontal line 201, vertical line 202, less than 203, right triangle 204, left triangle 205, square 206, and upper semicircle 207.

  The stroke recognition unit 155 recognizes the shape and size of the stroke from the input coordinate sequence sent from the stroke processing unit 154 and the information registered in the stroke dictionary 152. First, a method for recognizing the stroke size will be described. A method of recognizing the stroke shape will be described later.

  The stroke size is recognized by the following method. First, a rectangle circumscribing the input coordinate column is obtained. If the width of the rectangle is half or more of the width of the touch panel or the height of the rectangle is more than half of the height of the touch panel, the stroke size is recognized as “large”. Conversely, when the width and height of the rectangle are both less than half the height and width of the touch panel, the stroke size is recognized as “small”.

  The stroke classification unit 156 classifies the input strokes into categories using the shape and size recognized by the stroke recognition unit 155. The classification method follows the classification registered in the onomatopoeia dictionary 153. The onomatopoeia selection unit 157 selects an onomatopoeia corresponding to the classified category from the onomatopoeia dictionary 153. The selected onomatopoeia is sent to the audio output unit 158.

  An example of the onomatopoeia dictionary 153 will be described with reference to FIG. The onomatopoeia dictionary 153 in FIG. 3A corresponds to the stroke dictionary 152 in FIG. That is, all the shapes included in the stroke dictionary 152 of FIG. 2 are included. The shapes of “unknown 1”, “unknown 2”, “unknown 3”, and “unknown 4” included in the shape of the stroke in FIG. 3A will be described later. The onomatopoeia dictionary 153 has the following features so that the user can confirm the recognition result of the voice output device for the stroke input by the user from the onomatopoeia.

  In the onomatopoeia dictionary 153, shapes are classified into categories by using the number of line segments constituting the stroke. In the example of FIG. 3A, the shapes are classified into four categories. Each category includes one, two, three, and four shapes that form a stroke. The shapes of “horizontal line 201”, “vertical line 202”, “upper semicircle 207”, and “unknown 1” are included in one category with the number of line segments. Similarly, “less than 203” and “unknown 2” include the number of line segments in two categories. “Right-facing triangle 204”, “left-facing triangle 205”, and “unknown 3” are included in the three categories of line segments. “Square 206” and “Unknown 4” include the number of line segments in four categories. Assign different onomatopoeia for each classified category. By assigning onomatopoeia in this way, the following effects can be obtained. The number of line segments constituting the stroke is closely related to the shape of the stroke. Therefore, the recognized stroke shape can be imagined from onomatopoeia. Therefore, the user can confirm the shape of the recognized stroke by listening to the onomatopoeia that is output as voice.

  The onomatopoeia dictionary 153 classifies stroke sizes into categories. In the example of FIG. 3A, the data is classified into two categories, “large” and “small”. Assign different onomatopoeia for each category. By assigning onomatopoeia in this way, the following effects can be obtained. The recognized stroke size can be imagined from onomatopoeia. Therefore, the user can confirm the size of the recognized stroke by listening to the onomatopoeia that is output as voice.

  In the above description, the shape and size of the stroke are classified into categories, but it is also possible to classify only the shapes into categories. For example, when using a gesture that does not distinguish the size, it is not necessary to recognize the size of the stroke. The gesture will be described later.

  The onomatopoeia allocation method will be described in more detail using the example shown in FIG. First, a method for assigning onomatopoeia for each category into which shapes are classified will be described. Increase or decrease the number of beats of onomatopoeia according to the number of line segments that make up the stroke. The number of beats is a number in which beats that are the basis of onomatopoeia (such as “sa” or “su” described later) are consecutive. For example, when the stroke size is “small”, it is as follows. The basic beat is “sa”. Repeat the basic beat as many as the number of line segments that make up the stroke, and then add the beat “tsu” to make the onomatopoeia. When the number of line segments constituting the stroke is 2, “sasa”, which is obtained by repeating “sa” twice and adding “tsu”, is defined as onomatopoeia. However, the number of basic beats and the number of line segments constituting the stroke are not necessarily the same. For example, the same onomatopoeia can be assigned to a stroke having five or more line segments constituting the stroke. When the basic beat number of onomatopoeia is equal to the number of line segments constituting the stroke, the shape of the stroke can be easily imagined from the onomatopoeia that is output as voice.

  Next, a method for assigning onomatopoeia for each category in which the sizes are classified will be described. The basic beat is different according to the size of the stroke. For example, a basic beat for a stroke having a size of “small” is “sa”. A basic beat for a stroke having a size of “large” is defined as “su”. As a result, the recognized stroke size can be easily confirmed. Moreover, it is good to make the beat added after repeating a basic beat differ. For example, a stroke having a size of “small” adds one beat “t”. The stroke of “large” is added with 3 beats “ぅ ～ っ”. When the number of line segments composing the stroke is 2 and the size is “large”, “Susuzu ~ tsu” is set as the onomatopoeia. The onomatopoeia for the stroke of “large” has a higher number of beats than the onomatopoeia for the stroke of “small”. The user can easily imagine the size of the stroke from the onomatopoeia that outputs voice.

  Here, an example using an onomatopoeia as an onomatopoeia has been described. It is also possible to use a mimetic word instead of the onomatopoeia. For example, the categories are classified into strokes that include arcs and strokes that do not include arcs. The onomatopoeia for a stroke including an arc is the mimetic word “fluffy” or “fluffy”. Here, “fluff” is a case where the size is “small” and “fluffy” is a case where the size is “large”. The same onomatopoeia as in FIG. 3A is assigned to a stroke that does not include an arc.

  As described above, onomatopoeia such as onomatopoeia and mimetic words can be assigned to the shape and size of the stroke. The shape and size of the stroke can be easily imagined from the assigned onomatopoeia. Therefore, the user can confirm the shape and size of the recognized stroke by listening to the onomatopoeia that is output as voice. In addition, the use of onomatopoeia makes it possible to output an expressive voice. FIG. 3B is a diagram showing an example of the onomatopoeia dictionary 153 according to the second embodiment, which will be described in the second embodiment to be described later.

  The stroke holding unit 159 holds a column of input coordinates extracted from the pen event. The stroke word acquisition unit 160 acquires a stroke word corresponding to the recognized shape from the stroke dictionary 152. The stroke wording is a wording explaining the shape of the recognized stroke. Here, some correction may be added to the acquired stroke wording. The acquired stroke wording is sent to the audio output unit 158.

  On the other hand, the gesture processing unit 162 detects a gesture break for a plurality of input strokes. The detection method will be described later. One gesture is composed of one or a plurality of strokes.

  An example of the gesture dictionary 161 will be described with reference to FIG. For example, a “four-sheet display” gesture is composed of two strokes of “horizontal line 201” and “vertical line 202”. Further, the “single display” gesture is composed of one stroke of “square 206”. A column of input coordinates corresponding to each stroke included between the detected gesture breaks is extracted from the stroke holding unit 159. The extracted input coordinate sequence is sent to the gesture recognition unit 163.

  The example of the gesture dictionary 161 shown in FIG. 4 will be supplemented. The dotted rectangles included in the “four-sheet display” and “one-sheet display” gestures are guides representing the position and size of the touch panel. These gestures need to be input at a position and size according to the guide. Note that a gesture without a guide is input small at an arbitrary position on the touch panel.

  The gesture recognition unit 163 recognizes a gesture using the input coordinate sequence sent from the gesture processing unit 162. The gesture recognition method will be described later. The gesture word acquisition unit 164 acquires a gesture word corresponding to the recognized gesture from the gesture dictionary 161. Here, it is possible to add some correction to the extracted gesture wording. The acquired gesture wording is sent to the voice output unit 158. The gesture execution unit 165 executes a command corresponding to the recognized gesture. The help unit 166 acquires the correct writing method of the gesture when the likelihood of the recognized gesture is smaller than the threshold. Details of the likelihood will be described later.

  The voice output unit 158 outputs the onomatopoeia, stroke wording, gesture wording, or other wording sent as voice. Various methods can be used for audio output. For example, speech synthesis can be used. Also, sound corresponding to onomatopoeia or the like may be recorded in advance, and the recorded sound may be reproduced. For onomatopoeia, sound effects that can be imagined from the onomatopoeia can be recorded in advance. It is also possible to store MIDI data corresponding to onomatopoeia or the like and perform the MIDI data using a synthesizer or the like.

  The above is the description of the operation of each processing unit.

  Next, with reference to FIG. 5, a flowchart illustrating a processing procedure of the audio output device according to the present embodiment will be described. FIG. 6 is a diagram illustrating an operation example of the handwriting input device of the present embodiment, and is used to supplement the description of the flowchart of FIG.

  In step S501, the stroke processing unit 154 determines whether a penDown pen event has been received. If received (step S501; YES), the process proceeds to step S502. If not received (step S501; NO), the process proceeds to step S513.

  In step S502, the stroke processing unit 154 extracts input coordinates from the received penDown pen event. The stroke word acquisition unit 160 determines a rough position of the extracted input coordinates on the touch panel, and acquires a word corresponding to this position. For example, in the case of the vicinity of the left end of the touch panel, the wording “From Hidari no Hashi” is acquired as indicated by ID2 in FIG. It is assumed that a corresponding wording is held in advance for each of the rough positions on the touch panel. The acquired wording is sent to the audio output unit 158.

  In step S503, the voice output unit 158 outputs the sent word as voice. A penDown pen event is generated at the beginning of stroke writing. Here, the wording indicating the stroke start position recognized by the voice output device is outputted as voice. The user can confirm the writing start position recognized by the apparatus by listening to the output voice. The same applies to the case of ID6 in FIG.

  In step S504, the stroke processing unit 154 extracts input coordinates from the pen event. The extracted input coordinates are held by the stroke holding unit 159. In step S505, the stroke processing unit 154 determines whether or not a pen Move pen event has been received. If received (step S505; YES), the process returns to step S504. If not received (step S505; NO), the process proceeds to step S506.

  In step S506, the stroke processing unit 154 determines whether or not a penUp pen event has been received. If received (step S506; YES), the process proceeds to step S507. If not received (step S506; NO), the process returns to step S505. A penUp pen event is a stroke break. That is, the pen event sequence received in steps S501 to S506 is one stroke.

  In step S507, the stroke recognition unit 155 recognizes the shape and size of the input stroke. In step S508, the stroke classification unit 156 classifies the input strokes into categories using the recognized shape and size. In step S509, the onomatopoeia selection unit 157 selects an onomatopoeia corresponding to the classified category from the onomatopoeia dictionary 153. The selected onomatopoeia is sent to the audio output unit 158. It is also possible to select a simplified onomatopoeia when strokes are input continuously. For example, a simple onomatopoeia is obtained by reducing the number of basic beats.

  In step S <b> 510, the stroke word acquisition unit 160 acquires a stroke word corresponding to the recognized stroke shape from the stroke dictionary 152. The acquired wording is sent to the audio output unit 158.

  In step S511, the voice output unit 158 outputs the sent onomatopoeia as voice. This corresponds to the scene of ID4 in FIG. Next, the voice output unit 158 outputs the sent stroke word as voice. This corresponds to the scene of ID5 in FIG. The scene of ID5 is an example in which the stroke word acquisition unit 160 modifies the extracted word. In order to connect the onomatopoeia and the stroke wording without a sense of incongruity, the particle "to" is inserted at the beginning of the wording. Here, the result recognized by the apparatus is output by voice with respect to the input stroke. Onomatopoeia and stroke wording are sent separately to the voice output unit 158, as in “Su-u” and “To-yoko-bo”. They are not sent together like "Su-u, yokobo-". This is for facilitating processing to avoid redundant audio output, which will be described later. In the scene of ID8 in FIG. 6, both onomatopoeia and stroke wording are output. This is just a collection of the audio output divided into one scene.

  Next, in step S512, measurement by a timer is started. This timer is used to detect gesture breaks.

  In step S513, the gesture processing unit 162 determines whether a certain time has elapsed after the start of measurement by the timer. If the predetermined time has elapsed (step S513; YES), the process proceeds to step S514. If the predetermined time has not elapsed (step S513; NO), the process proceeds to step S501. If a certain time has elapsed after the last stroke is input, it is determined that the gesture is separated. One or more strokes input so far are collectively determined as one gesture.

  In step S514, measurement by the timer is stopped. In step S515, the gesture recognition unit 163 recognizes the input gesture. In step S <b> 516, the gesture word acquisition unit 164 acquires a gesture word describing a command corresponding to the recognized gesture from the gesture dictionary 161. The acquired gesture wording is sent to the voice output unit 158.

  In step S517, the voice output unit 158 outputs the received gesture text as a voice. This step corresponds to the scene of ID9 in FIG. Here, the result recognized by the apparatus is output as voice for the input gesture. After the onomatopoeia and stroke wording relating to a plurality of input strokes are output as voices, the gesture wording is output as voices.

  In step S518, the gesture execution unit 165 executes a command corresponding to the recognized gesture. In the example of FIG. 6, the “four-sheet display” gesture is recognized. Therefore, in the scene of ID10, a command for changing the screen display to the four-screen display is executed.

  Through the above processing, for example, the following voice is output in accordance with the user's gesture input as shown in FIG. “From Hidari no Hashi. Suu ~ tsutoyokobo ~. This is from the top. This is a voice explaining how to write a “4-sheet display” gesture. The writing is output as a gesture. Therefore, the user hears how to write the gesture many times, and the writing is easily fixed in the memory. Also, if you add a melody to the word and output it as a voice, you can learn how to write gestures in a fun way of drawing songs.

  Next, the processing procedure for recognizing the stroke shape in step S507 will be described in detail with reference to the flowchart of FIG. In step S <b> 701, the stroke recognition unit 155 performs matching between the input coordinate sequence corresponding to the input stroke and the reference stroke of the recognition candidate, and obtains the likelihood for each recognition candidate. A shape included in the stroke dictionary 152 is a recognition candidate. The reference stroke is data indicating how to write a model stroke. This is stored in the form of vector data or coordinate data.

  In step S702, it is determined whether or not the maximum likelihood value obtained in step S701 is equal to or greater than a predetermined threshold value. If it is equal to or greater than the threshold (step S702; YES), the process proceeds to step S706. On the other hand, when smaller than a threshold value (step S702; NO), it progresses to step S703. When the maximum likelihood value is smaller than the threshold value, it is determined that the input stroke does not match the stroke registered in the stroke dictionary 152. That is, that the maximum likelihood value is smaller than the threshold value means that the stroke shape has failed to be recognized.

  In step S703, the vertex of the input stroke is obtained using a known technique. For example, there are two vertices of the stroke shown in FIG. In step S704, the input stroke is divided into line segments at the vertex positions, and the number n of line segments constituting the stroke is obtained. In step S705, “unknown n” is used as the recognition result as the shape. In step S706, the recognition candidate having the maximum likelihood is set as the recognition result.

  The above processing has the following effects. Even if the shape recognition using the stroke dictionary 152 fails, a shape corresponding to the number of line segments constituting the stroke can be the result. Further, as shown in FIG. 3A, the onomatopoeia dictionary 153 classifies shapes into categories using the number of line segments constituting the stroke. Therefore, even when the input stroke shape is unsuccessful, it is possible to select and output the onomatopoeia on the basis consistent with the successful case.

  With reference to the flowchart of FIG. 9, the processing procedure for outputting the onomatopoeia and stroke word in step S511 of FIG. 5 will be described in detail. In step S901, the stroke recognizing unit 155 determines whether or not the stroke input this time is equal to the stroke input immediately before (previous). If the two strokes have the same shape and size, it is determined that the two strokes are equal. If equal (step S901; YES), the process proceeds to step S902. If they are not equal (step S901; NO), the process proceeds to step S909. In step S902, the audio output unit 158 determines whether or not the onomatopoeia related to the previous (previous) stroke is being output as audio. If the sound is being output (step S902; YES), the process proceeds to step S903. If the sound is not being output, that is, if the sound output has been completed (step S901; NO), the process proceeds to step S906.

  In step S903, the voice output unit 158 cancels the voice output of the stroke text related to the previous (previous) stroke. Further, the stroke word acquisition unit 160 acquires a simplified stroke word corresponding to the shape of the previous (previous) stroke and sends it to the voice output unit 158. That is, with respect to the previous (previous) stroke, the stroke word acquired in step S510 is changed to a simplified stroke word and the sound is output. Simplified stroke wording is retained in advance. For example, a simplified stroke word corresponding to the shape “vertical line 202” is “vertical”.

  In step S904, the voice output unit 158 cancels the onomatopoeia voice output related to the current stroke. In step S905, the voice output unit 158 cancels the voice output of the stroke word relating to the current stroke. Further, the stroke word acquisition unit 160 acquires a simplified stroke word corresponding to the shape of the current stroke and sends it to the voice output unit 158. That is, for the current stroke, the stroke word acquired in step S510 is changed to a simplified stroke word, and the sound is output.

  In step S <b> 906, the voice output unit 158 determines whether or not a voice message for the stroke word relating to the previous (previous) stroke is being output. If the sound is being output (step S906; YES), the process proceeds to step S907. If the sound is not being output, that is, if the sound output has been completed (step S906; NO), the process proceeds to step S909. In step S907, the voice output unit 158 cancels the onomatopoeia voice output related to the current stroke. In step S908, the voice output unit 158 cancels the voice output of the stroke word relating to the current stroke. Moreover, the stroke word acquisition part 160 acquires the word showing a repetition, and sends it to an audio | voice output part. That is, for the current stroke, the stroke word acquired in step S510 is changed to a word indicating repetition and output as a voice. Words indicating repetition are retained in advance. For example, “2 to” is a word representing repetition.

  In step S909, the onomatopoeia and stroke word relating to the current stroke are output as voice. When a stroke having a shape of “vertical line 202” and a size of “small” is input twice in succession, a voice “That, Tatebo. . This corresponds to the process through step S909. On the other hand, if the same stroke is input twice with a slightly good tempo, the sound “Sat, Totebo ~. Two ~” is output. This corresponds to the process through step S908. Then, when the same stroke is input twice and with a better tempo, a sound “simply, fresh” is output. This corresponds to the process through step S905. By performing the processing shown in FIG. 9, it is possible to avoid the redundant audio output and to output the audio in accordance with the input tempo. Since the onomatopoeia and the wording corresponding to the shape of the stroke are sent separately to the voice output unit 158, such a wording change process can be easily realized.

  In the above description, the case where two equal strokes, the previous (previous) stroke and the current stroke, are continuously input has been described. This is true even when two or more equal strokes are input consecutively. That is, the stroke word acquisition unit 160 changes the stroke word to a simplified stroke word when a plurality of equal strokes are continuously input. Furthermore, the audio output unit 158 cancels the onomatopoeia audio output related to the second and subsequent strokes.

  Whether or not two strokes are continuously input is determined as follows. When the next stroke is input during voice output of the onomatopoeia or stroke word for the previous (previous) stroke, it is determined that the stroke has been continuously input. Specifically, it is determined by the process of step S902 and the process of step S906.

  So far, the case where a plurality of equal strokes are continuously input has been described. Conversely, the present invention can be applied when unequal strokes are continuously input. Specifically, the process of step S901 is skipped. Then, the process of step S908 is replaced with the process of step S905. By performing processing as described above, even when unequal strokes are continuously input, it is possible to avoid redundant audio output and to output audio in accordance with the input tempo.

  Next, the processing procedure of steps S515 to S518 will be described in more detail with reference to the flowchart of FIG. When the likelihood of the recognized gesture is smaller than the threshold value, a function for outputting the correct way of writing the gesture to the user is added. A small likelihood means that there is a problem with the way the user writes. This problem can be pointed out to the user by outputting the correct writing method by voice.

  Steps S516 to S518 in FIG. 10 are the same processes as the steps having the same names described in FIG. From step S514 in FIG. 5, the process proceeds to step S1001. A case where the gesture dictionary 161 of FIG. 4 is used will be described.

  In step S <b> 1001, the gesture recognition unit 163 performs matching between the input coordinate sequence corresponding to the input gesture and the reference gesture reference criterion, and calculates the likelihood for each recognition candidate. Gestures included in the gesture dictionary 161 are recognition candidates (graphic candidates). The reference gesture is data indicating how to write a model gesture. This reference gesture is held in the form of vector data or coordinate data. Then, the gesture recognition unit 163 identifies a recognition candidate having the maximum likelihood as a gesture.

  In step S1002, the gesture recognition unit 163 determines whether the likelihood of the recognized gesture is equal to or greater than a predetermined threshold. If it is equal to or greater than the threshold (step S1002; YES), the process proceeds to step S516. On the other hand, when smaller than a threshold value (step S1002; NO), it progresses to step S1003. In step S1003, the sound output unit 158 outputs a sound indicating that there is a problem in how to write the gesture. That is, a voice is output to the effect that the figure does not correspond to the gesture figure stored in the storage unit. For example, the voice output unit 158 outputs a voice such as “There is a problem in how to write a gesture”. In step S1004, the audio output unit 158 determines whether or not the input stroke sequence is equal to the correct stroke sequence constituting the recognized gesture. Here, one or more strokes are referred to as a stroke sequence. The strokes included in each stroke sequence are compared one by one, and if the shapes and sizes are all equal, it is determined that the stroke sequences are equal. If the stroke strings are equal (step S1004; YES), the process proceeds to step S1006. If they are not equal (step S1004; NO), the process proceeds to step S1005. The correct stroke sequence constituting the gesture is determined in advance in the gesture dictionary 161. For example, as shown in FIG. 4, the stroke sequence constituting the gesture of “print setting (transition to print mode)” is “shape: square, size: small” and “shape: square, size: small”. Two strokes are assumed.

  The input stroke sequence is a result of the stroke recognition unit 155 recognizing the user input. Examples of input by the user are shown in FIGS. 8B and 8C. The stroke sequence corresponding to the input in FIG. 8B is “shape: square, size: small” and “shape: square, size: small”. This stroke sequence is equal to the correct stroke sequence constituting the “print setting (transition to print mode)” gesture. On the other hand, the stroke sequence corresponding to the input in FIG. 8C is “shape: circle, size: small” and “shape: square, size: small”. This stroke sequence is not equal to the correct stroke sequence constituting the “print setting (transition to print mode)” gesture.

  In step S1005, the voice output unit 158 acquires the normal correct way of writing the recognized gesture and outputs the voice. The normal correct way of writing is composed of an onomatopoeia for each stroke constituting the gesture, a stroke word, and a gesture word for the gesture. The audio output in this case is equal to the audio output according to the processing procedure of FIG. 5 when the gesture is input in the correct way. For example, the normal correct way of writing the gesture of “print setting (transition to print mode)” is “Now, it ’s all right.

  In step S <b> 1006, the voice output unit 158 acquires the detailed correct way of writing the recognized gesture and outputs the voice. The detailed correct writing is obtained by adding a word indicating the relative positional relationship and size between strokes to the normal correct writing. For example, the correct and correct way of writing the “print setting (transition to print mode)” gesture is as follows: “Now, it ’s really good. ~ ". Here, the part of “That's it, yokonagani” is a wording indicating the relative positional relationship and size between the strokes.

  Specifically, the gesture recognition unit 163 recognizes a “print setting (transition to print mode)” gesture in response to the inputs shown in FIGS. 8B and 8C. However, since there is a problem in the writing method for both the inputs in FIGS. 8B and 8C, the likelihood is smaller than a predetermined threshold. Therefore, the process proceeds from step S1002 to step S1003.

  The stroke sequence in FIG. 8B is equal to the correct stroke sequence constituting the gesture of “print setting (transition to print mode)”. Therefore, in step S1008, the audio output unit 158 outputs the detailed correct writing method as audio. However, the input of FIG. 8B has the same individual stroke, but there is a problem with the size of the second stroke. The second stroke needs to be horizontally longer than the first stroke (see FIG. 4). This problem can be solved by a detailed and correct writing method that adds the relative positional relationship and size between strokes. In addition, since each stroke is correct, the problem cannot be pointed out with normal correct writing.

  The stroke sequence in FIG. 8C is not equal to the correct stroke sequence constituting the “print setting (transition to print mode)” gesture. Therefore, in step S1005, the voice output unit 158 outputs a normal correct writing method as a voice. The input shown in FIG. 8C has a problem with the shape of the first stroke. This problem can be pointed out in the normal correct way of writing.

  On the other hand, it is also possible to execute a command corresponding to the gesture of “print setting (transition to print mode)” in response to the inputs shown in FIGS. 8B and 8C. However, in that case, there arises a problem that the user learns the wrong way of writing. When the likelihood of the recognized gesture is small, the voice output unit 158 determines that there is a problem with the user's writing method, and outputs the correct writing method without executing the command. Thereby, it is possible to avoid that the user learns the wrong way of writing.

(Second Embodiment)
The voice output device according to the present embodiment outputs a voice indicating how the device recognizes a gesture input by a user. Also, if there is a problem with the user's way of writing, it is possible to notify the correct way of writing the gesture by voice. Therefore, the operation can be performed without a screen. For example, it can be applied to a music player with a touch panel. The user operates by writing a gesture on the touch panel with a finger. The device outputs sound to the user through an earphone or the like. Since there is no need to look at the screen, the music player can be operated with it in a pocket or bag. Further, since the screen is not essential, it can be applied as a user interface for the visually impaired, for example. In general, an application has a plurality of screens. When operating without looking at the screen, it is desirable to be able to confirm by voice which screen is currently displayed.

  With reference to FIG. 3B, another example of the onomatopoeia dictionary 153 will be described. This is an example of an image viewer having three screens of “preview”, “slide show”, and “print”. The basic beat of onomatopoeia is different for each screen. This makes it possible to check which screen is currently displayed from the onomatopoeia output by the apparatus.

  In addition, the audio output device according to the present invention is not limited to a two-dimensional gesture, and a spatial (three-dimensional) gesture can also be used. In this case, a sensor for detecting the position of the hand or the like is used instead of the touch panel. For example, it may be configured to detect the position by irradiating light from the LED, measuring the time until the light reflected by the hand reaches the sensor.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (12)

An input means for inputting a series of handwriting from the beginning of drawing to the end of drawing,
Recognizing means for recognizing the shape and size of the handwriting input by the input means;
Classification means for classifying the handwriting recognized by the recognition means into categories for each number of line segments constituting the handwriting and the size of the handwriting,
A selection means for selecting an onomatopoeia or mimicry word corresponding to the category classified by the classification means from an onomatopoeia or mimicry word stored in advance in the storage means;
Output means for outputting the onomatopoeia or mimicry word selected by the selection means as speech;
An audio output device comprising: When the recognition unit recognizes that the same handwriting as the previous input is input by the input unit during the output of the voice by the output unit,
2. The voice output device according to claim 1, wherein the selection unit selects a simplified onomatopoeia or mimetic word from the storage unit. A first acquisition unit that acquires, from the storage unit, a first word representing the shape of the handwriting recognized by the recognition unit;
3. The voice output device according to claim 1, wherein the output unit outputs the first word acquired by the first acquisition unit as a voice after outputting the onomatopoeia or the mimetic word as a voice. . When the recognition means recognizes that the same handwriting as the previous input is input by the input means,
The voice output device according to claim 3, wherein the first acquisition unit acquires a simplified word representing a shape of a handwriting. When the first acquisition unit acquires a simplified word representing the shape of the handwriting,
The voice output device according to claim 4, wherein the output unit does not output the voice of the onomatopoeia or mimicry word corresponding to the same handwriting as the previous input. Identification means for identifying one or more handwritings input by the input means as a single figure as a whole;
A second acquisition unit that acquires, from the storage unit, a second word for executing the operation indicated by the identified graphic;
The output means outputs the onomatopoeia or mimetic word corresponding to one or more handwriting inputted by the input means and the first word as speech, and then outputs the second acquired by the second acquisition means. The voice output device according to any one of claims 3 to 5, wherein the wording is output as voice. A calculation means for calculating a likelihood for each of a plurality of graphic candidates corresponding to the whole of one or more handwritings input by the input means;
The voice output device according to claim 6, wherein the identifying unit identifies a graphic from a graphic candidate having the maximum likelihood. A determination unit for determining whether or not the likelihood of the graphic identified by the identification unit is smaller than a threshold;
When the determining means determines that the likelihood of the graphic identified by the identifying means is smaller than the threshold, the output means does not store the graphic corresponding to the input graphic in the storage means The voice output device according to claim 7, wherein the voice is output as a voice. Determining means for determining whether or not the likelihood of the graphic identified by the identifying means is smaller than a threshold;
Execution means for executing processing corresponding to the graphic identified by the identification means,
When the determination means determines that the likelihood of the graphic identified by the identification means is greater than or equal to a threshold value,
The voice output device according to claim 7, wherein the execution unit executes a process corresponding to the graphic identified by the identification unit. A determination unit for determining whether or not the likelihood of the graphic identified by the identification unit is smaller than a threshold;
When it is determined by the determining means that the likelihood of the graphic identified by the identifying means is smaller than a threshold value,
8. The audio output apparatus according to claim 7, wherein the output unit outputs a drawing method of the figure identified by the identification unit as a voice. An input process in which the input means inputs a series of handwriting from the start to the end of the drawing,
A recognition step for recognizing the shape and size of the handwriting input in the input step;
A classification step for classifying the handwriting recognized by the recognition step into categories according to the number of line segments constituting the handwriting and the size of the handwriting,
A selection step in which the selection means selects an onomatopoeia or mimetic word corresponding to the category classified in the classification step from an onomatopoeia or mimetic word stored in advance in the storage step;
An output step, wherein the output means outputs the onomatopoeia or mimetic word selected in the selection step as speech;
An audio output method comprising:   The program for making a computer perform the audio | voice output method of Claim 11.

Images