EP0743619A2 - Vorrichtung zur Editierung von Zeichensprache - Google Patents

Vorrichtung zur Editierung von Zeichensprache Download PDF

Info

Publication number
EP0743619A2
EP0743619A2 EP96107904A EP96107904A EP0743619A2 EP 0743619 A2 EP0743619 A2 EP 0743619A2 EP 96107904 A EP96107904 A EP 96107904A EP 96107904 A EP96107904 A EP 96107904A EP 0743619 A2 EP0743619 A2 EP 0743619A2
Authority
EP
European Patent Office
Prior art keywords
sign language
data
time series
series data
animation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP96107904A
Other languages
English (en)
French (fr)
Other versions
EP0743619B1 (de
EP0743619A3 (de
Inventor
Hisashi Ikeda
Hirohiko Sagawa
Tomoko Sakiyama
Masaru Ohki
Youichi Kaneko
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hitachi Ltd
Original Assignee
Hitachi Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hitachi Ltd filed Critical Hitachi Ltd
Publication of EP0743619A2 publication Critical patent/EP0743619A2/de
Publication of EP0743619A3 publication Critical patent/EP0743619A3/de
Application granted granted Critical
Publication of EP0743619B1 publication Critical patent/EP0743619B1/de
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/014—Hand-worn input/output arrangements, e.g. data gloves
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B21/00—Teaching, or communicating with, the blind, deaf or mute
    • G09B21/009—Teaching or communicating with deaf persons
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/24—Indexing scheme for image data processing or generation, in general involving graphical user interfaces [GUIs]

Definitions

  • the present invention relates to an apparatus which uses an animation for the sign language to transmit information and specifically to an apparatus for editing animation data for the sign language.
  • time series data for preparing the animation are prepared in a unit of sign language sentence or in a unit of sign language word constituting the sign language sentence and the time series data are combined to produce the animation.
  • the animation data for the sign language are prepared on the basis of data inputted from a glove-type sensor (for example, DATA GLOVE of VPL Research Inc.), the animation data sometimes contain noise which is an obstacle when the data is read as the sign language.
  • a glove-type sensor for example, DATA GLOVE of VPL Research Inc.
  • the finger movement data inputted using the glove-type sensor often contain error and correction thereof is required after inputting of the data.
  • the sign language is featured so that the position in which the finger movement is expressed is varied to distinguish persons connected. In order to attain this expression, it is necessary to change the position of the finger movement for the previously registered sign language word data in accordance with the context.
  • the sign language editing apparatus for performing preparation and editing of time series data for producing the sign language animation while viewing the displayed animation of the sign language by the computer graphics, comprises means for converting movement of the fingers into an electrical signal to input information concerning the finger movement of the sign language as time series data, means for inputting a heading for time series data expressing the finger movement of the sign language, means for performing fine correction with respect to position, direction and shape of fingers to the time series data expressing the inputted finger movement in accordance with instructions by the input means so that finger movement is corrected to be able to view the finger movement naturally as the sign language, means for registering the time series data expressing the corrected finger movement, means for reading out time series data expressing one or more of the registered finger movements constituting sign language word data, means for adding information other than finger movement required to express the sign language word in accordance with instructions by the input means to the read-out time series data and producing the sign language word data constituted by time series data, means for inputting a heading for the sign language word data, and means for converting movement of the fingers into an electrical signal to
  • the information other than the finger movement includes time series data of the shape of mouth, expression and gesture (movement of the upper body and the head or the like).
  • the present invention further comprises means for reading out the sign language word data necessary to prepare sign language sentence, means for correcting the time series data of the read-out sign language word data in accordance with instructions inputted by the input means and adding information required when the sign language sentence is expressed exactly to the time series data of the read-out sign language word data to prepare the time series data expressing the sign language for the sentence, means for preserving the time series data for the prepared sign language sentence, means for producing movement of the fingers at the time that the human being uses the sign language as animation by the computer graphics on the basis of the time series data expressing the sign language, and means for displaying the produced animation.
  • the means for preparing the time series data expressing the sign language for the sentence adds time series data other than finger movement such as the mouth shape, the expression and the gesture over a plurality of sign language words as the information required when the sign language sentence is expressed exactly in accordance with the instructions inputted by the input means.
  • the means for preparing the time series data expressing the sign language for the sentence changes portion concerning the finger movement of the time series data continuously smoothly in accordance with instructions for correction of the finger movement and designation of the beginning and the end of a certain range of the portion concerning the finger movement of the time series data inputted by the input means so as to smooth the finger movement of the animation in the range.
  • the means for preparing the time series data expressing the sign language for the sentence changes movement of the sign language of the animation in accordance with instructions for the mouth shape, the expression and the gesture and designation of the beginning and the end of a certain range of the movement of the sign language of the time series data inputted by the input means so at to change movement of the sign language of the animation of the range.
  • the present invention comprises means for inserting a pause between time series data of the sign language words in accordance with designation of a mark representative of a pause between the sign language words and a length thereof by the input means to intonate the animation expressing the sign language.
  • the present invention comprises means responsive to an expansion and contraction ratio and a range of time series data designated by the input means for changing a size of space occupied by loci of the finger movement of the time series data in the designated range in accordance with the designated expansion and contraction ratio.
  • the means for producing the sign language word data selects data concerning the shape of mouth from previously registered samples in accordance with instructions by the input means to thereby add the data concerning the shape of mouth to the time series data to be edited.
  • the means for producing the sign language word data can select, as a method of assigning the mouth shape data to the time series data, any of methods of adding the mouth shape data to the head of the time series data, adding the mouth shape data to the whole time series data uniformly, adding the mouth shape data to the middle of the time series data, and adding the mouth shape data to the end of the time series data.
  • the means for producing the sign language word data selects data concerning expression from previously registered samples in accordance with instructions by the input means to thereby add the data concerning expression to the time series data to be edited.
  • the means for producing the sign language word data selects data concerning gesture from previously registered samples in accordance with instructions by the input means to thereby add the data concerning gesture to the time series data to be edited.
  • the animation data for the sign language word can be expressed by the combined finger movements to share the data between the sign language words including the same finger movements so that the memory area can be reduced and to preserve the consistency between the sign language words by correction of the finger movement. Further, when the animation for the sign language sentence is prepared, the expression, mouth shape and gesture data can be designated over a plurality of sign language words to thereby prepare the sign language sentence animation more naturally.
  • Provision of the interface for facilitating the correction of the animation data can simplify the preparation work of the animation data which is a bottleneck when the sign language is displayed by the animation.
  • information such as the expression, the shape of mouth and the gesture can be selected from the previously registered data to thereby prepare the sign language animation data simply.
  • the pause can be inserted between the words easily and the sign language word data can be corrected in accordance with the context easily to thereby prepare the understandable sign language animation.
  • Fig. 1 is a schematic diagram illustrating the whole configuration of an embodiment of the present invention.
  • Numeral 101 denotes a glove-type sensor for converting movements of the fingers in the sign language into electrical signals to input the signals as time series data when the sign language is expressed by the animation by the computer graphics.
  • Numeral 102 denotes a sign language word data editing device for correcting data relative to the movement of the fingers inputted in a unit of sign language word by the glove-type sensor 101 and adding information such as expression of the face and the shape of the mouth or the direction of the head and the upper body other than the movement of the fingers to thereby process the time series data so as to-easily understand the animation of the sign language.
  • Numeral 103 denotes a sign language word dictionary for storing the time series data prepared in a unit of sign language word by the sign language word data editing device 102 in order to prepare the animation by the computer graphics.
  • Numeral 104 denotes a sign language animation synthesizing device for synthesizing a model of the human body of the computer graphics on the basis of the time series data edited in the sign language word editing device to express movement of the sign language.
  • Numeral 105 denotes a monitor for displaying the animation by the computer graphics of the sign language, and 106 a keyboard for inputting characters.
  • Numeral 107 denotes a sign language sentence data editing device for taking out the time series data relative to the necessary sign language words from the sign language word dictionary 103 in which the time series data are stored in a unit of sign language word on the basis of arrangement of the sign language words inputted from the keyboard 106 and adding information such as corrected movement of the fingers, expression and the shape of the mouth and movements of the head and the upper body to prepare the time series data for the sign language sentence.
  • Numeral 108 denotes a sign language sentence dictionary for storing the time series data for the sign language sentence prepared in the sign language sentence data editing device 107.
  • Fig. 2 is a schematic diagram illustrating a hardware of an embodiment of the present invention.
  • Numeral 201 denotes an arithmetic unit for controlling various programs and performing information processing for preparation of the sign language animation by the computer graphics.
  • Numeral 202 denotes a keyboard and a mouse for inputting a command to the sign language editing apparatus and a heading given to each sign language word.
  • Numeral 203 denotes an input unit for interpreting data inputted from the keyboard and the mouse 202 and converting the data into a format readable by the computer.
  • Numeral 204 denotes a glove-type sensor for inputting movement of the fingers as an electrical signal to the computer
  • numeral 205 denotes a pattern input unit for converting the electrical signal inputted from the glove-type sensor 204 into the time series data readable by the computer.
  • Numeral 206 denotes a monitor for displaying the sign language animation by the computer graphics.
  • Numeral 207 denotes an output unit for producing the computer graphics data displayed in the monitor 206.
  • Numeral 208 denotes a finger movement inputting program for controlling the glove-type sensor 204 and the pattern input unit 205 when movement of the fingers is inputted.
  • Numeral 209 denotes a sign language word data editing program for correcting the time series data relative to the movement of the fingers inputted from the glove-type sensor 204 or adding information other than the movement of the fingers to the time series data relative to the movement of the fingers to prepare the time series data for each sign language word.
  • Numeral 210 denotes a sign language time series data editing program for editing the computer graphics animation for the sign language sentence by using the time series data for each sign language word prepared by the sign language word data editing program 209.
  • Numeral 211 denotes a sign language animation synthesizing program for synthesizing a model of the human body of the computer graphics from the time series data prepared by the sign language word data editing program 209 and the sign language time series data editing program 210.
  • Numeral 212 denotes data relative to the shape of mouth, expression and gesture used when the time series data for the sign language word and the sign language sentence are produced by the sign language word data editing program 209 and the sign language time series data editing program 210.
  • Numeral 213 denotes time series data relative to the movement of fingers inputted from the glove-type sensor 204 and corrected by means of the sign language word data editing program 209 and stored in the memory.
  • Numeral 214 denotes time series data in a unit of sign language word which are the stored time series data for each sign language word prepared by the sign language word data editing program 209.
  • Numeral 215 denotes sign language displaying time series data which are the stored time series data for the sign language sentence prepared by the sign language time series data editing program 210.
  • Numeral 216 denotes a memory unit in which the programs 208, 209, 210 and 211 and the data 212, 213, 214 and 215 are stored and which corresponds to the sign language sentence dictionary 108 and the sign language word dictionary 103 of Fig. 1.
  • the glove-type sensor 204 and the pattern input unit 205 can use, for example, a DATA GLOVE and an interface thereof of VPL Research Inc. to attain the function thereof.
  • the glove-type sensor 204 and the pattern input unit 205 are previously used to input the time series data of a predetermined number of sign language words and store the data in the memory area for the time series data 214 so that the data are used for preparation of the sign language sentence later.
  • Fig. 3 shows a relation of the time series data stored in the memory area for the time data series 215 for the sign language sentence and the time series data stored in the memory area for the time series data 214 for the sign language word and contents of the time series data for the sign language word.
  • Numeral 301 represents data for a sign language sentence of "I want to go to hospital”.
  • Numeral 302 represents a sign language word "I” constituting the data 301, 302 a sign language word “want to”, 303 a sign language word “go to”, and 304 a sign language word "hospital”. These words can be arranged in order of this description to express the sign language sentence of the data 301.
  • the sign language word is constituted by data of movement of the fingers, a series of shapes of mouth, expression and gesture.
  • the gesture represents movement of the human body other than the fingers, that is, inclination and rotation of the head and the upper body.
  • numeral 306 represents that movement of the fingers constituting the sign language word "hospital” is constituted by a combination of movement of the fingers of the sign language word “pulsation” and movement of the fingers of the sign language word "building".
  • Numeral 307 represents that movement of the mouth of the animation for the sign language word "hospital” is expressed by a series of shapes of the mouth composed of four shape data of the mouth including "hos”, “pi”, “ta” and "I".
  • Numeral 308 represents that expression data of "puzzlement” is assigned or added to the time series data for the sign language word “hospital” over from the previous sign language word “go to” to the sign language word "hospital”.
  • Numeral 309 represents that gesture data of inclination of the head is assigned or added to the time series data for the sign language word “hospital” over from the previous sign language word “go to” to the midway of the sign language word "hospital”.
  • the sign language includes a lot of words, such as the sign language word "hospital” explained in Fig. 3, composed of movements of the fingers representative of a plurality of sign language words ("pulsation” and "building”). It is not efficient that the same data are stored for such sign language words. That is, it is not efficient that the finger movement data of "pulsation” is stored for the sign language word “pulsation”, the finger movement data of "building” is stored for the word "building”, and the sign language data of "pulsation” and "building” are stored for the word "hospital” because the sign language data are stored in repetition.
  • the finger movement data are arranged in order in a unit of movement of the fingers having some meaning and the sign language word is expressed by a combination of movements of the fingers.
  • samples are prepared in a unit of syllable and the samples are selected to prepare the time series data for the sign language word and the sign language sentence.
  • sample data are prepared in the same manner as the shape of mouth and the sample data are selected to prepare the time series data for the sign language word and the sign language sentence.
  • the user can prepare the time series data for the animation by the computer graphics easily without preparing the additional information such as the shape of mouth, the expression and the gesture from the beginning.
  • Fig. 4 schematically illustrates dictionaries for storing data described in Fig. 3 and the relation among the dictionaries.
  • the dictionaries are constituted by the memory unit 216 of Fig. 2.
  • Numeral 401 denotes a preparation sign language sentence dictionary for storing the time series data for the sign language sentence obtained by adding information concerning the context to a combination of time series data each registered for each sign language word.
  • Numeral 402 denotes a sign language word dictionary for storing the time series data for each sign language word.
  • Numeral 403 denotes a mouth shape dictionary for storing mouth shape data expressing the shapes of the mouth used to express movement of the mouth of a model of the human body of the animation.
  • Numeral 404 denotes an expression dictionary for storing expression data used to express the expression of the face of the human body model of the animation.
  • Numeral 405 denotes a gesture dictionary for storing gesture data expressing direction and inclination of the head and the upper body used to express movement other than the finger of the human body model of the animation.
  • Numeral 406 denotes a finger movement dictionary for storing finger movement data expressing movement of the fingers of the human body model prepared by correcting data inputted by means of the glove-type sensor.
  • the sign language word data are composed of the finger movement data, the mouth shape data, the expression data and the gesture data and the sign language sentence data are expressed by a combination of sign language words.
  • Numeral 407 denotes a Japanese word to sign language word conversion table for causing Japanese words to correspond to sign language words in headings.
  • a concept of drink when expressed, it is sometimes expressed as one movement including an objective such as beer and sake.
  • the heading of the sign language word therefor is described by "drink (beer)" and this is caused to correspond to the heading "drink" of the Japanese word.
  • Fig. 5 shows the format of the time series data for the sign language sentence.
  • Numeral 501 represents a sign language sentence label given to identify the sign language sentence. Sentences used in certain scenes such as "Operation Guidance of Automatic Ticket Vending Machine" are collected and the sign language sentence label is given in the form of the order that the sentence is positioned in the collected sentences.
  • Numeral 502 represents Japanese (translated) which describes contents of the sign language sentence in Japanese.
  • Numeral 503 represents a heading of the time series data for the sign language word (for example, "change") constituting the time series data for the sign language sentence. In this case, the heading is "change”.
  • Numeral 504 represents the number of finger movement data constituting the sign language data for the sign language sentence.
  • Numeral 505 represents headings of the finger movement data at the time when the time series data concerning the finger movements constituting the sign language sentence are arranged in order of appearance in the sign language sentence.
  • the finger movements constituting the sign language sentence include “pulsation” and "building” and the headings of the finger movement data in this case are “pulsations and "buildings, which are determined by designating the sign language word heading 503.
  • Numeral 506 represents a pointer to mouth shape data expressing information relative to the mouth shape of the time series data for the sign language sentence.
  • Numeral 507 represents a pointer to expression data expressing information relative to the expression of the time series data for the sign language sentence.
  • Numeral 508 represents a pointer to gesture data expressing information relative to the gesture of the time series data for the sign language sentence.
  • Numeral 509 represents change data for storing changed portions of the time series data concerning the finger movement from the time series data concerning the original finger movement, which occur due to correction of the finger movement dependent on the context after the time series data for the sign language sentence are constituted by designating the sign language word heading 503 and arranging the time series data for the sign language word.
  • Fig. 6 shows a format of the time series data for the sign language word.
  • Numeral 601 represents a sign language word heading
  • numeral 602 represents a Japanese heading expressing a sign language word by a Japanese word corresponding thereto
  • 603 represents the number of time series data relative to the finger movements constituting the sign language word
  • numeral 604 represents a finger movement heading describing the finger movement data constituting the time series data of the sign language word in order of appearance thereof
  • numeral 605 represents a pointer to mouth shape data attendant on the sign language word
  • numeral 606 represents a pointer to expression data attendant on the sign language word
  • numeral 607 represents a pointer to gesture data attendant on the sign language word.
  • the mouth shape series data are constituted by the mouth shape data prepared in a unit of syllable such as "byo", "u", "ki” and the mouth shape series data are assigned to some portions of the time series data for the sign language sentence to thereby express the mouth shape.
  • One mouth shape series data is assigned to the time series data for the sign language word.
  • Numeral 701 represents the number of mouth shape series assigned to each of the sign language word.
  • the mouth shape data are assigned to the whole sections thereof and accordingly the number of mouth shape series is 1.
  • Numeral 702 represents a pointer to mouth shape series data assigned to the sign language sentence and the sign language word actually and numeral 703 represents a heading of the mouth shape series data.
  • the mouth shape is expressed together in a unit of word such as "hospital” and accordingly the mouth shape series data is provided in corresponding manner to a word.
  • Numeral 704 represents the number of the beginning frame in the range in which the mouth shape series data are assigned in the time series data for the respective sign language sentence and word
  • numeral 705 represents the number of the end frame to which the mouth shape series data are assigned.
  • Numeral 706 represents how to assign the mouth shape series data to the section designated by the beginning frame number 704 and the end frame number 705 and it is selected that display is made to any of the beginning, the middle, the end and the uniformity as a whole.
  • Numeral 707 designates the size of movement of the mouth shapes in order to express the difference in movement of the mouth such as quiet movement and emphatical large movement of the mouth.
  • Numeral 708 represents a pointer to data of the mouth shapes
  • numeral 709 represents a heading of the mouth shape data prepared in each syllable such as "byo" and "i”
  • numeral 710 represents the number of the head frame in the range in which the mouth shape data are assigned in the time series data for the sign language sentence and word
  • numeral 711 represents the number of the end frame in the assigned range.
  • the mouth shape data includes data of one frame (named a characteristic frame) expressing the characteristic mouth shape and a frame length of default and a range (head and end frames) in which the characteristic frame is assigned in an area having the frame length of default.
  • Numeral 712 represents the number of the head frame in the range in which the characteristic frame is assigned and numeral 713 represents the frame length of the frame range in which the characteristic frame is assigned.
  • Numeral 801 represents the number of expression data constituting expression assigned to the computer graphics animation designated by the pointers 507 and 606 to expression data in the sign language word data and the sign language sentence data. For the sign language word, the number 801 is 1.
  • the time series data for the expression data is also prepared so that each portion of the face is moved smoothly by designating the data of one characteristic frame, the range to which expression is added and the range in which the characteristic frame is assigned and interpolating the front and rear portions of the characteristic frame.
  • Numeral 802 represents a pointer to expression data including data corresponding to one characteristic frame of expression stored in the expression dictionary
  • numeral 803 represents a heading of expression data
  • numeral 804 represents the number of the head frame in the range in which expression data are assigned in the time series data
  • numeral 805 represents the number of the end frame in the range in which expression data are assigned.
  • Numeral 806 represents the number of the head frame in which the characteristic frame is assigned in the range in which the expression designated by the frames 804 and 805 is added, and numeral 807 represents the number of frames in the range in which the characteristic frame is assigned.
  • the time series data for the gesture data including inclination and rotation of the head and the upper body is also prepared so that each portion of the face is moved smoothly by designating the data of one characteristic frame, the range in which gesture is added and the range in which the characteristic is assigned and interpolating the front and rear portions of the characteristic frame.
  • Numeral 901 represents the number of gesture data constituting gesture assigned to the computer graphics animation designated by the pointers 508 and 607 to gesture data in the sign language word data and the sign language sentence data. For the sign language word, the number 901 is 1.
  • Numeral 902 represents a pointer to gesture data including data corresponding to one characteristic frame of gesture stored in the gesture dictionary
  • numeral 903 represents a heading of gesture data
  • numeral 904 represents the number of the head frame in the range in which gesture data are assigned in the time series data
  • numeral 905 represents the number of the end frame in which gesture data are assigned.
  • Numeral 906 represents the number of the head frame to which the characteristic frame is assigned in the range in which the gesture designated by the frames 904 and 905 is added, and numeral 907 represents the number of frames in the range in which the characteristic frame is assigned.
  • Numeral 1001 of Fig. 10 represents a heading of finger movement for identifying finger movement data
  • numeral 1002 represents a kind of hands used in the finger movement, that is, any of both the hands, the right hand and the left hand.
  • Numeral 1003 represents of a time length representative of the number of frames constituting the finger movement data
  • numeral 1004 represents the time series data of finger movement relative to the right hand
  • numeral 1005 represents the time series data of finger movement relative to the left hand.
  • Numeral 1006 represents elbow position data representative of an opening angle of the right elbow
  • numeral 1007 represents elbow position data representative of an opening angle of the left elbow.
  • time series data 1004 and 1005 are produced by the glove-type sensor 101 and edited by the sign language word data editing device 102 to be stored in the memory 216 as finger movement data 213.
  • Numeral 1101 represents data relative to a position of the palm in the three-dimensional space
  • numeral 1102 represents data relative to a direction of the palm
  • numeral 1103 represents data relative to the shape of the palm.
  • the data 1101 relative to the position of the palm are expressed by describing the position of the palm at a certain time by coordinate values on the three-dimensional coordinate axes x, y and z and arranging the coordinate values successively as shown by time series data 1104.
  • the data 1102 relative to the direction of the palm include a vector 1105 vertical to the plane of the palm and a vector 1106 on the plane of the palm, and both the vectors are expressed by three-dimensional vector data of x, y and z as shown by 1107 and 1108.
  • the data 1103 relative to the shape of the palm include data expressing the thumb 1109, the forefinger 1110, the middle finger 1111, the medical finger 1112 and the little finger 1113.
  • the data 1109 of the thumb are expressed by a bending angle 1114 of the root or base (named the 0-th articulation or joint) of the thumb, a bending angle 1115 of the first articulation of the thumb, a bending angle 1116 of the second articulation of the thumb and an opening angle 1117 in the same plane as the palm of the thumb.
  • the data 1110 of the forefinger are expressed by the time serial data including a bending angle 1118 of the first articulation of the forefinger, a bending angle 1119 of the second articulation of the forefinger and an opening angle 1120 of the finger in the palm plane.
  • the data of the middle finger, the medical finger and the little finger are also expressed by the time serial data in the form of from 1118 to 1120 in the same manner as the forefinger.
  • Fig. 12 illustrates an opening angle of the elbow.
  • a position of the palm at a certain time is T 1201 and positions of the shoulder and the elbow are A 1202 and F 1203, respectively.
  • a middle point of the position A of the shoulder and the position T of the palm is C 1204
  • the position E 1206 can move on the circumference having a radius of a line CF and included in a plane vertical to a line AT and an angle ECF at this time is the opening angle of the elbow.
  • the angle ECF is varied to adjust the position of the elbow in the computer graphics animation.
  • Fig. 13 shows a format of the mouth shape data in the mouth shape dictionary.
  • Numeral 1301 represents a heading of mouth shape for identifying mouth shape data prepared in a unit of syllable such as "a”, “ba” and "gya”.
  • Numeral 1302 represents data relative to the characteristic points, which are expressed by coordinates on the three-dimensional coordinate axes x, y and z as shown by 1303.
  • Fig. 14 shows a format of expression data in the expression dictionary.
  • the expression data for preparing the computer graphics animation are expressed by respective positions of the right and left eyebrows and eyes.
  • Numeral 1401 represents a heading of expression for identifying the expression data prepared in a unit of expression such as a smiling face and an angry face.
  • Numeral 1402 represents data relative to a position of the right eyebrow
  • numeral 1403 represents data relative to a position of the right eye
  • numeral 1404 represents data relative to a position of the left eyebrow
  • numeral 1405 represents data relative to a position of the left eye.
  • Numeral 1406 represents the number of characteristic points of the eyebrow necessary to express the data 1402 relative to the position of the right eyebrow, and the characteristic points are expressed as values 1407 on the three-dimensional coordinate axes x, y and z.
  • the data 1405 relative to the position of the left eye are expressed by the number 1408 of characteristic points for expressing the eye and values 1409 on the three-dimensional coordinate axes x, y and z of the characteristic points.
  • Fig. 15 shows a format of gesture data in the gesture dictionary.
  • Numeral 1501 represents a heading of gesture for identifying gesture data.
  • the gesture data are constituted by data relative to directions and angles of the head and the upper body.
  • Numeral 1502 represents data relative to inclination and rotation of the head, and the data 1502 are expressed by an inclination angle 1503 in the right and left direction of the head, an inclination 1504 in the front and rear direction of the head and a rotation angle 1505 of the head.
  • Numeral 1506 represents a heading of data relative to the upper body, and the data are expressed by an inclination angle 1507 in the right and left direction of the upper body, an inclination angle 1508 in the front and rear direction of the upper body and a rotation angle 1509 of the upper body.
  • Sample data of the expression data, the mouth shape data and the gesture data of the above data previously prepared in the expression dictionary, the mouth shape dictionary and the gesture dictionary can be selected by the user to prepare the animation data.
  • the finger movement constituting the sign language word is first read out from the dictionary or prepared newly to be designated (step 1601).
  • the sign language word of, for example, "hospital” is prepared, the finger movement meaning “pulsation” and the finger movement meaning “building” are designated to thereby describe data relative to the finger movement of the sign language word "hospital”.
  • step 1602 Whether the time series data relative to the finger movement designated in step 1601 is registered in the finger movement dictionary or not is examined (step 1602).
  • the finger movement is inputted by the glove-type sensor 101 (step 1603) and the inputted finger movement data are corrected with respect to the position of the finger, the direction of the finger, the shape of the finger and the position of the elbow (step 1604).
  • the resultant finger movement data are stored in the finger movement dictionary together with the heading (step 1605).
  • step 1602 When the finger movement designated in step 1602 has been registered and after the finger movement data has been registered in the finger movement dictionary in step 1605, information relative to the mouth shape, the expression and the gesture is added by selecting data from the respective dictionaries (step 1606) and the resultant sign language word data is registered in the sign language word dictionary (step 1607).
  • the time series data relative to the sign language word are prepared in a unit of finger movement.
  • the editing work in this case is made while the animation is displayed during the editing and the edited result is confirmed.
  • the user first designates the sign language words constituting the sign language sentence (step 1701).
  • step 1702 Whether the time series data for the headings of the sign language words designated in step 1701 are registered in the sign language word dictionary or not is then examined (step 1702).
  • the sign language words are registered (step 1703).
  • the registration of the sign language word data is made by the method described in Fig. 16.
  • step 1702 when the time series data relative to the sign language words are registered or after the time series data relative to the sign language words have been registered in step 1703, pauses are inserted in portions required between the sign language words in the animation (step 1704).
  • Insertion of the pauses between the sign language words can intonate the animation and prepare the readable or understandable computer graphics animation of the sign language as compared with simple arrangement of the time series data prepared for each sign language word.
  • the data registered for each sign language word are modified with respect to finger movement, expression, mouth shape and gesture in accordance with the context (step 1705).
  • the sign language sentence data resulted from the work performed until step 1705 are stored in the preparation sign language sentence dictionary (step 1706).
  • the editing work in this case is made while the animation is displayed during the editing and the edited result is confirmed in the same manner as upon editing of the sign language word data. Editing of the time series data for the sign language animation described above is described below.
  • Fig. 18 shows an example of an editing picture of the sign language word data.
  • Numeral 1801 represents a file pull-down menu for calling the time series data for the sign language word onto the editing picture, storing the time series data in the sign language word dictionary or the like.
  • Numeral 1802 represents an editing pull-down menu for performing cutting, pasting, interpolation of movement, adjustment of display speed, compression and expansion of movement and the like for the time series data relative to the finger movement, specifically, of the time series data for the sign language words.
  • Numeral 1803 represents a mode pull-down menu for switching editing of the time series data for the sign language word and editing of the time series data for the sign language sentence.
  • Numeral 1804 represents a display pull-down menu for designating whether parameter values relative to position, direction and shape of the finger are displayed together with the sign language animation or not.
  • Numeral 1805 represents a dictionary pull-down menu for selecting a dictionary in which the time series data for the edited result are registered.
  • Numeral 1806 represents a display column for the sign language word heading in which the heading 601 of the sign language word to be edited is inputted and displayed.
  • a knob or handle 1807 is moved right and left, the sign language word heading 1806 is scrolled right and left in response to the movement thereof.
  • Numeral 1808 represents a display column for the Japanese word heading in which the Japanese heading 602 (Fig. 6) for the sign language word is inputted and displayed.
  • Numeral 1809 represents a display column for the finger movement heading in which the finger movement headings 604 constituting the time series data for the sign language word to be edited are inputted and displayed in order of construction or arrangement thereof.
  • Numeral 1810 represents a button for retrieving the finger movement data for the finger movement heading inputted in the display column 1809 from the finger movement dictionary. This retrieval corresponds to the operation in step 1602 (Fig. 16).
  • Numeral 1811 represents a sign language word basic information table for displaying information relative to finger movement, expression, mouth shape and gesture constituting the sign language word. Information relative to the sign language word obtained as a result of the retrieval by the button 1810 is displayed in the table 1811.
  • Numerals 1812 and 1813 represent the numbers of the beginning frame and the end frame, respectively, when frames in a certain section are selected in the editing work.
  • Numeral 1814 represents a frame indicator used when a frame in a certain section is selected in the editing work. When a frame is selected by the user, a portion corresponding to the selected frame is reversely displayed.
  • the expression heading assigned as expression data is displayed in area 1815
  • the mouth shape heading assigned as mouth shape data is displayed in area 1816
  • the gesture heading assigned as gesture data is displayed in area 1817.
  • Numeral 1818 represents a finger movement editing button for correcting finger movement data relative to a frame selected by depressing the button 1818 after a certain frame section is selected in 1814.
  • Numeral 1819 represents an expression editing button for editing expression data by depressing the button 1819 after a certain frame section is selected in 1814.
  • Numeral 1820 represents a mouth shape editing button for editing the shape of mouth by depressing the button 1820 after a certain frame section is selected in 1814.
  • Numeral 1821 represents a gesture editing button for editing gesture data by depressing the button 1821 after a certain frame section is selected in 1814.
  • Numeral 1822 represents a sign language animation display window for displaying the sign language animation.
  • Numeral 1823 represents an indicator indicating the number of whole frames constituting the time series data for the sign language word and a position of the frame displayed currently.
  • -Numeral 1824 represents a knob or handle for changing the direction of the displayed sign language animation in the vertical direction.
  • Numeral 1825 represents a knob or handle for changing the displayed sign language animation in the horizontal direction.
  • Numeral 1826 represents a button for reproducing the sign language animation in the reverse direction, 1827 a button for starting reproduction of the sign language animation, 1828 a button for reproducing the sign language animation in the reverse direction at a double speed, 1829 a button for reproducing the sign language animation in the forward direction at a double speed, 1830 a button for reproducing the sign language animation in the reverse direction at a half speed, 1831 a button for reproducing the sign language animation in the forward direction at a half speed, 1832 a button for reproducing the sign language animation in the reverse direction by one frame, 1833 a button for reproducing the sign language animation in the forward direction by one frame, 1834 a button for moving the sign language animation to the beginning frame, 1835 a button for stopping reproduction of the sign language animation, 1836 a button for interrupting reproduction of the sign language animation, and 1837 a button for moving the sign language animation to the end frame.
  • Such pictures are used to edit the sign language word data.
  • New registration of finger movement is selected (1901) from the file menu 1801 to thereby display a finger movement input window of Fig. 19B.
  • Numeral 1902 represents a display column of finger movement heading for displaying the finger movement heading.
  • the finger movement heading inputted in the display column 1809 is displayed in the display column 1902.
  • a specific position (home position) is previously designated and one finger movement until the finger is returned to the home position after the finger begins to move from the home position is gathered.
  • Numeral 1903 represents a home position button for designating a position of the finger used as the beginning and the end for gathering data when the finger movement is inputted.
  • the position of the finger at the time that this button is depressed is a position for performing the function of switch.
  • Numeral 1904 represents a normalizing button used when information relative to a size of the body of a finger movement registration person is inputted in order to correct a deviation of the relative positional relation between the finger and the body caused by a difference of the size of the body of the finger movement registration person equipped with the glove-type sensor.
  • the top of the head is indicated by the finger equipped with the glove-type sensor to thereby input the relative positional relation of the top of the head and the finger (step 2001).
  • the chin is then indicated by the finger equipped with the glove-type sensor to thereby input the relative positional relation of the chin and the finger (step 2002).
  • the left shoulder is indicated by the finger of the right hand and the right shoulder is indicated by the finger of the left hand to thereby input the relative positional relation (step 2003).
  • the positional information of the abdomen is then inputted (step 2004).
  • step 2005 information relative to the palm is normalized. More particularly, the shapes of fingers in three states including all fingers stretched, only the thumb stretched and other fingers stretched except the thumb are registered.
  • step 2006 When the inputting of the normalization information performed so far is not made exactly, the same operation is made again from step 2001 and when the inputting is made satisfactorily, the process proceeds to next step (step 2006).
  • step 2007 the home position for changing over the beginning and interruption of inputting of the finger movement is inputted (step 2007).
  • the registration person sits and makes the sign language, the person puts both the hands on the knees to input this state to thereby register the home position.
  • a specific position of the fingers or hand for performing the duty of the switch for starting gathering the finger movement data is previously registered to thereby move the fingers or hand to the registered position after depressing of the button, so that the time series data relative to the subsequent finger movement can be gathered.
  • Numeral 1905 represents an input button for starting inputting of the finger movement.
  • Numeral 1906 represents a stop button for stopping inputting of the finger movement
  • numeral 1907 represents a monitoring display button for displaying the sign language computer graphics animation prepared on the basis of the time series data relative to the inputted finger movement in the animation display window 1822
  • numeral 1908 represents a finger movement retrieval button for searching the finger movement dictionary for the sign language word including the finger movement while the inputted finger movement is used as a key
  • numeral 1909 denotes a clear button for clearing the time series data relative to the input finger movement from the memory.
  • the time series data relative to the inputted finger movement and the time series data relative to the finger movement already registered in the dictionary are collated to thereby retrieve the sign language word constituted by the inputted finger movement.
  • the retrieval can be attained by the sign language recognition technique described in JP-A-6-83938.
  • the time series data relative to the inputted finger movement is stored in the finger movement dictionary.
  • the time series data for a plurality of frames which are parts of the series of frames constituting the computer graphics animation for the sign language or a single frame is corrected.
  • the frame to be corrected must be selected before the correction method of data is selected. Selection of the frame to be corrected is described with reference to Fig. 21.
  • Numeral 2101 represents an editing pull-down menu for selecting a kind of editing for the series of frames.
  • the 11st frame 2102 is selected by the input device such as a mouse and the frames from the 11st frame to the 16th frame 2103 which is an end of the frame series to be corrected are dragged or selected.
  • the area 2104 selected on the sign language word basic information table is reversely displayed.
  • the editing item can be selected from the editing pull-down menu to thereby make various correction to the time series data of the designated range.
  • Fig. 22 shows a picture at the time that the position of the finger is corrected.
  • the case where the position of the finger is corrected over a plurality of frames and the case where the position of the finger is corrected for a single frame can be both treated.
  • An amount of variation relative to the position of the finger given herein represents an amount of relative movement of the position of the fingers.
  • Numeral 2201 represents a position correction window of the finger, 2202 a Z-axis slider for correcting the position of the finger in the Z-axis direction (vertical direction), 2203 a Y-axis slider for correcting the position of the finger in the Y-axis direction (in the right and left direction), and 2204 an X-axis slider for correcting the position of the finger in the X-axis direction (in the front and rear direction).
  • the sliders are moved to correct the position of the finger while viewing the displayed computer graphics animation for the sign language on the animation display window 1822. Further, coordinate values of Z, Y and X can be directly inputted in 2205, 2206 and 2207 to thereby correct the position of the finger.
  • Numeral 2208 represents a setting button for preserving the edited result and numeral 2209 represents a cancel button for canceling the edited result.
  • Fig. 23 shows a picture at the time that the direction of the finger is corrected.
  • Numeral 2301 represents an editing window for the direction of the palm, 2302 a bending slider for correcting a bending angle in the front and rear direction of the palm, 2303 a column for directly designating the angle, 2304 a bending slider for correcting a bending angle in the right and left direction of the palm, 2304 a column for directly designating the angle, 2306 a rotation slider for correcting a rotation angle of the wrist, and 2307 a column for directly designating the angle.
  • the sliders are moved to perform correction while viewing the displayed computer graphics animation for the sign language on the animation display window 1822.
  • Numeral 2308 represents a setting button for preserving the edited result in the finger movement dictionary, and 2309 a cancel button for canceling the edited result.
  • Fig. 24 shows a picture at the time that the shape of the fingers or hand is edited.
  • Numeral 2401 represents a finger or hand shape selection window, and 2402 the shape of fingers or hand selected from a list of shapes of fingers or hand previously prepared by means of a mouse or the like.
  • Numeral 2403 represents a scroll in the right and left direction, and 2404 a scroll in the vertical direction.
  • Numeral 2405 represents an image of the shape of the fingers or hand selected in 2402 and which is a data for one frame.
  • Numeral 2406 represents a fine correction button for performing more detailed correction to the selected shape of the fingers or hand, 2407 a setting button for preserving the selected result, 2408 a cancel button for canceling the selected result.
  • the shape of the fingers or hand of the animation can be expressed by only selection of the prepared shape of the fingers or hand.
  • the fine correction button 2406 is depressed to adjust parameters for expressing the shape of the fingers or hand.
  • Fig. 25 shows a picture displayed when the fine correction button 2406 is depressed in the finger shape selection window 2401.
  • Numeral 2401 represents a finger shape correction window, 2402 a thumb 0 slider for correcting a bending angle of the 0-th articulation of the thumb, 2404 a thumb 1 slider for correcting a bending angle of the first articulation of the thumb, 2406 a thumb 2 slider for correcting a bending angle of the second articulation of the thumb, 2408 a thumb opening slider for correcting an opening angle in the palm plane of the thumb, 2410 a thumb rotation slider for correcting a rotation angle of the thumb, 2412 a forefinger 2 slider for correcting a bending angle of the second articulation of the forefinger, 2414 a forefinger 1 slider for correcting a bending angle of the first articulation of the forefinger, 2516 a forefinger opening slider for correcting an opening angle in the palm plane of the forefinger, 2418 a middle finger 2 slider for correcting a bending angle of the second articulation of the middle finger, 2420 a middle finger 1 slider
  • these parameters can be corrected by directly designating values thereof in the columns indicated by 2403, 2405, 2407, ..., 2425.
  • Correction is made by moving the sliders while viewing the computer graphics animation for the sign language displayed on the sign language animation display window 1822.
  • Numeral 2436 represents a setting button for preserving the corrected result in the finger movement dictionary, and 2437 a cancel button for canceling the edited result.
  • the setting button is depressed to preserve the corrected result for the finger shape selected in the finger shape selection window 2401 in the finger movement dictionary.
  • Fig. 26 shows a picture for correcting the position of the elbows.
  • the position of the elbows cannot be inputted by the glove-type sensor and accordingly the position of the default thereof is obtained by calculation on the basis of the positions of the shoulder and the hands.
  • the functions described herein are prepared.
  • Numeral 2601 represents an elbow position correction window, 2602 a right slider for adjusting a rotation angle of the elbow of the right hand. The rotation angle thereof can be corrected by directly designating the angle into 2603.
  • Numeral 2604 represents a left slider for adjusting an opening angle of the elbow of the left hand. The opening angle thereof can be corrected by directly designating the angle into 2605.
  • Numeral 2606 represents a setting button for preserving the edited result, and 2607 a cancel button for canceling the edited result.
  • Fig. 27 shows a picture for expanding and contracting the hand movement.
  • Numeral 2702 represents an X axis expansion and contraction designating button for designating whether expansion and contraction in the front and rear direction in the computer graphics picture is made or not, 2703 a Y axis expansion and contraction designating button for designating whether expansion and contraction in the right and left direction in the computer graphics picture is made or not, 2704 a Z axis expansion and contraction designating button for designating whether expansion and contraction in the vertical direction in the computer graphics picture is made or not, 2705 an expansion and contraction ratio designating column for designating the expansion and contraction ratio of hand movement to the coordinate axes designated by 2702, 2703 and 2704, 2706 an execution button for varying the hand movement in accordance to the expansion and contraction ratio inputted in 2705, and 2707 a cancel button for canceling expansion and contraction of the hand movement.
  • a range of the time series data to be expanded and contracted and the expansion and contraction ratio are designated (step 2901).
  • p0 (2801), p1 (2802), p2 (2803), p3 (2804), p4 (2805) and p5 (2806) of Fig. 28A are positions of the hand at certain time, and this hand movement series is expanded and contracted.
  • a middle point C (2807) between both ends p0 and p5 of the time series data is calculated (step 2902).
  • a point q0 on a straight line p0-C is calculated in accordance with an inputted expansion and contraction ratio (step 2904).
  • the point q0 corresponds to 2808 of Fig. 28B.
  • q1 (2809), q2 (2810), q3 (2811), q4 (2812) and q5 (2813) are calculated.
  • q0, q1, q2, q3, q4 and q5 are loci formed by contracting the original hand movement data.
  • the contraction can be applied to each of the X, Y and Z axes representative of the position of the hand.
  • 2101 is the editing pull-down menu.
  • the "paste" is selected from the editing pull-down menu, so that data for the frame introduced previously into the buffer by execution of the cut or copy are pasted.
  • Numeral 3001 represents a paste window displayed when the "paste" is selected in 2101, 3002 a forward direction switch for selecting the paste in the original time series order of the time series data introduced into the buffer, 3003 a reverse direction switch for selecting the paste in the reverse direction to the original time series order of the time series data in the buffer, 3004 an execution button for pasting the time series data actually on the basis of the direction of the paste set by the switches 3002 and 3003, and 3005 a cancel button for canceling the execution of the paste.
  • the introduction of the time series data into the buffer is made by selecting the frame by means of the frame indicator 1814 of the sign language word basis information table and selecting "cut" or "copy” of the editing pull-down menu.
  • Numeral 3101 represents an interpolation window
  • 3102 a switch for selecting interpolation of the position of the right hand
  • 3103 a switch for selecting interpolation of the position of the left hand
  • 3104 a switch for selecting the direction of the right hand
  • 3105 a switch for selecting interpolation of the direction of the left hand
  • 3106 a switch for selecting interpolation of the shape of the right hand
  • 3107 a switch for selecting interpolation of the shape of the left hand.
  • the interpolation process is performed as to the items to which execution of interpolation is designated by the switches 3102 to 3107.
  • Numeral 3108 represents an interpolation button for executing the interpolation on the basis of the selected result by the switches 3102 to 3107, and 3109 a cancel button for erasing the interpolation window 3101 from the picture without interpolation.
  • the user can select the switches 3102 to 3107 to perform interpolation with respect to any item.
  • Fig. 32 is a diagram illustrating addition of the expression to the sign language word.
  • the expression editing window 3201 is displayed.
  • Numeral 3202 represents a list of headings for expression data previously registered in the expression dictionary. The user can select a proper heading from the list to add the expression data to the sign language word data. The selected heading is reversely displayed.
  • Numeral 3203 represents a scroll bar for scrolling the expression heading list in the vertical direction
  • 3204 a scroll bar for scrolling the expression heading list in the horizontal direction.
  • Numeral 3205 represents an expression display window for displaying the expression data selected from the list 3202.
  • the expression data corresponding to the reversely displayed expression heading is loaded from the expression dictionary 404 and is displayed in the window 3205.
  • Numeral 3206 represents a setting button for actually adding the expression data selected from the list to the sign language word, and 3207 a cancel button for canceling the above work.
  • the user can merely select the heading of expression to add the expression to the sign language animation.
  • Fig. 33 is a diagram illustrating addition of the mouth shape to the sign language word.
  • a mouth shape editing window 3301 is displayed.
  • Numeral 3302 represents a list of headings for mouth shape data previously registered in the mouth shape dictionary. The user can select a proper heading from the list to prepare the mouth shape series data and add the data to the sign language word data.
  • Numeral 3303 represents a scroll bar for scrolling the mouth shape heading list in the vertical direction
  • 3304 a scroll bar for scrolling the mouth shape heading list in the horizontal direction.
  • Numeral 3305 represents a mouth shape display window for displaying the mouth shape data selected from the list 3302.
  • Numerals 3306 to 3309 represent buttons for designating how to assign the mouth shape series to the selected frame data.
  • the button 3306 is selected, the mouth shape series is assigned to the head of the frame data series.
  • the button 3307 is selected, the mouth shape series is assigned uniformly over the whole frame data series.
  • the button 3308 is selected, the mouth shape series is assigned to the middle of the frame data series.
  • the button 3309 is selected, the mouth shape series is assigned to the end of the frame data series.
  • Difference in timing of the expression of the mouth shape and the finger movement can be added to each sign language word so that the mouth shape is expressed slowly while performing the finger movement and the mouth shape is expressed at a breath at the beginning of the finger movement.
  • the meaning corresponding to the context can be added to the sign language animation.
  • Numerals 3310 to 3312 represent buttons for adding variation to the size of movement of the lips.
  • the button 3310 is selected, the standard mouth shape data in the mouse shape dictionary is assigned.
  • the button 3311 is selected, the mouth shape data having large movement of the lips in the mouth shape dictionary is assigned.
  • the mouth shape data having small movement of the lips in the mouth shape dictionary is assigned. More particularly, changed parameters expressing the lips in the mouth shape data 709 in the mouth shape dictionary is assigned.
  • These buttons are also exclusive and only one button can be selected.
  • Numeral 3313 represents a mouth shape series display column in which the headings of the mouth shape data selected from the list 3302 are displayed in order of selection.
  • the mouth shape data series displayed therein is the mouth shape series 702.
  • Numeral 3314 represents a setting button for preserving the edited result as the sign language word data, and 3314 a cancel button for canceling the edited result.
  • Fig. 34 is a diagram illustrating addition of gesture to the sign language word.
  • a gesture editing window 3401 is displayed.
  • Numeral 3402 represents a list of headings for gesture data registered previously in the gesture dictionary. The user can select a proper heading from the list 3402 to add the gesture data to the sign language data. The selected heading of the sign language word is reversely displayed as shown in Fig. 34.
  • Numeral 3403 represents a scroll bar for scrolling the list of the gesture data headings in the horizontal direction
  • 3404 a scroll bar for scrolling the list of the gesture data heading in the vertical direction.
  • Numeral 3405 represents a gesture display window for displaying gesture data in the gesture dictionary with respect to the gesture data heading selected from the list 3402, and 3406 a slider for rotating the direction of a model of the human body of the computer graphics animation for the sign language.
  • Numeral 3407 represents a setting button for preserving the edited result as the sign language word data, and 3408 a cancel button for canceling the edited result.
  • the user can merely select the heading of the gesture data to thereby add the gesture data to the sign language word.
  • the sign language word preservation is selected from the file pull-down menu 1801 to preserve the sign language word data in the sign language word dictionary. This process corresponds to the process in step 1606.
  • the above process is a flow of the editing process of the sign language word data.
  • the sign language sentence data editing picture of Fig. 35 is displayed.
  • Numeral 3501 represents a file pull-down menu for calling the sign language sentence data prepared by combining the time series data for the sign language words and preserving the data in the sign language sentence dictionary.
  • Numeral 3502 represents an editing pull-down menu for performing cut, copy, interpolation, adjustment of display speed, compression and expansion of operation and the like with respect to the time series data relative to the finger movement of the sign language data and changing data registered as the original sign language word data.
  • Numeral 3503 represents a mode pull-down menu for switching editing of the sign language sentence data and editing of the sign language word data.
  • Numeral 3504 represents a display pull-down menu for designating whether parameters relative to the position, the direction and the shape of the fingers are displayed together with the sign language animation or not.
  • Numeral 3505 represents a dictionary pull-down menu for selecting the dictionary in which the animation data of the edited result are registered.
  • Data concerning the prepared sign language sentences are collected as a collected sentences (folder) for each kind of sentences used in application, for example, such as sentences used in the automatic cash dispenser.
  • Numeral 3506 represents a folder name designating column for inputting and displaying a heading of the collected sentences of the heading 501 (Fig. 5) of the sign language sentence to be edited.
  • Numeral 3507 represents a file name designating column for inputting and displaying the sign language sentence heading given to each sign language sentence data in order to identify the sign language data to be edited.
  • Numeral 3508 represents a translated Japanese display column for inputting and displaying the translated Japanese 502 for the sign language sentence, and 3509 a scroll bar for scrolling the translated Japanese in the horizontal direction.
  • Numeral 3510 represents a sign language word input display column for inputting and displaying the sign language word heading 505 constituting the sign language data to be edited in order of arrangement thereof, and 3511 a scroll bar for scrolling the sign language word series 3510 in the horizontal direction.
  • the sign language word data for the heading is retrieved from the sign language word dictionary. This retrieval corresponds to the process in step 1702 (Fig. 17).
  • the number of the beginning frame thereof is displayed in 3512 and the number of the end frame is displayed in 3513.
  • Numeral 3514 represents a sign language sentence basic information table for displaying information relative to finger movement, expression, mouth shape and gesture registered in the sign language word data constituting the sign language sentence. The retrieved result of the sign language word data by 3510 is displayed.
  • the heading of the sign language word constituting the sign language sentence data to be edited is displayed in 3515 and the heading of the finger movement data constituting the sign language word data is displayed in 3516.
  • Numeral 3517 represents a frame indicator used when a frame of a certain section is selected in the edited work.
  • the expression heading assigned as the expression data is displayed in 3518
  • the mouth shape heading assigned as the mouth shape data is displayed in 3519
  • the gesture heading assigned as the gesture data is displayed in 3520.
  • Numeral 3522 represents a scroll bar for scrolling the data displayed in the sign language sentence basic information table 3514 in the horizontal direction.
  • Numeral 3523 represents a finger movement editing button for correcting the finger movement data by depressing the button 3523 after a certain frame section is selected in the sign language sentence basic information table 3514.
  • Numeral 3524 represents a finger movement position editing button for editing the position of the palm by depressing the button 3524 after a certain frame section is selected in the sign language sentence basic information table 3514.
  • Numeral 3525 represents an expression editing button for editing expression data by depressing the button 3525 after a certain frame section is selected in the sign language basic information table 3514.
  • Numeral 3526 represents a mouth shape editing button for editing the shape of mouth by depressing the button 3526 after a certain frame section is selected in the sign language sentence basic information table 3514.
  • Numeral 3527 represents a gesture editing button for editing the gesture data by depressing the button 3527 after a certain frame section is selected in the sign language sentence basic information table 3524.
  • Numeral 3528 represents a sign language animation display window for displaying the computer graphics animation of the sign language, 3529 an indicator for indicating the total number of frames constituting the sign language word data and the position of the currently displayed frame, 3530 a knob or handle for changing the direction of the expression of the sign language animation in the vertical direction, and 3531 a knob or handle for changing the display of the sign language animation in the horizontal direction.
  • Numeral 3532 represents a button for reproducing the sign language animation in the reverse direction, 3533 a button for starting reproduction of the sign language animation, 3534 a button for reproducing the sign language animation in the reverse direction at a double speed, 3535 a button for reproducing the sign language animation in the forward direction at a double speed, 3536 a button for reproducing the sign language animation in the reverse direction at a half speed, 3537 a button for reproducing the sign language animation in the forward direction at a half speed, 3538 a button for reproducing the sign language animation in the reverse direction by one frame, 3539 a button for reproducing the sign language animation in the forward direction by one frame, 3540 a button for moving to the beginning frame of the sign language animation, 3541 a button for stopping reproduction of the sign language animation, 3542 a button for interrupting reproduction of the sign language animation, and 3543 a button for moving to the end frame of the sign language animation.
  • Such pictures are used to edit the sign language sentence data.
  • step 1702 when the necessary sign language word data are not registered, the sign language word data are edited and registered by the method illustrated in Fig. 16. Insertion of pause in the sign language sentence data in step 1704 is attained by inputting a mark of "P, number" as shown in Fig. 36 in the sign language word series column 3510 of the sign language sentence editing window (3601).
  • the final frame of the sign language word data just before "P, number” is displayed repeatedly by the number of times designated by the "number” to thereby express the pause.
  • the pause is displayed in the sign language sentence basic information table as shown by 3602 of Fig. 36.
  • Correction of the finger movement, editing of the expression, editing of the mouth shape and editing of the gesture performed for the edited result in step 1705 can be attained by the same method as the method described in editing of the sign language word data.
  • the sign language sentence data prepared as above are preserved in the sign language display dictionary.
  • the animation data for the sign language word can be expressed by the combined finger movements to thereby share data between the sign language words having the same finger movement to reduce a memory area and maintain the consistency between the sign language words by correction of the finger movement.
  • the expression data, the mouth shape data and the gesture data can be designated not only to the sign language word but also to the sign language sentence when the animation of the sign language sentence is prepared, to thereby prepare the more natural sign language animation.
  • provision of the interface for facilitating correction of the animation data can simplify the preparation work of the animation data which is the bottleneck when the sign language is displayed by the animation.
  • information relative to the expression, the shape of mouth, the gesture and the like can be selected from the previously registered data to thereby prepare the sign language animation simply.
  • pause can be inserted between words easily and the sign language word data are corrected easily in accordance with the context to thereby prepare the understandable sign language animation.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • General Health & Medical Sciences (AREA)
  • Business, Economics & Management (AREA)
  • Educational Administration (AREA)
  • Educational Technology (AREA)
  • Processing Or Creating Images (AREA)
EP96107904A 1995-05-18 1996-05-17 Vorrichtung zur Editierung von Zeichensprache Expired - Lifetime EP0743619B1 (de)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
JP144123/95 1995-05-18
JP14412395 1995-05-18
JP7144123A JPH08315185A (ja) 1995-05-18 1995-05-18 手話編集装置

Publications (3)

Publication Number Publication Date
EP0743619A2 true EP0743619A2 (de) 1996-11-20
EP0743619A3 EP0743619A3 (de) 1997-02-26
EP0743619B1 EP0743619B1 (de) 2001-01-17

Family

ID=15354737

Family Applications (1)

Application Number Title Priority Date Filing Date
EP96107904A Expired - Lifetime EP0743619B1 (de) 1995-05-18 1996-05-17 Vorrichtung zur Editierung von Zeichensprache

Country Status (4)

Country Link
US (1) US5990878A (de)
EP (1) EP0743619B1 (de)
JP (1) JPH08315185A (de)
DE (1) DE69611570D1 (de)

Families Citing this family (55)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8352400B2 (en) 1991-12-23 2013-01-08 Hoffberg Steven M Adaptive pattern recognition based controller apparatus and method and human-factored interface therefore
JPH07168851A (ja) * 1993-12-16 1995-07-04 Canon Inc 画像表示方法及び装置
US7904187B2 (en) 1999-02-01 2011-03-08 Hoffberg Steven M Internet appliance system and method
US6539354B1 (en) 2000-03-24 2003-03-25 Fluent Speech Technologies, Inc. Methods and devices for producing and using synthetic visual speech based on natural coarticulation
AU2001292963A1 (en) 2000-09-21 2002-04-02 The Regents Of The University Of California Visual display methods for use in computer-animated speech production models
US6957389B2 (en) * 2001-04-09 2005-10-18 Microsoft Corp. Animation on-object user interface
US20040012643A1 (en) * 2002-07-18 2004-01-22 August Katherine G. Systems and methods for visually communicating the meaning of information to the hearing impaired
JP3968277B2 (ja) 2002-08-26 2007-08-29 株式会社日立製作所 メールサーバ、そのメールサーバを実現するためのプログラム及び携帯端末
US7565295B1 (en) * 2003-08-28 2009-07-21 The George Washington University Method and apparatus for translating hand gestures
US7707039B2 (en) 2004-02-15 2010-04-27 Exbiblio B.V. Automatic modification of web pages
US8442331B2 (en) 2004-02-15 2013-05-14 Google Inc. Capturing text from rendered documents using supplemental information
US7812860B2 (en) 2004-04-01 2010-10-12 Exbiblio B.V. Handheld device for capturing text from both a document printed on paper and a document displayed on a dynamic display device
US10635723B2 (en) 2004-02-15 2020-04-28 Google Llc Search engines and systems with handheld document data capture devices
US7990556B2 (en) 2004-12-03 2011-08-02 Google Inc. Association of a portable scanner with input/output and storage devices
US8146156B2 (en) 2004-04-01 2012-03-27 Google Inc. Archive of text captures from rendered documents
US9143638B2 (en) 2004-04-01 2015-09-22 Google Inc. Data capture from rendered documents using handheld device
US20060081714A1 (en) 2004-08-23 2006-04-20 King Martin T Portable scanning device
US8081849B2 (en) 2004-12-03 2011-12-20 Google Inc. Portable scanning and memory device
US9008447B2 (en) 2004-04-01 2015-04-14 Google Inc. Method and system for character recognition
US9116890B2 (en) 2004-04-01 2015-08-25 Google Inc. Triggering actions in response to optically or acoustically capturing keywords from a rendered document
US7894670B2 (en) 2004-04-01 2011-02-22 Exbiblio B.V. Triggering actions in response to optically or acoustically capturing keywords from a rendered document
US20060098900A1 (en) 2004-09-27 2006-05-11 King Martin T Secure data gathering from rendered documents
US8713418B2 (en) 2004-04-12 2014-04-29 Google Inc. Adding value to a rendered document
US8489624B2 (en) 2004-05-17 2013-07-16 Google, Inc. Processing techniques for text capture from a rendered document
US8620083B2 (en) 2004-12-03 2013-12-31 Google Inc. Method and system for character recognition
US8874504B2 (en) 2004-12-03 2014-10-28 Google Inc. Processing techniques for visual capture data from a rendered document
US8346620B2 (en) 2004-07-19 2013-01-01 Google Inc. Automatic modification of web pages
US20060134585A1 (en) * 2004-09-01 2006-06-22 Nicoletta Adamo-Villani Interactive animation system for sign language
US7746985B2 (en) * 2005-09-14 2010-06-29 Sorenson Communications, Inc. Method, system and device for relay call transfer service
US7746984B2 (en) * 2005-09-14 2010-06-29 Sorenson Communications, Inc. Method and system for call initiation in a video relay service
US7742068B2 (en) * 2005-09-14 2010-06-22 Sorenson Communications, Inc. Method and system for auto configuration in a video phone system
US7769141B2 (en) * 2005-09-23 2010-08-03 Sorenson Communications, Inc. Method and system for visual spatial caller identification
US20080036737A1 (en) * 2006-08-13 2008-02-14 Hernandez-Rebollar Jose L Arm Skeleton for Capturing Arm Position and Movement
US20100023314A1 (en) * 2006-08-13 2010-01-28 Jose Hernandez-Rebollar ASL Glove with 3-Axis Accelerometers
EP2067119A2 (de) 2006-09-08 2009-06-10 Exbiblio B.V. Optische scanner, zum beispiel tragbare optische scanner
US8566077B2 (en) * 2007-02-13 2013-10-22 Barbara Ander Sign language translator
US20080195373A1 (en) * 2007-02-13 2008-08-14 Barbara Ander Digital Sign Language Translator
US8566075B1 (en) * 2007-05-31 2013-10-22 PPR Direct Apparatuses, methods and systems for a text-to-sign language translation platform
US9282377B2 (en) 2007-05-31 2016-03-08 iCommunicator LLC Apparatuses, methods and systems to provide translations of information into sign language or other formats
JP5628691B2 (ja) * 2009-01-27 2014-11-19 洵人 大木 手話キーボードおよびそれを用いた手話検索装置
EP2399385B1 (de) 2009-02-18 2019-11-06 Google LLC Automatische informationserfassung wie etwa informationserfassung anhand einer dokumentbewussten vorrichtung
US8447066B2 (en) 2009-03-12 2013-05-21 Google Inc. Performing actions based on capturing information from rendered documents, such as documents under copyright
WO2010105246A2 (en) 2009-03-12 2010-09-16 Exbiblio B.V. Accessing resources based on capturing information from a rendered document
US9081799B2 (en) 2009-12-04 2015-07-14 Google Inc. Using gestalt information to identify locations in printed information
US9323784B2 (en) 2009-12-09 2016-04-26 Google Inc. Image search using text-based elements within the contents of images
US8493174B2 (en) * 2010-03-24 2013-07-23 Dharma P. Agrawal Apparatus for instantaneous translation of sign language
CN104599554B (zh) * 2014-12-29 2017-01-25 闽南师范大学 一种基于二维码识别的手语教学系统及方法
CN104599553B (zh) * 2014-12-29 2017-01-25 闽南师范大学 一种基于条形码识别的手语教学系统及方法
JP6795996B2 (ja) * 2017-02-08 2020-12-02 日本放送協会 手話ｃｇ編集装置及びプログラム
US10129395B1 (en) 2017-10-26 2018-11-13 Sorenson Ip Holdings Llc Systems and related methods for visual indication of callee ID information for an incoming communication request in a hearing-impaired environment
WO2019160105A1 (ja) 2018-02-16 2019-08-22 日本電信電話株式会社 非言語情報生成装置、非言語情報生成モデル学習装置、方法、及びプログラム
JP7280512B2 (ja) 2018-02-16 2023-05-24 日本電信電話株式会社 非言語情報生成装置及びプログラム
US20210005218A1 (en) * 2018-02-16 2021-01-07 Nippon Telegraph And Telephone Corporation Nonverbal information generation apparatus, method, and program
CN108766127A (zh) * 2018-05-31 2018-11-06 京东方科技集团股份有限公司 手语交互方法、设备、装置和存储介质
RU197549U1 (ru) * 2018-11-27 2020-05-13 Федеральное государственное автономное образовательное учреждение высшего образования "Национальный исследовательский Томский государственный университет" (ТГУ, НИ ТГУ) Устройство динамической коррекции движения руки человека

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0466154A2 (de) * 1990-07-12 1992-01-15 Hitachi, Ltd. Verfahren und Gerät zum Bewegungsdarstellen eines Multiverbindungsobjektes, graphisches Rechnergerät und Robotersteuergerät
EP0586259A2 (de) * 1992-09-03 1994-03-09 Hitachi, Ltd. System und Verfahren zum Lernen der Zeichensprache

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4884972A (en) * 1986-11-26 1989-12-05 Bright Star Technology, Inc. Speech synchronized animation
JPH06337627A (ja) * 1993-05-27 1994-12-06 Hitachi Ltd 手話通訳装置
JP3169727B2 (ja) * 1993-02-25 2001-05-28 株式会社日立製作所 手話生成装置
US5659764A (en) * 1993-02-25 1997-08-19 Hitachi, Ltd. Sign language generation apparatus and sign language translation apparatus
JPH06337628A (ja) * 1993-05-31 1994-12-06 Hitachi Ltd 手話翻訳確認装置
JP3669515B2 (ja) * 1994-05-16 2005-07-06 株式会社日立製作所 手話通訳装置

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0466154A2 (de) * 1990-07-12 1992-01-15 Hitachi, Ltd. Verfahren und Gerät zum Bewegungsdarstellen eines Multiverbindungsobjektes, graphisches Rechnergerät und Robotersteuergerät
EP0586259A2 (de) * 1992-09-03 1994-03-09 Hitachi, Ltd. System und Verfahren zum Lernen der Zeichensprache

Non-Patent Citations (6)

* Cited by examiner, † Cited by third party
Title
APPLICATIONS AND IMPACTS: INFORMATION PROCESSING '94, HAMBURG, AUG. 28 - SEPT. 2, 1994, vol. 2, 28 August 1994, BRUNNSTEIN K;RAUBOLD E (EDITORS), pages 702-707, XP000478851 SCHULMEISTER R: "COMPUTER ASSISTANCE IN LEARNING SIGN LANGUAGE" *
COMPUTER GRAPHICS, vol. 17, no. 3, July 1983, US, pages 105-114, XP000195811 LOOMIS ET AL: "Computer Graphic Modeling of American Sign Language" *
ELECTRONICS & COMMUNICATIONS IN JAPAN, PART III - FUNDAMENTAL ELECTRONIC SCIENCE, vol. 77, no. 3, March 1994, NEW YORK US, pages 62-74, XP000468348 XU ET AL: "Gesture Description and Structure of a Dictionary for Intelligent Communication of Sign Language Images" *
ESCA WORKSHOP KTH , 31 May 1993 - 2 June 1993, STOCKHOLM, pages 83-85, XP000563114 FUKUDA ET AL: "Design of a system for Electronic Dictionary of Japanese Sign Language" *
EUROGRAPHICS, VIENNA, SEPT. 2 - 6, 1991, no. CONF. 12, 2 September 1991, POST F H;BARTH W, pages 33-45, XP000300197 MANJULA PATEL ET AL: "FACES: FACIAL ANIMATION, CONSTRUCTION AND EDITING SYSTEM" *
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS, COMMUNICATIONS AND COMPUTER SCIENCES, vol. E75 - A, no. 2, 1 February 1992, pages 112-121, XP000301804 TAKAO KUROKAWA: "GESTURE CODING AND A GESTURE DICTIONARY FOR A NONVERBAL INTERFACE" *

Also Published As

Publication number Publication date
JPH08315185A (ja) 1996-11-29
US5990878A (en) 1999-11-23
EP0743619B1 (de) 2001-01-17
DE69611570D1 (de) 2001-02-22
EP0743619A3 (de) 1997-02-26

Similar Documents

Publication Publication Date Title
US5990878A (en) Sign language editing apparatus
US5659764A (en) Sign language generation apparatus and sign language translation apparatus
KR100339764B1 (ko) 애니메이션작성장치및애니메이션작성방법
US5111409A (en) Authoring and use systems for sound synchronized animation
EP0856814B1 (de) Anordnungen zur Anzeige von Bildern von Objekten
US9431027B2 (en) Synchronized gesture and speech production for humanoid robots using random numbers
US20060087510A1 (en) Device and method of keyboard input and uses thereof
EP0835481B1 (de) Verfahren und vorrichtung zum steuern eines beweglichen geräts
JP3906729B2 (ja) 手話教育用システム及び該システムを実現するためのプログラム
Kojima et al. Labaneditor: Graphical editor for dance notation
Benoit et al. Audio-visual and multimodal speech systems
Veen et al. TROTS: A computer graphics system for three-dimensional reconstruction from serial sections
Wachsmuth et al. Lifelike gesture synthesis and timing for conversational agents
JP3169727B2 (ja) 手話生成装置
JP2998663B2 (ja) アニメーション作成装置
JPH08115408A (ja) 手話認識装置
JPH11184370A (ja) 手話情報提示装置
JP4831788B2 (ja) 概念モデル可視化システムとプログラムと記録媒体
JP3505361B2 (ja) アニメーション作成装置および記録媒体
JP3179636B2 (ja) 手話合成装置
JP2000089660A (ja) 手話学習支援装置および手話学習支援プログラムを記録した記録媒体
JP2005099977A (ja) 手話編集方法および装置
JP2697775B2 (ja) 会話システム
JPH1049701A (ja) 多関節構造体の動作取り込み方法
Adamo-Villani et al. A new method of hand gesture configuration and animation

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): DE FR GB

PUAL Search report despatched

Free format text: ORIGINAL CODE: 0009013

AK Designated contracting states

Kind code of ref document: A3

Designated state(s): DE FR GB

17P Request for examination filed

Effective date: 19970227

17Q First examination report despatched

Effective date: 19970421

GRAG Despatch of communication of intention to grant

Free format text: ORIGINAL CODE: EPIDOS AGRA

GRAG Despatch of communication of intention to grant

Free format text: ORIGINAL CODE: EPIDOS AGRA

GRAH Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOS IGRA

GRAG Despatch of communication of intention to grant

Free format text: ORIGINAL CODE: EPIDOS AGRA

GRAH Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOS IGRA

GRAH Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOS IGRA

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): FR GB

RBV Designated contracting states (corrected)

Designated state(s): FR GB

REF Corresponds to:

Ref document number: 69611570

Country of ref document: DE

Date of ref document: 20010222

ET Fr: translation filed
PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

REG Reference to a national code

Ref country code: GB

Ref legal event code: IF02

26N No opposition filed
PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20100519

Year of fee payment: 15

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20100428

Year of fee payment: 15

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20110517

REG Reference to a national code

Ref country code: FR

Ref legal event code: ST

Effective date: 20120131

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20110531

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20110517