GB2464486A

GB2464486A - Control of a robotic arm by the recognition and analysis of facial gestures.

Info

Publication number: GB2464486A
Application number: GB0818942A
Authority: GB
Inventors: Aaron Shafir
Original assignee: Shafir Production Systems Ltd
Current assignee: Shafir Production Systems Ltd
Priority date: 2008-10-16
Filing date: 2008-10-16
Publication date: 2010-04-21
Also published as: WO2010044073A1; GB0818942D0

Abstract

A system for aiding a disabled person in activities such as eating and drinking. The system includes a robotic arm; at least one camera to produce image(s) of the face of the disabled person; an image processing module that processes the image(s); and a control sub-system for controlling the position of the robotic arm based on repositioning of one or more facial elements by the disabled person. The facial elements can be naturally occurring facial features or marks applied to the face, for example stickers which may be visible to the camera(s) but transparent to the eye, or marks projected onto the face by the camera(s).

Description

SYSTEM AND METHOD FOR AIDING A DISABLED PERSON

FIELD OF THE INVENTION

The present invention relates to a system for aiding a disabled person, particularly to aid disabled person with limited use of the upper limbs.

BACKGROUND OF THE INVENTION

Disabled people have difficulties in carrying out simple every day operations such as eating and drinking.

To help such disabled persons, patent application JP09224965 discloses a meal support robot dedicated to handling food. A person having handicap in an upper limb can independently operate the robot by providing a light projecting part for projecting a directional light, a light receiving part for receiving the light, and a holding part which is elastically deformed by contact with an operator and irradiating the light in a desired position on the light receiving part.

U.S. patent 6,961,623 discloses a remote control method for use by disabled person, e.g. to actuate a muscle stimulator cuff, which involves triggering a signal by using mechanical vibrations detected to control operation of the device or process.

SUMMARY OF THE INVENTION

According to one aspect of the present invention there is provided a system for aiding a disabled person comprising: at least one robotic arm; at least one camera for producing at least one image of the face of the disabled person; an image processing module for processing the image; and a control sub-system for controlling the position of the at least one robotic arm, based on repositioning of one or more facial elements by the disabled person.

The term "facial elements" herein the specification and claims is used in its broadest sense and includes: facial movement or gestures (e.g. eye, ears, mouth and nose; smiling, tongue movement, raising an eyebrow, and the like); head movements (e.g. nodding yes or no or tilting, etc). The term also relates to the use of features of the face including natural features (wrinkles, the tip of the nose, edge of a lip, and so on) and artificial features such as marks applied on the face (e.g. a sticker with an "x" written placed on the face or an "x" projected onto the face such as by a projector, laser, etc., which capability according to some embodiments is incorporated into the camera(s)).

According to another aspect, the present invention relates to a method for aiding a disabled person comprising detecting one or more facial elements of the disabled person and moving an appliance attachable to a controllable robotic arm with corresponding to the detected facial movement.

The present system and method can be used for a variety of activities including eating/feeding, drawing, writing, teeth brushing and the like.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be understood more clearly and other features and advantages shall become apparent from the following detailed description of exemplary embodiments, with reference to the following figures, wherein: Fig. 1 is an isometric view of an exemplary system in accordance with the present invention as operated by a person in a wheel chair; Fig. 2A is an isometric front view of the disable person face; Fig. 2B is an isometric profile view of the disable person face; Fig. 3 is an isometric side view of a robotic arm; Fig. 4A is a front view of the disable person face including arrows designating the person face movement; and Fig. 4B is a profile view of the disable person face including arrows designating the person face movement.

DESCRIPTION OF EMBODIMENTS OF THE PRESENT INVENTION

Fig. 1 illustrates an embodiment of a system for aiding a disabled person in accordance with the present invention, operated by the disabled person, for example, in a chair or wheel chair 32. A digital camera 34 is attached to a chair support 36 of chair 32. Resting upon a table 38, a food plate 40, a robotic arm 42, and a storage means 44 which stores appliances that can be attached to a robotic catch 45, disposed on an end 46 of the robotic arm. Examples of such appliances include a fork, a cup and so forth. Robotic arm 42 has freedom to move in one or more axes and preferably about those axes. Examples of such robotic arms are the VP 5-axis series and VP 6-axis series arms (DENSO Robotics, 3900 Via Oro Avenue, Long Beach, CA 90810, USA), which can perform in five and six freedoms of movement, respectively.

Typically, an additional camera 48 is attached to table 38. Camera 48 is images a front view of face 58 of disabled person 30 and camera 34 is used for imaging the profile of the disabled person's face. The digital images are stored in a digital memory means, not shown, for further processing.

According to some embodiments, the system further includes a voice recognition sub-system, not shown, for recognizing voice commands of disabled person 30. An example of such voice commands are, "change tool", which commands robot arm 42 to substitute the appliance currently attached to catch 45 with another appliance that is stored in storage means 44. The command!!open catch" for example is another voice command that commands catch 45 of robotic arm 42 to open.

An isometric front view and isometric profile view of disabled person 30 is shown in Figs. 2A and 2B respectively, to which references are now made.

In this example, on the face 58 of the disabled person 30 are placed four marks 60, 62, 64 and 66 (e.g. via stickers or projected thereon). These marks can be natural marks appearing on face 58, facial gestures or marks artificially placed on the disabled person face 58. Marks 60, 62, 64 and 66 are detected by cameras 34 and 48 as known in the art, implemented by an image processing module, not shown. Examples of useful natural marks are wrinkles and moles.

In some embodiments of the present invention, a transparent sticker which is invisible to the human eye but is detectable by cameras 34 and 48 can be applied on the disabled person's face 58. Mark 60 is positioned on the forehead of disabled person 30. The disabled person moving his forehead is accompanied by a corresponding movement of mark 60. Mark 64 is disposed on the chin of the disabled person. When the disabled person moves his chin, mark 60 moves as well. Marks 62 and 66 are positioned on the cheeks of the disabled person. In some embodiments of the present invention marks 62 and 66 are moved by moving the tongue of the disabled person towards the person's right and left cheeks, respectively.

A side view of robotic arm 42 is shown in Fig. 3 to which reference is now made. Robotic arm 42 can be described as being controllable with reference to a Cartesian grid. Double headed arrow 70 designates the is movement direction of robotic arm 42 in the x-axis. Double headed arrow 72 designates the movement direction of robotic arm 42 in the z-axis. Double headed arrow 74 designates the movement direction of robotic arm 42 in the y-axis. Rotational movements around axes 70, 72 and 74 are designated by arrows 76, 78 and 80 respectively.

A front and profile view of the face 58 of disabled person 30, including arrows describing the facial movement, is shown in Figs. 4A and 4B to which reference is now made. The movements of the disabled person's head is analyzed by determining the distance between any of marks 60, 62, 64 and 66 before and after a facial movement(s) or facial gestures respectively are performed. Nodding of the disabled person's head up and down, in the direction designated by double headed arrow 90, is accompanied by a corresponding movement of marks 60, 64. Sideways turning of the disabled person's head in the direction designated by double headed arrow 92, is accompanied by a corresponding movement of marks 60 and 64 to the left or to the right.

Sideways tiliting of the disabled person head in the direction designated by double headed arrow 94, is accompanied by a corresponding movement of mark 60 to the right, and mark 64 to the left, or, mark 60 moves to the left and mark 64 moves to the right. The movements of the disabled person's chin in the direction designated by double headed arrow 96 moves mark 64 up or down in respect to mark 60. Movement of the disabled person's chin in the direction designated by arrows 98 is accompanied by a corresponding movement of mark 64 left or right with respect to mark 60. A movement of the disabled person's tongue in the direction designated by arrow 100 can cause mark 66 to move to the left with respect to marks 60 and 64. The movement of the disabled person's tongue in the direction designated by arrow 102 moves mark 62 to the right with respect to marks 60 and 64.

A control sub-system, not shown, controls the movement of robotic arm 42. Robotic arm 42 is controlled/driven by analyzing the aforementioned movement(s) of the head of disabled person 30 and issuing commands to driving mechanisms, not shown, for controlling the robotic arm (Fig. 3). Head is movement in the directions designated by double headed arrow 90 will actuate robotic arm 42 in the directions designated by double headed arrow 70. A movement in direction 92 will actuate robotic arm 42 in the direction designated by double headed arrow 72. A movement in direction 96 will actuate robotic arm 42 in the direction designated by double headed arrow 74. A movement in direction 98 will actuate robotic arm 42 in the direction designated by arrow 76.

A movement in direction 94 will actuate robotic arm 42 in the direction designated by arrow 78. A movement in direction 100 or direction 102 will actuate robotic arm 42 in the direction designated by arrow 80.

In some embodiments of the present invention marks 60, 62, 64 and 66 can be projected by one of the cameras 34 and 48 or an additional other component onto the person's face 58; and robotic arm 42 is controlled by the relative movement of the face and the mark.

Claims

CLAIMS1. A system for aiding a disabled person comprising: * at least one robotic arm; * at least one camera for producing at least one image of the face of said disabled person; * an image processing module for processing said image; and * a control sub-system for controlling the position of said at least one robotic arm, based on repositioning of one or more facial elements by said disabled person.
2. A system as in claim 1, comprising two cameras, wherein one camera images a front view of the face of said disabled person and a second camera images a profile of the face of said disabled person.
3. A system as in claim 1, wherein said system further comprises storage means which stores appliances that can be attached to said at least one robotic arm.
4. A system as in claim 1, wherein said robotic arm is adapted to move in a plurality of axes.
5. A system as in claim 1, further comprises a voice recognition sub-system for recognizing voice commands of said disabled person.
6. A system as in claim 1, wherein the facial element is a natural mark existing on the disabled person's face.
7. A system as in claim 6, wherein said artificial mark is a transparent sticker which is invisible to a human eye but can be detectable by said camera(s).
8. A system as in claim 1, wherein the facial element(s) are applied on the face of the disabled person.
9. A system as in claim 8, wherein the applied facial element(s) are a mark(s) disposed on the face of the disabled person via a sticker.
10. A system as in claim 8, wherein the applied facial element(s) are a mark(s) projected on the face of the disabled person.
11. A system as in claim 10, wherein the camera(s) are adapted to project the mark.
12. A method for aiding a disabled person comprising detecting one or is more facial elements of said disabled person and moving a robotic arm corresponding to movement of the detected facial element(s).
13. A method according to claim 12, wherein prior to detecting one or more facial elements, a facial element is applied to the face of the disabled person.