JP2019530479A - Mobility aid control method, apparatus and system - Google Patents

Abstract

In this release, the user's eyeball image is captured, the user's eyeball motion is identified based on the user's eyeball image, and a movement assist tool operation command is generated to perform an operation corresponding to the user's eyeball motion. There is provided a method for controlling a movement aid including instructing the movement aid to execute. Furthermore, a camera for taking a user's eyeball image, an image processing circuit connected to the camera, for identifying the user's eyeball movement based on the user's eyeball image, and connected to the image processing circuit And a movement assist tool control device including a control circuit for generating a movement assist tool operation command and instructing the movement assist tool to execute an operation corresponding to the user's eye movement. Furthermore, the movement assistance tool control system containing a movement assistance tool and a movement assistance tool control apparatus is provided.

Description

The present application is a Chinese patent application No. 201610398690 filed on June 7, 2016. X claims the priority of the specification, the entire contents of which are hereby incorporated by reference.
The present disclosure relates generally to control technology, and more particularly to a movement assist tool control method, a movement assist tool control device, and a movement assist tool control system.

  In the current mobility aid technology, a conventional wheelchair needs to be operated with a user's hand or foot, or is driven by power and operated by pressing a button. However, this is difficult for people with limb disorders, such as patients with amyotrophic lateral sclerosis who cannot use hands and sounds, so these conventional wheelchairs cannot be used . Thus, there is a need for a wheelchair that can be operated without moving body parts such as legs, arms or hands.

  In order to solve the problems associated with the current mobility aid technology, this disclosure provides a mobility aid control method, a mobility aid control device, and a mobility aid control system.

  This publication discloses as a first aspect, a movement assistance tool control method for controlling a movement assistance tool by a user. The method captures a user's eyeball image, identifies the user's eyeball movement based on the user's eyeball image, generates a movement aid operation command, and performs an operation corresponding to the user's eyeball movement. It includes three steps: instructing the mobility aid to perform.

  According to some embodiments of the present disclosure, identifying the user's eye movement based on the user's eye image includes determining coordinates of at least one pupil from the user's eye image and at least one eye Determining the eye movement of the user by comparing the coordinates of the pupil with the coordinates of at least one of the at least one pre-stored eye image of the user, wherein: Each of the user's pre-stored eyeball images corresponds to one eyeball movement of the user.

  Here, the sub-step of determining the coordinates of at least one pupil from the user's eyeball image is performed based on a gray value difference between the white eye, the iris, and the pupil in the user's eyeball image.

  Here, the sub-step of determining the eye movement of the user by comparing the coordinates of the at least one pupil with the coordinates of at least one pupil of the at least one pre-stored eyeball image comprises the at least one eyeball Determining whether the difference between the coordinates of the pupil and the coordinates of at least one pupil of any eye image stored in advance is within a predetermined range, and storing the coordinates of at least one pupil in advance Determining that the user's eye movement is an eye movement corresponding to any pre-stored eye image when the difference between the coordinates of at least one pupil of any eye image is within a predetermined range; Is further included.

  The method includes identifying a user's eye movement based on the user's eye image, generating a movement aid operation command, and instructing the movement aid to execute an operation corresponding to the user's eye movement; If an eyeball control start command is received from the user during this period, the step of starting eyeball control and whether or not the movement assisting tool is in an operation ready state are determined. Generating a command, instructing the movement assisting tool to adjust to the operation ready state, and causing the movement assisting tool to perform an operation corresponding to the user's eye movement.

  In this method, whether or not to perform an operation corresponding to the user's eye movement is generated after the step of generating the movement assist tool operation command and instructing the movement assist tool to perform an operation corresponding to the user's eye movement. It further includes a step of prompting the user and a step of transmitting a movement assisting tool operation command to the movement assisting tool upon receiving a confirmation command from the user.

  The method further includes a step of ending the eyeball control when receiving an eyeball control end command from the user after the step of transmitting the movement assisting device operation command to the movement assisting device when receiving a confirmation command from the user.

  In any of the above-described embodiments, the eye movement includes leftward, rightward, upward, and downward directions corresponding to the leftward movement, the rightward movement, the forward movement, and the backward movement, respectively.

  In this disclosure, as a second aspect, a camera for photographing a user's eyeball image, an image processing circuit connected to the camera and identifying a user's eyeball operation based on the user's eyeball image, and an image processing circuit And a control circuit for generating a movement assist tool operation command and instructing the movement assist tool to execute an operation corresponding to the user's eye movement.

  In some embodiments of the mobility aid control device, the image processing circuit includes a coordinate determination subcircuit for determining coordinates of at least one pupil from the user's eyeball image, and coordinates of the at least one pupil. An action determining sub-circuit for determining a user's eye movement by comparing to at least one pupil coordinate of at least one pre-stored eye image, wherein the user's pre-stored Each eyeball image corresponds to one eyeball movement of the user.

  In some embodiments of the present disclosure, the movement assist tool control device is connected to the image processing circuit and the movement assist tool is ready for operation after the user's eye movement is identified by the image processing circuit. It further includes an operation preparation circuit that determines whether or not and receives an eyeball control start command from the user. When the movement assist tool is not in the operation ready state, the operation preparation circuit generates an operation preparation command, instructs the movement assist tool to adjust to the operation ready state, and performs an operation corresponding to the eye movement of the user. The movement aid is executed.

  In some embodiments of the present disclosure, the movement assist device control device prompts the user to execute an operation corresponding to the user's eye movement after the user's eye movement is identified by the image processing circuit. And a transmission circuit for transmitting a movement assisting tool operation command to the movement assisting tool upon receiving a confirmation command from the user.

  The movement assist tool control device receives an eyeball control end command from the user, generates an operation end command based on the eyeball control end command from the user, and stops the movement assist tool and shuts down the transmission circuit. A termination circuit is further included.

  In some embodiments of the present disclosure, the mobility aid control device further includes a communication circuit connected to the camera and the image processing circuit for transmitting the user's eyeball image to the image processing circuit.

  In any embodiment of the mobility aid control device, the camera is located in goggles worn by the user.

  The present disclosure provides, as a third aspect, a movement assistance tool system including a movement assistance tool and a movement assistance tool control device based on any of the above-described embodiments.

  In the movement assistance tool system, the movement assistance tool includes at least one wheel for providing a movement device for the movement assistance tool, a motor for driving at least one wheel, and a command of the movement assistance tool control device. A motor driver connected to the output side for controlling the motor.

  In some embodiments of the mobility aid system, the at least one wheel comprises at least one omnidirectional wheel. Here, the at least one omnidirectional wheel includes at least one mecanum wheel.

  According to some embodiments of the present disclosure, the movement assist device system is connected to a stop button for receiving a forced stop command, and the stop button and the motor driver, respectively, and when the forced stop command is received from the stop button, And a safety control panel for transmitting a motor stop command to the motor driver.

  Other embodiments will become apparent with reference to the following description and attached drawings.

  In order to more clearly describe some embodiments, the following is a brief description of the drawings. The drawings in the following description are merely illustrative of some embodiments. Those skilled in the art can make other drawings of other embodiments clear based on these drawings.

1 is a schematic diagram of a mobility aid control device according to some embodiments of the present disclosure. FIG. Fig. 6 shows goggles of a mobility aid control device according to some embodiments of the present disclosure. 1 is a schematic diagram of a mobility aid control device according to some embodiments of the present disclosure. FIG. The eyeball image image | photographed previously when the user is looking forward is shown. The eyeball image image | photographed previously when the user is looking at the left is shown. 1 is a schematic diagram of a mobility aid control device according to some embodiments of the present disclosure. FIG. 1 is a schematic diagram of a mobility aid control device according to some embodiments of the present disclosure. FIG. 1 is a schematic diagram of a mobility aid control device according to some embodiments of the present disclosure. FIG. 1 is a schematic diagram of a mobility aid system according to some embodiments of the present disclosure. FIG. 1 illustrates a mobility aid system according to some embodiments of the present disclosure. It is the schematic of the movement assistance tool system shown in FIG. 2 shows an operational flowchart of a wheelchair system according to some embodiments of the present disclosure. The coordination of the four Mecanum wheels in a wheelchair system that realizes various wheelchair movements is shown.

  In the following, with reference to the drawings of various embodiments disclosed herein, the technical solutions of the presently disclosed embodiments will be described in a clear and completely understandable manner.

  It will be appreciated that the described embodiments are only a part of the published embodiments and not all. Based on the described embodiments of the present disclosure, those skilled in the art can obtain other embodiments within the protection scope of the present disclosure.

  In order to solve the problem that it is generally inconvenient or impossible to use conventional mobility aids such as wheelchairs for people with disabilities or other disabilities, Provided are a control method, a movement assist device control apparatus, and a movement assist device system.

  This publication discloses a movement auxiliary tool control device as one mode. FIG. 1 is a schematic diagram of a mobility aid control device according to some embodiments of the present disclosure. As shown in FIG. 1, the movement assist tool control device includes a camera 11, an image processing circuit 12, and a control circuit 13.

  The camera 11 captures a user's eyeball image. The image processing circuit 12 is connected to the camera 11 and identifies the user's eye movement when receiving the user's eyeball image. The control circuit 13 generates a movement assistance tool operation command (that is, a movement assistance tool control command for executing a certain operation) based on the user's eye movement.

  In the movement assisting instrument control device as described above, a plurality of eye movements and a plurality of movement assisting instrument operation commands can be preset and stored in advance. Each of the plurality of eye movements corresponds to a plurality of movement aid operation commands.

  As an example, the movement assistance tool is a wheelchair, and Table 1 shows correspondence relationships between a plurality of eye movements and a plurality of wheelchair operation commands.

  As shown in Table 1, a plurality of preset eye movements stored in advance include “blink once”, “blink twice”, “blink three times”, “look left”, “right” Includes "see", "look up", and "look down". The action “Look Left” corresponds to an instruction to turn the wheelchair to the left. The action of “look right” corresponds to an instruction to turn the wheelchair to the right. The action of “looking up” corresponds to an instruction to move the wheelchair forward. The action of “looking down” corresponds to an instruction to move the wheelchair backward. The operation of “blinking once” corresponds to a confirmation command (that is, a command for instructing confirmation). The “twice blink” operation corresponds to a wheelchair stop instruction. The “blink three times” operation corresponds to a command to start an eyeball control operation.

  The correspondence between the eyeball movement and the wheelchair operation command is arbitrary, and can be set based on practical conditions. Such correspondence is set before the wheelchair is put on the market or customized by the user. Further, the movement assist tool may be a balancer beagle (eg, Segway) or an electric unicycle. There is no limit here.

  The following is a detailed description of a mobility aid control device taking a wheelchair as an example. During the operation, the camera 11 is used to capture the user's eyeball image, and the user's eyeball image is transmitted to the image processing circuit 12 by wired or wireless communication. Upon receiving the user's eyeball image, the image processing circuit 12 identifies the user's eyeball movement by image identification.

  Next, the control circuit 13 inquires a correspondence table including a correspondence relationship that is set in advance and stored in advance, and generates a corresponding wheelchair operation command based on the user's eyeball motion. To do.

  For example, after image processing, if the image processing circuit 12 identifies that the eye movement is “look left”, the control circuit 13 generates a command to rotate the wheelchair to the left and transmits it to the power mechanism of the wheelchair. The wheelchair turns left.

  According to the movement assisting device control apparatus as described above, the operation of the movement assisting device by eyeball control can be realized without having to move body parts such as legs, arms, and hands. Therefore, it is possible to effectively solve the problem that it is difficult for a disabled person or a person with a handicap to operate a conventional mobility aid such as a wheelchair.

  As shown in FIG. 2, according to some embodiments of the present disclosure, the movement assist device control device as described above may further include goggles, and the camera 11 may be disposed in the goggles. Goggles provide convenience for the user to wear and can block the curiosity and attention from others by blocking the movement of the user's eyeballs while operating the mobility aids .

  In the preferred embodiment shown in FIG. 2, the camera 11 is attached to one lens of goggles. On one side of the camera 11, a communication circuit (for example, Bluetooth (registered trademark) wireless communication circuit 111) and a power source (for example, a battery 131) are arranged. The power supply supplies power to the camera 11 and the communication circuit. The eyeball image photographed by the camera 11 is transmitted to the image processing circuit 12 via the communication circuit.

  FIG. 3 illustrates a mobility aid control device according to some embodiments of the present disclosure. As shown in FIG. 3, the image processing circuit 12 includes a coordinate determination subcircuit 121 and an operation determination subcircuit 122.

  The coordinate determination sub-circuit 121 determines the coordinates of the user's pupil from the user's eyeball image captured by the camera 11 based on the difference between the gray values of the white eye, the iris, and the pupil of the eyeball.

  The action determination subcircuit 122 compares the coordinates of the user's pupil in the current eyeball image (that is, the user's eyeball image captured by the camera 11) with a plurality of prestored eyeball coordinates. Determining whether or not the difference between the pupil coordinates of the user in the current eyeball image and the pupil coordinates of any of the eyeball images stored in advance is within a predetermined range, It is determined that the user performs an eye movement corresponding to one pre-stored user's eye image. Here, the plurality of eyeball images stored in advance are user's eyeball images captured in advance.

  The process of determining the coordinates of the user's pupil from the user's eyeball image is a conventional method. Thus, this process includes:

  First, the coordinates of the user's pupil are determined based on the difference between the gray values of the user's white eye, iris, and pupil in the user's eyeball image captured by the camera 11.

  The above steps are realized by the following sub-steps. In the first sub-step, the Otsu method for binarization (maximizing interclass variance) is used to segment the image, thereby determining the iris edges. In the second sub-step, the coordinates of the iris center are determined by gray projection. Finally, in the third sub-step, the coordinates of the center of the pupil are determined by the Hough transform based on the circle and the least square method.

  Next, the coordinates of the pupil of the user obtained in the first step are compared with the coordinates of the pupils of a plurality of eyeball images stored in advance. Here, the plurality of eyeball images stored in advance are, for example, images having the user's eye movements that have been captured in advance and that are used for the first time by the user in the trial run stage.

  For example, FIG. 4 shows a user's pre-captured eyeball image when looking straight ahead, and the coordinates of the pupil are designated as the origin of the coordinates. FIG. 5 shows a pre-captured eyeball image when the user is looking left.

  When the eyeball rotates to the left, that is, when the user is looking at the left, the relative position of the pupil moves from the center (ie, origin) to the left. Similarly, when the eyeball rotates to the right, up, or down, the position of the pupil moves correspondingly based on the relatively dark color of the pupil.

  The position of the pupil (that is, the coordinates) in the entire eyeball can be accurately determined by the image analysis method (that is, image identification) shown in the first step described above. Then, in the second step, the pupil coordinates obtained in the first step are compared with the pupil coordinates in a plurality of eyeball images stored in advance.

  Compared with the pre-captured eyeball image shown in FIG. 5, the pupil coordinates obtained in the first step are within the predetermined range (that is, the area surrounded by the dotted line) because the pupil coordinates obtained in the first step are within a predetermined range (ie, the area surrounded by the dotted line) And the pupil coordinates of a pre-captured eyeball image (as shown in FIG. 5) are considered to be within a predetermined range, and eventually the user's eyeball action is “Look Left Is determined.

  FIG. 6 is a schematic diagram of a mobility aid control device according to some other embodiments of the present disclosure. As shown in FIG. 6, the movement assist tool control device further includes an operation preparation circuit 14. The operation preparation circuit 14 is connected to the image processing circuit 12, and after the image processing circuit 12 identifies the eye movement, it is determined whether or not the movement assisting device is in a preset operation preparation state and input from the user. An eyeball control start command is accepted. When not in the operation preparation state, the operation preparation circuit 14 generates an operation preparation command. The operation preparation command instructs the movement assist tool to switch from the current state to the operation preparation state. The movement assist tool is suitable for operation according to a movement assist tool operation command corresponding to the user's eye movement when in the operation preparation state.

  Before the movement assist tool performs an operation corresponding to the user's eye movement, it is necessary to determine the current state of the movement assist tool and whether or not it is appropriate to perform the operation. If it is not appropriate, the movement assisting tool needs to adjust its state to switch to the operation ready state. Thereby, the movement assistance tool can perform said operation safely, and can prevent an accident. The operation preparation state can be set in advance and differs depending on the operation to be executed.

  Taking a wheelchair as an example, assume that the wheelchair is currently moving forward at high speed. When the user instructs the wheelchair to turn left by eye movement, the user is prompted to “whether or not to start eyeball control”. The user transmits an eyeball control start command by eyeball movement (for example, “blink once”). Thereafter, after imaging by the camera 11 and image processing by the image processing circuit 12, the operation preparation circuit 14 receives an eyeball control start command input from the user, and the current state is not appropriate for performing a “right turn” operation. Confirmed (ie, confirmed that the wheelchair is not ready for operation), generates an operation preparation command, and then instructs the wheelchair to decelerate or stop, and the wheelchair performs a “left turn” operation corresponding to the eye movement. Prepare to avoid accidents when turning left in a wheelchair.

  The movement aid obtains the current state by feeding back or recording the result of the previous operation. The operation preparation circuit 14 does not generate an operation preparation command when it is determined that the current state of the movement assist tool is appropriate for performing an operation corresponding to the eyeball movement. The wheelchair performs the operation as it is.

  The current state of the movement aid includes, but is not limited to, the movement speed, movement direction, and angle of each wheel of the movement aid.

  FIG. 7 is a schematic diagram of a mobility aid control device according to some other embodiments of the present disclosure. As shown in FIG. 7, the movement assistance tool control device is based on the movement assistance tool control device shown in FIG. 6, and further includes a prompt circuit 15 and a transmission circuit 16.

  The prompt circuit 15 prompts the user to execute an operation corresponding to the eyeball movement after the image processing circuit 12 identifies the user's eyeball movement. The transmission circuit 16 receives a confirmation instruction from the user and transmits a movement assisting tool operation command to the motor driver of the movement assisting tool.

  Here, the prompt circuit 15 can prompt the user by voice, image, or other prompting scheme. Since the transmission circuit 16 transmits the movement assistance tool operation command after receiving a confirmation command from the user, the movement assistance tool operation command can be canceled before transmission, preventing malfunction and improving safety. it can.

  FIG. 8 is a schematic diagram of a movement assist tool control device according to another embodiment of a part of the present disclosure. As illustrated in FIG. 8, the movement assist tool control device is based on the movement assist tool control device illustrated in FIG. 7, and further includes an operation end circuit 17. The operation end circuit 17 receives the eyeball control end command input by the user, generates an operation end command based on the eyeball control end command, stops the movement assist tool, and shuts down the transmission circuit 16. As described above, after the user inputs the eyeball control end command, the operation end circuit 17 instructs the stop of the movement assist tool and shuts down the transmission circuit 16, thereby avoiding an erroneous operation.

  The above-described eyeball control start command, confirmation command, and eyeball control end command can be acquired by the image processing circuit identifying the user's eyeball image captured by the camera.

  With the movement assisting device control apparatus as described above, the operation of the movement assisting device by eyeball control can be realized without the need to move the legs, arms, hands or other parts of the body. In this way, it is possible to effectively solve the problem that it is difficult for persons with disabilities and handicap to operate conventional mobility aids such as wheelchairs, and at the same time, it is possible to guarantee safety and prevent malfunctions. It can be avoided.

  As another aspect, the present disclosure provides a mobility aid system. The mobility aid system includes a mobility aid and a mobility aid control device according to any of the embodiments described above.

  The mobility aid may be a wheelchair. As shown in FIG. 9, in the movement assistance tool system according to some embodiments of the present disclosure, the wheelchair 20 includes an omnidirectional moving wheel 23, a motor 22 for driving the omnidirectional moving wheel 23, and a motor. And a motor driver 21 for controlling the motor 22.

  The command output terminal of the movement assist tool control device 10 is connected to the motor driver 21. Here, the connection between the control device 10 and the motor driver 21 includes communication that can be wired communication or wireless communication. Wireless communication is realized by a wireless adapter.

  In the movement assistance tool system (for example, a wheelchair) as described above, the operation of the movement assistance tool by eyeball control can be realized without the need to move the legs, arms, or other parts of the body. Thus, it is possible to effectively solve the problem that it is difficult for a disabled person or a person with a handicap to operate a conventional mobility aid such as a wheelchair.

  According to some embodiments, the mobility aid of the mobility aid system as described above further includes a stop button and a safety control panel, as shown in FIG. The stop button 171 receives a forced stop command and transmits it to the safety control panel 161. The safety control panel 161 is connected to the stop button 171 and the motor driver 21, receives a forced stop command from the stop button 171, and transmits a motor stop command to the motor driver 21. In some embodiments, the safety control panel 161 is also connected to an alarm indicator 181 and controls the alarm indicator 181 to warn the surrounding environment when a forced stop command is received from the stop button 171.

  The above configuration is used for the following purposes. Since the user of the mobility aid system is usually a person with a disability or handicap, if a situation that requires stopping of the mobility aid (such as an accident) occurs, the operation of the eye control is usually slow. It is convenient and quick for the person to press the stop button, thereby realizing emergency braking of the mobility aid.

This publication provides as another aspect a method for controlling a mobility aid. The method includes the following steps.
Step 1: A user's eyeball image is imaged.
Step 2: The user's eye movement is identified based on the user's eye image by image processing and identification.
Step 3: A movement assistance tool operation command is generated from the user's eye movement.

  In the control method of the movement assist tool as described above, first, after the user's eyeball image is captured, the user's eyeball motion is identified based on the user's eyeball image by image processing and identification, and finally the user's eyeball motion. Is converted into a movement assisting tool operation command. Therefore, the operation of the movement assisting tool by eyeball control can be expressed without having to move the legs, arms, hands, or other parts of the body. Therefore, it is possible to effectively solve the problem that it is difficult for a disabled person or a person with a handicap to operate a conventional mobility aid such as a wheelchair.

  Before the step 3, the method receives an eyeball control start command from the user, determines whether or not the current state of the movement assist tool is a predetermined operation preparation state, and if not the predetermined operation preparation state, Generate an operation preparation command. The operation preparation command instructs the movement assist tool to switch from the current state to the operation preparation state. When the movement assist tool is in an operation preparation state, the movement assist tool performs an operation based on a movement assist tool operation command corresponding to the user's eye movement.

This method further includes the following steps after step 3.
Step 4: Prompt the user whether or not to perform an operation corresponding to the eye movement, receive a confirmation command from the user, and transmit a movement assisting tool operation command to the motor driver of the movement assisting tool.

  Here, since it is the structure which transmits a movement assistance tool operation command after receiving confirmation instruction from a user, a movement assistance tool operation command can be canceled before transmission, preventing malfunctioning and improving safety. it can.

According to some embodiments of the present disclosure, the method for controlling a mobility aid further includes the following steps.
Step 5: An eyeball control end command is received from the user, an operation end command is generated based on the eyeball control end command, the movement assist tool is stopped, and transmission of the movement assist tool operation command to the movement assist tool is terminated. .

  The following is a detailed description of a specific embodiment of the movement assist tool and its control method.

  10 and 11 illustrate a wheelchair system according to some embodiments of the present disclosure.

  As shown in FIGS. 10 and 11, the wheelchair system includes goggles 18 and a wheelchair. The goggle 18 includes a camera 11, a Bluetooth wireless communication circuit 111, and a battery 131, and captures and transmits a user's eyeball image in a real-time mode.

  The wheelchair includes a chair 24, a set of four omnidirectional wheels 23 attached to the bottom of the chair 24, a set of in-wheel motors 221, and a set of motor drivers 21. Each in-wheel motor 221 is connected to the omnidirectional wheel 23 and the motor driver 21. The wheelchair further includes other components including a processor 19, a memory circuit (not shown), a Bluetooth circuit 141, a voice prompt circuit 151, a power source (eg, a battery), and an air switch.

  The wheelchair receives the user's eyeball image and identifies the user's eyeball movement by an image analysis algorithm. The processor 19 transmits a wheelchair operation command corresponding to the user's eye movement. The omnidirectional movement type wheel 23 is adjusted in the moving direction such as forward, backward, and turning.

  It should be noted that the processor 19 can implement various circuit functions as described above in some embodiments of the present disclosure by executing predetermined software programs. For example, the processor 19 can realize the functions of the image processing circuit 12, the control circuit 13, the operation preparation circuit 14, and the operation end circuit 17, and can partially realize the function of the transmission circuit 16.

  Table 1 shows the correspondence between eye movements and wheelchair operation commands. As shown in the table, the eye movements of “Look Left”, “Look Right”, “Look Up” and “Look Down” move the wheelchair to the left, right, forward, back, respectively. Corresponding to The eye movements of “blink once”, “blink twice”, and “blink three times” correspond to confirmation, stop, and start, respectively. Note that the above correspondence can be customized.

  FIG. 12 shows an operational flowchart of a wheelchair system according to some embodiments of the present disclosure. In the wheelchair system, goggles integrated with the camera 1 are worn by the user. When activated, the camera 11 captures a real-time eyeball image of the user at a speed of 10 times / second (the speed can be customized). The user's eyeball image is transmitted to the processor 19 via Bluetooth wireless communication. The processor 19 processes the user's eyeball image in real time to identify the user's eye movement.

  The user can perform access control by blinking three times before starting control of the wheelchair. When the user turns the eyeball to the left, right, up, or down once, the system performs the left, right, forward, or voice by voice based on the eyeball image identification result and the correspondence between the eyeball motion and the wheelchair operation command. Prompt whether or not to move backwards. After the user blinks once for confirmation, the wheelchair performs an operation corresponding to the movement of the eyeball until the user wants to stop. The user blinks twice and the control for the wheelchair ends.

  Note that for the same wheelchair operation command, each omnidirectional wheel has a different nominal motion.

  The omnidirectional wheel in the embodiment described above is preferably a Mecanum wheel. Mecanum wheels are based on conventional wheels and include a plurality of freely rotatable miniature rollers disposed on the wheel rim and having an angle α (usually 45 degrees).

  Thus, when the wheel (ie, the central wheel) is rolling, the small roller has a lateral movement. By coordinating the four Mecanum wheels of the wheelchair, the wheelchair system can achieve omnidirectional movement. Moreover, the wheelchair system provided with the Mecanum wheels as described above has advantages that the supporting force is strong, the structure is simple, and the motion control is flexible, and is suitable for a wheelchair.

FIG. 13 illustrates the coordination of the four wheels (ie, Mecanum wheels) in a wheelchair that realizes various main movements of the wheelchair.
When the wheelchair moves forward, all four wheels (ie Mecanum wheels) rotate forward.
When the wheelchair moves backwards, all four wheels rotate backwards.
When the wheelchair moves to the right, the left front wheel and the right rear wheel rotate forward, and the right front wheel and the left rear wheel rotate backward.
When the wheelchair moves to the left, the left front wheel and the right rear wheel rotate backward, and the right front wheel and the left rear wheel rotate forward.
When the wheelchair rotates clockwise, the left front wheel and the left rear wheel rotate forward, and the right front wheel and the right rear wheel rotate backward.
When the wheelchair rotates counterclockwise, the left front wheel and the left rear wheel rotate backward, and the right front wheel and the right rear wheel rotate forward.
When the wheelchair moves to the right front, the left front wheel and the right rear wheel rotate forward, and the right front wheel and the left rear wheel do not rotate.
When the wheelchair moves to the left front, the right front wheel and the left rear wheel rotate forward, and the left front wheel and the right rear wheel do not rotate.

  In the above, the forward or backward rotation of each Mecanum wheel is the rotation direction of the central wheel of each Mecanum wheel.

  In the Mecanum wheel, each small roller rotates independently. When the Mecanum wheel is rotating, the combined speed of the Mecanum wheel is perpendicular to the small roller and can be divided in the longitudinal and lateral directions.

  In this specification, the wheelchair moves to the right as an example. As shown in the schematic diagram of “move right” in FIG. 13, the direction of each arrow in addition to each Mecanum wheel indicates the rotation direction of the corresponding Mecanum wheel (that is, the rotation direction of the central wheel of the Mecanum wheel). . When the speed of each Mecanum wheel is divided into the vertical direction and the horizontal direction, it can be seen that the speed along the longitudinal direction is removed and only the speed in the horizontal direction (right direction) remains. In this way, the wheelchair can be moved to the right.

  The above is a known technique in this field, and the description thereof is omitted here for the sake of simplicity.

  In the wheelchair system as described above, control of the wheelchair can be realized by monitoring and identifying the user's eye movement including blinking and movement of the eyeball. Specifically, since the omnidirectional wheel is used in the wheelchair system, control of the wheelchair movement is realized by a specific eye movement and corresponding coordinated rotation of each wheel even if the turning radius is zero.

One control mechanism according to some embodiments of the present disclosure can be as follows.
The user's real-time eyeball image is compared with a predetermined image sample previously determined by the camera to determine changes in the coordinates of the pupil center. Next, the user is prompted by a voice whether or not to take a certain action. After confirmation from the user, the processor controls each motor by a motor driver to coordinately control each omnidirectional wheel, and transmits a command for realizing the operation of the wheelchair user corresponding to the eyeball.

  Here, the eye movements of “Look Left”, “Look Right”, “Look Up”, and “Look Down” correspond to the movement of the wheelchair to the left, to the right, to the front, to the back, respectively . In order to avoid interference of unintentional movement of the eyeball, the validity of the movement can be confirmed by blinking.

  In any embodiment of the present disclosure as described above, the number of steps is not limited to the limitation of the order of steps. Any changes made by those skilled in the art with respect to the order of the steps should be considered within the scope of this disclosure.

  Various embodiments of the present disclosure are described progressively, and descriptions of the same or similar parts in different embodiments can be referred to each other.

  It should be noted that all or part of the steps of the above method can be realized by a computer program that indicates the corresponding hardware. Here, the computer program can be stored in a computer-readable storage medium. At run time, the computer program can include the steps of the methods described in any of the above embodiments. The storage medium is a magnetic disk, a CD, a read only memory (ROM), a random access memory (RAM), or the like, and is not limited here.

  All references cited in this publication are incorporated by reference in their entirety. Although specific embodiments have been described in detail above, the description is for illustrative purposes only. Thus, it should be understood that many of the aspects described above are not intended as essential or essential elements unless specifically stated otherwise.

In addition to the above, various modifications and equivalent operations corresponding to the disclosed aspects of the exemplary embodiments can be made by those skilled in the art. The scope should be accorded the broadest interpretation so as to encompass such modifications and equivalent constructions without departing from the spirit and scope of the invention as set forth in the following claims.

Claims (20)

Taking a user's eyeball image,
Identifying the user's eye movement based on the user's eye image;
A method for controlling a movement assistance tool by a user, including generating a movement assistance tool operation command and instructing the movement assistance tool to perform an operation corresponding to the user's eye movement. Identifying the user's eye movement based on the user's eye image;
Determining coordinates of at least one pupil from the user's eyeball image;
Determining the eye movement of the user by comparing the coordinates of at least one pupil with the coordinates of at least one pupil of at least one pre-stored eye image of the user;
2. The method of claim 1, wherein each of the user's pre-stored eyeball images corresponds to one eye movement of the user. Determining the coordinates of at least one pupil from the user's eyeball image is
The method of claim 2, wherein the method is performed based on differences in gray values between white eyes, irises, and pupils in the user's eyeball image. Determining the eye movement of the user by comparing the coordinates of the at least one pupil with the coordinates of the at least one pupil of the at least one pre-stored eye image;
Determining whether the difference between the coordinates of the at least one pupil and the coordinates of at least one pupil of any previously stored eyeball image is within a predetermined range;
If the difference between the coordinates of the at least one pupil and the coordinates of at least one pupil of any eyeball image stored in advance is within a predetermined range, the eye movement of the user is The method of claim 2, comprising determining that the eye movement corresponds to an eyeball image. Identifying the user's eye movement based on the user's eye image, generating a movement assist tool operation command, and instructing the movement assist tool to perform an operation corresponding to the user's eye movement. Between,
Receiving an eyeball control start command from the user, starting eyeball control;
Determine whether or not the mobility aid is in an operation ready state, and if the mobility aid is not in an operation ready state, generate an operation preparation command and instruct the mobility aid to adjust to the operation ready state. The method according to claim 1, further comprising causing the movement assisting tool to perform an operation corresponding to the eyeball movement. After generating a movement assistance tool operation command and instructing the movement assistance tool to perform an operation corresponding to the user's eye movement,
Prompting the user to perform an operation corresponding to the user's eye movement;
The method according to claim 5, further comprising: transmitting a movement assistance tool operation command to the movement assistance tool when receiving a confirmation command from the user. After receiving a confirmation command from the user and transmitting a movement aid operation command to the movement aid,
The method according to claim 6, further comprising ending the eyeball control upon receiving an eyeball control end command from the user.   The method according to claim 1, wherein the eye movement includes leftward, rightward, upward, and downward directions corresponding to leftward movement, rightward movement, forward movement, and backward movement, respectively. A camera for taking images of the user's eyeballs;
An image processing circuit connected to the camera for identifying the user's eye movement based on the user's eye image;
A movement assist tool control device connected to the image processing circuit and including a control circuit for generating a movement assist tool operation command and instructing the movement assist tool to perform an operation corresponding to the user's eye movement. A coordinate determination subcircuit for determining the coordinates of at least one pupil from the user's eyeball image;
An action determination sub-circuit for determining the eye movement of the user by comparing the coordinates of the at least one pupil with the coordinates of at least one pupil of the at least one pre-stored eye image of the user; Including
Here, each of the eyeball images stored in advance by the user corresponds to the movement of one eyeball of the user. Connected to the image processing circuit;
After the user's eye movement is identified by the image processing circuit and receiving an eye control start command from the user, it is determined whether or not the movement assisting tool is in an operation ready state,
When the movement assist tool is not in the operation ready state, an operation preparation command is generated, the movement assist tool is instructed to adjust to the operation ready state, and the operation corresponding to the user's eye movement is executed by the movement assist tool. The movement assist tool control apparatus according to claim 9, further comprising a preparation circuit. A prompt circuit for prompting the user to execute an operation corresponding to the user's eye movement after the user's eye movement is identified by the image processing circuit;
The movement assistance tool control device according to claim 9, further comprising a transmission circuit for transmitting a movement assistance tool operation command to the movement assistance tool when receiving a confirmation command from the user.   An operation end circuit for receiving an eyeball control end command from the user, generating an operation end command based on the eyeball control end command from the user, and stopping the movement aid and shutting down the transmission circuit. 12. The movement assistance tool control device according to 12.   The movement assistance tool control device according to claim 9, further comprising a communication circuit connected to the camera and the image processing circuit for transmitting an eyeball image of the user to the image processing circuit.   The movement assistance tool control device according to claim 9, wherein the camera is located in goggles worn by a user.   The movement assistance tool system containing the movement assistance tool as described in any one of Claims 9-15, and a movement assistance tool control apparatus. The movement aid is
At least one wheel for providing a moving device for the movement aid;
A motor for driving the at least one wheel;
The movement assistance tool system according to claim 16, further comprising: a motor driver connected to a command output side of the movement assistance tool control device and controlling the motor.   The movement assistance tool system according to claim 17, wherein the at least one wheel includes at least one omnidirectional wheel.   19. The mobility aid system of claim 18, wherein the at least one omnidirectional wheel comprises at least one mecanum wheel. A stop button for receiving a forced stop command;
The movement assistance tool system according to claim 16, further comprising a safety control panel connected to a stop button and a motor driver, respectively, and a safety control panel for transmitting the motor stop command to the motor driver when the forced stop command is received from the stop button. .

Images