KR102594335B1 - Artificial Intelligence-based Electric Wheelchair Control Method and Device and System Therefor - Google Patents

Abstract

According to one aspect, an artificial intelligence-based electric wheelchair control method includes pairing with a user device by performing a boot procedure according to power application, obtaining user information from a server through the paired user device, and based on the user information. Determining the motion control mode of the electric wheelchair, recognizing a predefined start command through a sensor corresponding to the determined motion control mode with the AOD (Always On Display) screen inactive, and following the start command Activating the deactivated screen, outputting a user interface screen for controlling the electric wheelchair to the display, waiting for a user input signal, and performing machine learning on the user input signal input through the sensor to identify the user. And performing authentication and based on successful user authentication, identifying a control command corresponding to the user input signal to control the operation of the electric wheelchair, wherein the user information includes disability type information and health. It may contain at least one of status information.

Description

Artificial Intelligence-based Electric Wheelchair Control Method and Device and System Therefor}

The present invention relates to electric wheelchair control technology, and in detail, to an artificial intelligence-based electric wheelchair control method capable of adaptively controlling an electric wheelchair according to the user's disability or health condition based on artificial intelligence, and devices and systems therefor. It's about.

Recently, cars are equipped with lane departure warning (LDW), adaptive cruise control (ACC), forward-collision warning (FCW), and blind-spot detection (BSD) systems. Safety and convenience are provided to drivers through various advanced driver assistance systems (ADAS).

The most important technology in this intelligent system is the technology to recognize external objects. It uses various sensors such as cameras, radar, ultrasonic waves, and lidar to detect driving and parking situations through lane detection, vehicle and pedestrian detection in front, rear, and blind spots, etc. It can provide safety and convenience.

Cameras are essential devices used in level 2 or higher autonomous driving environments such as collision prevention, lane keeping, and parking assistance, and their proportion is increasing as many countries require their installation.

Additionally, these sensors can be linked to the V2X (Vehicle to Everything) communication system to effectively avoid accidents even in situations that are difficult to detect with environmental sensors, such as intersections.

Sensor fusion technology that combines sensor information to overcome the weaknesses of individual sensors is being actively developed, and technology development for next-generation sensors such as stereo cameras and 3D image sensors is also being actively developed.

Recently, sensor technologies applied to vehicles are also being adopted and utilized in electric wheelchairs.

Recently released electric wheelchairs use cameras and ultrasonic sensors to detect obstacles in front that are difficult for the user to perceive - for example, bumps, stairs, puddles, cliffs, manholes, etc. - and provide a warning alarm to the user or automatically A stopping technology is being used.

However, there is a problem in that people with hearing impairment cannot hear the warning alarm, and people with visual impairment cannot see the warning alarm displayed on the display device.

In addition, in the case of patients with cauda equina, amyotrophic lateral sclerosis (ALS), etc., it is difficult to properly control steering devices such as joysticks provided in electric wheelchairs in response to warning alarms, making it impossible to drive while avoiding obstacles.

In particular, since electric wheelchairs used in medical or nursing facilities are shared by many people, there is a problem in that they may be difficult to use depending on the user's disability or health condition.

Therefore, there is a demand for an artificial intelligence-based electric wheelchair that operates adaptively in consideration of the user's disability status and health condition.

1. Korean Patent Publication No. 10-2016-0091193 (2016.08.02) 2. Korean Patent Publication No. 10-2146206 (2020.08.12) 3. Korean Patent Publication No. 10-2018-0076536 (2018.07.06)

The purpose of the present invention is to provide an artificial intelligence-based electric wheelchair control method and a device and system therefor.

Another object of the present invention is to provide an artificial intelligence-based electric wheelchair capable of automatically switching operation modes based on the user's disability status and health status.

Another purpose of the present invention is to obtain template data for each authentication method through prior learning of sensing information collected from various authentication sensors, and to compare the characteristic data for each authentication method obtained in real time with the corresponding template data to determine the authentication weight for each authentication method. The aim is to provide an artificial intelligence-based electric wheelchair control method and a device and system therefor that can perform customized user authentication by determining and setting an optimal authentication level for each user based on the determined authentication weight.

Another object of the present invention is to obtain information about the user's disability type and health status from the server through a user device paired with the electric wheelchair, and adaptively set the motion control mode of the electric wheelchair based on the obtained user information. By doing so, an artificial intelligence-based electric wheelchair control method that can provide user convenience and driving safety and a device and system therefor are provided.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below.

According to one aspect, an artificial intelligence-based electric wheelchair control method includes pairing with a user device by performing a boot procedure according to power application, obtaining user information from a server through the paired user device, and based on the user information. Determining the motion control mode of the electric wheelchair, recognizing a predefined start command through a sensor corresponding to the determined motion control mode with the AOD (Always On Display) screen inactive, and following the start command Activating the deactivated screen, outputting a user interface screen for controlling the electric wheelchair to the display, waiting for a user input signal, and performing machine learning on the user input signal input through the sensor to identify the user. And performing authentication and based on successful user authentication, identifying a control command corresponding to the user input signal to control the operation of the electric wheelchair, wherein the user information includes disability type information and health. It may contain at least one of status information.

In an embodiment, the method further includes setting an authentication application user mode, wherein the user authentication includes user-dependent authentication that performs authentication only for specific preset users and authentication for all users registered in the authentication database. May include user-independent authentication that performs:

In an embodiment, the operation control mode may include at least one of a manual control mode, a voice recognition control mode, a facial recognition control mode, a gesture recognition control mode, and an autonomous driving control mode.

In an embodiment, the method includes determining a view type corresponding to a recognized voice command based on the operation control mode being a voice recognition control mode and installing at least one camera to receive an image corresponding to the determined view type. Determining and configuring a view screen based on the determined image obtained from the at least one camera and displaying the view screen on the screen of the user device or the display, wherein the view type is front It may include at least one of the view, left side view, right side view, rear view, left/right side view, front surround view, rear surround view, and bird's eye view.

In an embodiment, the method further includes detecting a nearby obstacle using at least one of a SPAS (Smart Parking Assistance System) sensor, an ultrasonic sensor, and a radar provided while the electric wheelchair is traveling, and the nearby obstacle is detected. Based on this, information about the detected nearby obstacle may be displayed on one side of the view screen.

In an embodiment, the sensor includes a microphone, a camera, and a biometric sensor, and the biometric sensor includes at least one of a fingerprint sensor, an iris sensor, and a body pressure sensor, and a user registration procedure according to selection of a predetermined user menu on the user interface screen. Disclosed is that the user registration procedure includes extracting first characteristic data by recognizing the user's voice through the microphone and extracting second characteristic data by recognizing the user's iris from an image captured through the camera. extracting third characteristic data by recognizing the user's fingerprint through a fingerprint sensor; extracting fourth characteristic data by recognizing the user's face from an image captured by the camera; and corresponding to each of the first to fourth characteristic data. A step of determining an authentication weight for each characteristic data based on the pre-learned first to fourth template data and determining an authentication level for the user based on the authentication weight determined for each characteristic data, wherein each user The extracted characteristic data and information about the determined authentication level are stored and managed in an internal memory, and the user authentication can be performed according to the authentication level determined for each user.

According to another aspect, an artificial intelligence-based electric wheelchair control system for controlling an electric wheelchair includes a display board that displays a user interface screen on a display, a sensor board that generates sensing information based on signals input from a sensor, and the electric wheelchair. A peripheral board including a communication board that performs communication with an external device, a driving board that controls a motor provided in the electric wheelchair, and a steering board that controls steering of the electric wheelchair, and a peripheral board that interlocks with the peripheral board to control the electric wheelchair. It includes an electric wheelchair control device that controls the overall operation, wherein the electric wheelchair control device performs a boot procedure according to power application to perform pairing with a user device, and obtains user information from a server through the paired user device. , determines the motion control mode of the electric wheelchair based on the user information, and recognizes a predefined start command through the sensor corresponding to the determined motion control mode while the AOD (Always On Display) screen is deactivated, Activates the deactivated screen according to the start command, outputs a user interface screen for controlling the electric wheelchair to the display, waits for a user input signal, and performs machine learning on the user input signal input through the sensor. Perform user identification and authentication, and based on successful authentication, identify a control command corresponding to the user input signal to control the operation of the electric wheelchair, wherein the user information includes disability type information and health status. It may contain at least one piece of information.

In an embodiment, the electric wheelchair control device performs the user authentication in a preset authentication application user mode, but the user authentication performs user-dependent authentication in which authentication is performed only for specific preset users and all users registered in the authentication database. It may include user-independent authentication that performs authentication against .

In an embodiment, the operation control mode may include at least one of a manual control mode, a voice recognition control mode, a facial recognition control mode, a gesture recognition control mode, and an autonomous driving control mode.

In an embodiment, the sensor includes a microphone, a camera, and a biometric sensor, the biometric sensor includes at least one of a fingerprint sensor, an iris sensor, and a body pressure sensor, and the electric wheelchair control device selects a user menu on the user interface screen. A user registration procedure is performed according to the following, wherein the user registration procedure includes extracting first characteristic data by recognizing the user's voice through the microphone and extracting second characteristic data by recognizing the user's iris from an image captured through the camera. Extracting, extracting third characteristic data by recognizing the user's fingerprint through the fingerprint sensor, extracting fourth characteristic data by recognizing the user's face from the image captured by the camera, and the first to fourth A step of determining an authentication weight for each characteristic data based on first to fourth template data pre-learned corresponding to each characteristic data, and a step of determining an authentication level for the user based on the authentication weight determined for each characteristic data. Including, the extracted characteristic data for each user and information regarding the determined authentication level are stored and managed in an internal memory, and the user authentication may be performed according to the authentication level determined for each user.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below. You will be able to.

The present invention has the advantage of providing an artificial intelligence-based electric wheelchair control method and a device and system therefor.

Additionally, the present invention has the advantage of providing an artificial intelligence-based electric wheelchair capable of automatically switching the motion control mode of the electric wheelchair based on the user's disability and health status.

In addition, the present invention acquires template data for each authentication method through prior learning of sensing information collected from various authentication sensors, determines the authentication weight for each authentication method by comparing the characteristic data for each authentication method obtained in real time with the corresponding template data, and , there is an advantage in providing an artificial intelligence-based electric wheelchair control method and a device and system therefor that enable customized user authentication by setting the optimal authentication level for each user based on the determined authentication weight.

In addition, the present invention obtains information about the user's disability type and health status from the server in real time through a user device paired with the electric wheelchair, and adaptively sets the motion control mode of the electric wheelchair based on the obtained user information. , it has the advantage of providing an artificial intelligence-based electric wheelchair control method that can provide user convenience and driving safety, as well as devices and systems therefor.

Additionally, the present invention has the advantage of providing an artificial intelligence-based electric wheelchair that can be efficiently shared and used in facilities such as medical institutions and nursing institutions.

In addition, various effects that can be directly or indirectly identified through this document may be provided.

1 is a state diagram of an artificial intelligence-based electric wheelchair control device according to an embodiment.
Figure 2 is a flowchart for explaining the initial booting procedure of the electric wheelchair control device according to an embodiment.
Figure 3 is a flowchart for explaining a procedure for setting the initial operation control mode of an electric wheelchair according to an embodiment.
4 is a flowchart for explaining a method of controlling an electric wheelchair in an electric wheelchair control device according to an embodiment.
Figure 5 is a flowchart for explaining a user registration procedure in an electric wheelchair control device according to an embodiment.
Figure 6 is a flowchart for explaining the user authentication process in the electric wheelchair control device according to an embodiment.
Figure 7 is a flow chart to explain the user authentication process according to the authentication application user mode according to the embodiment.
Figure 8 is a diagram for explaining detailed operations of an electric wheelchair that provides a voice recognition AI-based camera view screen according to an embodiment.
9 is an example of a view screen provided by an electric wheelchair control device according to an embodiment.
Figure 10 is a block diagram for explaining the structure of an electric wheelchair system according to an embodiment of the present invention.
11 shows an example of an electric wheelchair according to an embodiment.

Hereinafter, some embodiments of the present invention will be described in detail through illustrative drawings. When adding reference numerals to components in each drawing, it should be noted that identical components are given the same reference numerals as much as possible even if they are shown in different drawings. Additionally, when describing embodiments of the present invention, if detailed descriptions of related known configurations or functions are judged to impede understanding of the embodiments of the present invention, the detailed descriptions will be omitted.

In describing the components of the embodiments of the present invention, terms such as first, second, A, B, (a), and (b) may be used. These terms are only used to distinguish the component from other components, and the nature, sequence, or order of the component is not limited by the term. Additionally, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the present invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted in an ideal or excessively formal sense unless explicitly defined in the present application. No.

1 is a state diagram for an artificial intelligence-based electric wheelchair control device according to an embodiment.

Referring to Figure 1, the electric wheelchair control status is largely divided into standby status (Standby Status, 10), call status (Wakeup Status, 20), analysis status (30), and result reporting and control status (Result Feedback and Control Status). , 40) and lock status (Lock Status, 50).

When the initial booting procedure is completed, the electric wheelchair control device may transition to the standby state (10).

The electric wheelchair control device according to the embodiment may automatically pair with a predefined (or nearby) user device while performing the initial boot procedure or when the initial boot procedure is completed, and obtain user information from the server through the paired user device. can do. Here, the user information may include at least one of information about the type of disability and information about health status. The server according to the embodiment may be a server managed by a pre-registered medical institution and/or nursing institution, or a dedicated server separately operated to provide artificial intelligence-based electric wheelchair service.

The electric wheelchair control device may determine the initial motion control mode of the electric wheelchair based on user information obtained from the server. Here, the initial operation control mode may include, but is not limited to, a manual control mode, voice recognition control mode, facial recognition control mode, gesture recognition control mode, and autonomous driving control mode. For example, if the user has free use of his or her hands, the initial motion control mode is set to the manual control mode, and if the user has a hearing impairment, the initial motion control mode is set to the gesture recognition control mode, and if the user has a hearing impairment, the initial motion control mode is set to the gesture recognition control mode. If present, the initial motion control mode is set to autonomous driving control mode, and if the user is a patient with cauda equina or amyotrophic lateral sclerosis (ALS), the initial motion control mode is set to facial recognition control mode or voice recognition control mode. It can be.

The standby state (10) is when the electric wheelchair control device is in the AOD (Always On Display) operation mode while the electric wheelchair is stopped or running, the user's specific input through a sensor corresponding to the initial motion control mode - that is, a start command ( It may be waiting for a Wakeup Command. In the standby state 10, the display screen of the electric wheelchair control device - for example, the touch LED screen - may be in a deactivated state, that is, in an OFF state.

As an example, a preset/registered gesture according to the initially set motion control mode - for example, a hand gesture -, a preset/registered facial movement, a preset/registered voice keyword - for example, a wakeup voice command. If - is input by the user, the electric wheelchair control device in the standby state (10) recognizes the user's input through the provided sensor, and if the recognized input signal matches the preset/registered information, the electric wheelchair control device in the standby state (10) is in an inactive state. The display screen can be switched to an active state - that is, an ON state. When the display screen is activated, the electric wheelchair control device can configure and display a user interface screen.

When the display screen is activated, the electric wheelchair control device transitions from the standby state (10) to the call state (20), and an operation is set through a speaker provided on one side of the electric wheelchair, a paired user device, or a display screen provided. Depending on the control mode, it may be notified that the electric wheelchair is available for user control.

The electric wheelchair control device waits for a user input signal through a specific sensor corresponding to the motion control mode set in the call state 20, and can transition to the analysis state 30 when the user input signal is detected.

The electric wheelchair control device can identify the user's control command by analyzing the signal input by the user in the analysis state 30 through a mounted artificial intelligence engine. Here, the artificial intelligence engine may include at least one of a voice recognition engine, a facial recognition engine, a gesture recognition engine, and a touch recognition engine.

When the analysis of the user input signal is normally completed, the electric wheelchair control device transitions to the result reporting state 40 and can transmit a predetermined control command corresponding to the analysis result to the corresponding system (or device or board). For example, if, as a result of analyzing the user input signal, it is a steering control command, the electric wheelchair control device may transmit the steering control command to the driving board that controls the motor. As another example, as a result of analyzing the user input signal, if it is a camera view screen control command, the electric wheelchair control device transmits the camera control command to the camera system to acquire the image captured by the camera, and displays the acquired image on the view screen. It can be configured and displayed on a corresponding display device - for example, a screen of a user device or a display screen provided in an electric wheelchair.

In an embodiment, an electric wheelchair may be equipped with a plurality of cameras, and the electric wheelchair control device may request and receive images captured by the plurality of cameras from the camera system according to a user input signal. In this case, the electric wheelchair control device can adaptively configure the view screen according to the types of cameras from which the image was received and output it through the corresponding display screen. As an example, the camera may include a front camera, left and right side view cameras, and a rear camera.

For example, when the electric wheelchair control device receives an image captured by a front camera, it can control the image captured by the front camera to be displayed on the screen of the user device or a provided display screen.

As another example, when the electric wheelchair control device receives left and right side view camera images, the view screen is divided into two areas - a first area (left) and a second area (right). It can be controlled so that the left side view camera captured image is displayed in area 1, and the right side view camera captured image is displayed in area 2.

As another example, when the electric wheelchair control device receives left/right side view cameras and front camera images, the electric wheelchair control device may configure and output a front surround view screen based on the received images.

As another example, when the electric wheelchair control device receives left/right side view cameras and rear camera images, the electric wheelchair control device may configure and output a rear surround view screen based on the received images.

As another example, when the electric wheelchair control device receives images taken from the front camera, left/right side view cameras, and rear camera, the electric wheelchair control device configures and outputs a bird's eye view screen based on the received images. You can.

If the electric wheelchair control device fails to recognize the user input signal in the analysis state (30), it transitions to the result reporting state (40) and sends a predetermined notification message through the speaker (or user device) indicating that it has failed to recognize the user input signal. Can be printed.

Depending on the preset security and/or lock mode, the electric wheelchair control device may perform machine learning (or deep learning) on user input signals to perform user identification and authentication.

If user identification and authentication are successful, the electric wheelchair control device can perform electric wheelchair control according to the user input signal. If user identification or authentication fails, the electric wheelchair control device outputs a predetermined warning alarm message indicating that user identification or authentication has failed, and may transition from the result reporting state (40) to the call state (20).

If the electric wheelchair control device fails to identify and authenticate the user more than a predetermined number of times, it may output a predetermined warning alarm message indicating that the user is an unregistered user and then transition to the locked state (50). As an example, in the locked state 50, the electric wheelchair control device may provide authentication failure information including current location information to the server (or a preset administrator terminal) through the user device. At this time, the current location of the electric wheelchair can be measured using a GPS receiver provided in the user device, but this is only one example, and the GPS receiver may be mounted on one side of the electric wheelchair.

In the locked state 50, depending on the preset security and/or lock mode level, the electric wheelchair control device may perform user identification and authentication by performing at least one biometric identification of fingerprint recognition, iris recognition, facial recognition, and weight recognition. You can. If biometric identification is successful, the electric wheelchair control device may transition to the call state 20 and wait for a user input signal.

In an embodiment, when the result report is completed, the electric wheelchair control device may return to the call state 20 and wait for the next user command.

The electric wheelchair control device may start a call timer for a predefined period of time each time the call state 20 is entered. The electric wheelchair control device can wait for a user input signal for controlling the electric wheelchair only when the call timer is running. If the call timer expires, the electric wheelchair control device may transition to the standby state (10) and deactivate the display screen.

The electric wheelchair control device according to the embodiment may transition from the call state (20) to the standby state (10) when user identification and authentication fails a certain number of times according to menu settings.

The electric wheelchair control device according to the embodiment is in a standby state (10) when a pre-registered Quit Command is detected in any one of the call state (20), analysis state (30), and result reporting state (40). ) can also transition to .

Figure 2 is a flowchart for explaining the initial booting procedure of the electric wheelchair control device according to an embodiment.

When power is applied, the electric wheelchair control device can initialize the main controller - that is, the main processor - provided through the boot/loader (S210).

The main controller can initialize the communication board for communication with external devices and systems (S220).

As an example, once the communication board is initialized, the electric wheelchair control device can discover user devices and pair with the discovered user devices. The electric wheelchair control device may be linked to an external device - for example, an intersection (or traffic) signal system, a Road Side Unit (RSU), a base station, etc. - through a user device. As an example, the electric wheelchair control device may obtain information about the current traffic light signal from the intersection signal system through the user device. As another example, the electric wheelchair control device may acquire user information by linking with a server through a base station.

The main controller can activate the AOD operation mode by initializing the display board (S230).

The main controller can initialize the sensor board (S240). The main controller can receive video signals captured by the camera(s) mounted on the electric wheelchair and user voice signals input through the microphone through the sensor board. Additionally, the main controller may receive an iris recognition signal, a fingerprint recognition signal, a gesture recognition signal, a touch recognition signal, and an illumination sensing signal through the sensor board. In addition, the sensor board can be connected to ultrasonic sensors, radar, and SPAS (Smart Parking Assistance) sensors to receive sensing signals from those sensors.

The main controller can perform calibration of the camera and various sensors connected to the sensor board to maintain the resolution and sensing accuracy of the camera and sensors at a predefined level. .

The main controller can complete the boot procedure by initializing the steering board and driving board (S250 and S260). The driving board may be configured to control the motor(s) for driving the electric wheelchair.

The main controller according to the embodiment may automatically connect to a pre-registered user device wirelessly or wired when the communication board is initialized. As an example, an electric wheelchair control device and a user device may exchange information through short-range wireless communication - for example, Bluetooth communication.

3 is a flowchart illustrating a procedure for setting an initial operation control mode according to an embodiment.

Referring to FIG. 3, when the initial booting procedure is completed, the electric wheelchair control device may be paired with the user device (S310).

The electric wheelchair control device can acquire user information by accessing the server through the paired user device (S320).

The electric wheelchair control device can determine and set the initial motion control mode based on the acquired user information (S330).

Through the above-described embodiments, the present invention has the advantage of minimizing user inconvenience and controlling the electric wheelchair more safely by adaptively determining and setting the initial operation control mode according to the user's disability type and health condition.

4 is a flowchart for explaining a method of controlling an electric wheelchair in an electric wheelchair control device according to an embodiment.

In detail, FIG. 4 is a diagram to explain how the electric wheelchair control device provides a view screen in conjunction with the camera system when the initial operation control mode is set to the voice recognition control mode.

Referring to FIG. 4, when the LED screen in the AOD (Always-On Display) operation mode is deactivated - that is, the screen is off - the electric wheelchair control device can detect the user's voice signal input through the microphone (S410 ).

The electric wheelchair control device analyzes the detected voice signal through a voice recognition engine, and as a result of the analysis, if the detected voice signal is a predefined start command, the display screen is switched from off to on to display the user interface screen. can be displayed on a equipped display (S420).

The electric wheelchair control device may wait for a user voice command (S430).

The electric wheelchair control device detects the user's voice command, and if the detected voice command is a camera control command, the electric wheelchair control device may acquire an image captured by at least one camera by transmitting the corresponding control command to the camera system (S440).

Here, the camera control commands include the front view display command, left side view display command, right side view display command, rear view display command, left/right side view display command, front surround view display command, rear surround view display command, and bird's eye display command. It may include, but is not limited to, view display commands.

The electric wheelchair control device may configure a view screen based on the acquired image(s) and display it on the display (or user device) (S450).

Figure 5 is a flowchart for explaining a user registration procedure in an electric wheelchair control device according to an embodiment.

Referring to Figure 5, the electric wheelchair control device may initiate the user registration process according to the user menu selection on the user interface screen (S510).

The electric wheelchair control device can recognize the user's voice through a microphone and extract first characteristic data (S520).

The electric wheelchair control device can recognize the user's iris through the iris sensor and extract second characteristic data (S530).

The electric wheelchair control device can recognize the user's fingerprint through a fingerprint sensor and extract third characteristic data (S540).

The electric wheelchair control device may determine the authentication weight for each characteristic data based on the first to third template data pre-learned corresponding to each of the first to fourth characteristic data (S550).

Based on the authentication weight determined for each characteristic data, the authentication level for the user can be determined and registered (S560). Here, information on the extracted characteristic data for each user and the determined authentication level may be stored, maintained, and managed in a user database.

For example, some users may have stronger voice characteristics than other users. This may mean that user identification and authentication is easier through voice. Additionally, some users may have weak voice characteristics but strong iris characteristics. In this case, user identification and authentication accuracy can be improved by setting the authentication weight for the iris higher than that for the voice. As described above, the present invention not only improves the reliability of user identification and authentication by adaptively determining the authentication weight based on the characteristics of authentication data for each user and determining the optimal authentication level accordingly. It has the advantage of minimizing user inconvenience by minimizing unnecessary user authentication.

In the embodiment of FIG. 5, voice recognition, iris recognition, and fingerprint recognition are described as user authentication methods, but this is only one embodiment, and other biometric recognition methods such as facial recognition and weight recognition may be additionally applied. .

Figure 6 is a flowchart for explaining the user authentication process in the electric wheelchair control device according to an embodiment.

Referring to FIG. 6, when the user authentication process is initiated, the electric wheelchair control device may activate the biometric engine (S610).

Here, the biometric recognition engine may include at least one of a voice recognition engine, an iris recognition engine, a fingerprint recognition engine, a facial recognition engine, and a gesture recognition engine.

The electric wheelchair control device may acquire at least one characteristic data through at least one biometric engine (S620).

The electric wheelchair control device may select a candidate user from the user database based on the acquired characteristic data (S630). Here, there may be multiple candidate users depending on characteristic data.

The electric wheelchair control device may identify the pre-registered authentication level corresponding to the selected candidate user (S640).

The electric wheelchair control device can determine whether additional authentication is required based on the identified authentication level (S650).

If additional authentication is required, the electric wheelchair control device can perform final authentication for the candidate user(s) by performing additional authentication procedures according to the corresponding authentication level (S660).

In step 650, if it is determined that additional authentication is not required, the electric wheelchair control device may determine the selected candidate user as the final authenticated user.

As described above, the present invention has the advantage of being able to identify and authenticate users more accurately by adaptively determining the biometric means for user authentication according to the pre-registered authentication level.

Additionally, the present invention has the advantage of minimizing user inconvenience by performing a minimum authentication procedure according to a predetermined authentication level for each user.

Figure 7 is a flowchart for explaining a user authentication procedure according to an authentication application user mode according to an embodiment.

Referring to Figure 7, the electric wheelchair control device can set the authentication application user mode through the user's selection of a predetermined menu (S710).

Here, the authentication application user mode can be divided into a user-dependent mode and a user-independent mode.

The user-dependent mode is a mode in which the user authentication process is performed only for users who have been pre-registered (selected) by the user.

User-independent mode is a mode that performs user authentication procedures for all users registered in the authentication database.

In the case of user-dependent mode, the user authentication process is performed only for specific users, so it has the advantage of faster user authentication, but authentication fails for users who are not registered (or selected) in user-dependent mode. There is a problem with the proposed use of electric wheelchairs.

On the other hand, user-independent mode performs user authentication for all users registered in the authentication database, so it has the advantage of having a very low probability of authentication failure. However, it has the disadvantage of increasing the time required for user authentication compared to user-dependent mode. there is.

When a user input signal is detected, the electric wheelchair device may initiate a user authentication process (S720).

When the user authentication process is initiated, the electric wheelchair control device may determine whether the currently set authentication application user mode is user-dependent or user-independent (S730).

As a result of the determination, in the case of the user-dependent mode, the electric wheelchair control device can perform user authentication only for specific users who have been registered (or selected) in advance (S740).

As a result of the determination in step 730, in the case of the user-independent mode, the electric wheelchair control device can perform user authentication for all users registered in the authentication database (750).

Figure 8 is a diagram for explaining detailed operations of an electric wheelchair that provides a voice recognition AI-based camera view screen according to an embodiment.

Referring to FIG. 8, the electric wheelchair 800 may be configured to include a microphone 810, a voice recognition engine 820, a main controller 830, a camera system 840, and a display 850.

The camera system 840 according to the embodiment may include a left side view camera 841, a right side view camera 842, a front view camera 843, and a rear view camera 844, but it is one This is only an example, and may be configured to include more or fewer cameras.

In another embodiment, the camera system 840 may be configured to include a Surround View Monitor (SVM) camera. SVM cameras are mounted on the front/rear/left/right sides of the vehicle, providing not only a wide view (front camera view), but also a front top view (or front surround view) (front/left/right camera composite view), and a left side view (left camera view). ), right side view (right camera view), rear view (rear camera view), rear top view (or rear surround view) (rear/left/right camera composite view), etc. can be provided.

In another embodiment, the camera system 840 may include a Smart Parking Assistance System (SPAS) sensor. In this case, the main controller 830 can detect a nearby obstacle based on sensing information received from the SPAS sensor. When detecting a short-distance obstacle, the main controller 830 can display short-distance obstacle detection information through the display 850. As an example, the main controller 830 may identify the location of a detected obstacle based on sensing information received from the SPAS sensor. The main controller 830 may match the identified obstacle to the camera view screen corresponding to the identified obstacle location. For example, when an obstacle is detected on the left side of the vehicle, the main controller 830 may configure a view screen including the obstacle detection result in the left side camera captured image and display it on the display 850. Through this, the user has the advantage of being able to intuitively check where there are nearby obstacles.

Hereinafter, the procedure for configuring the view screen by the voice recognition AI-based electric wheelchair 800 will be described in detail.

The user's voice signal input through the microphone 810 may be input to the voice recognition engine.

The voice recognition engine 820 may remove noise included in the received user voice signal and then process the voice signal into natural language to extract a keyword - that is, a user voice command.

The main controller 830 may identify a view type based on the user voice command extracted by the voice recognition engine 820 and request an image corresponding to the identified view type from the camera system 840. The camera system 840 may transmit at least one camera captured image corresponding to the identified view type to the main controller 830.

The main controller 830 configures a view screen to be displayed on the display 850 based on the image received from the camera system 840 and displays the configured view screen through the display 850. As shown in FIG. 9, the view screen that can be configured by the main controller 830 can be configured in various ways according to the user's voice command.

9 is an example of a view screen provided by an electric wheelchair control device according to an embodiment.

Referring to FIG. 9, the view screens that can be displayed on the display 850 are largely the left camera view screen 910, the right camera view screen 920, both camera view screens 930, and the front camera view screen 940. , may include a front (or rear) surround view screen 950 and a bird's eye view screen 960.

The electric wheelchair 800 can adaptively configure a view screen according to a user's voice command and display it on the provided display 850.

When the electric wheelchair 800 according to the embodiment detects a nearby obstacle in conjunction with the SPAS sensor, the view screen may be configured to display additional obstacle detection information on one side of the corresponding view screen. Here, the obstacle detection information may include, but is not limited to, information about the location of the obstacle and information about the type of the obstacle.

Figure 10 is a block diagram for explaining the structure of an electric wheelchair system according to an embodiment of the present invention.

As shown in FIG. 10, the electric wheelchair system 1000 largely includes an electric wheelchair control device 1010, a peripheral control board connected to the electric wheelchair control device 1010, a user device 1051, and a server 1052. It can be configured.

The peripheral control board may include a communication board 1050, a display board 1060, a sensor board 1070, a steering board 1080, and a driving board 1090.

The communication board 1050 is equipped with an antenna and a transceiver for communication with an external device, sets up a communication channel with the user device 1051, and exchanges information through the set communication channel. As an example, the communication board 1050 may provide at least one communication function among Bluetooth communication, 4G Long Term Evolution (LTE) communication, 5G New Radio (NR) communication, and Wi-Fi communication.

Additionally, the communication board 920 may be connected to the user device 1051 and transmit a view screen configured by the electric wheelchair control device 1010 to the user device 1051. The user device 1051 can display the received view screen on one side of the provided screen.

The display board 1060 can be connected to the LED 1061 equipped with a touch sensor, and displays the user interface screen and various status information of the electric wheelchair 1000 according to the control signal from the electric wheelchair control device 1010 through the LED 1061. It can be displayed in . The display board 1060 can control the LED 1061 in the AOD operation mode.

The sensor board 1070 includes an illumination sensor 1071, a proximity sensor 1072, a touch sensor 1073, a fingerprint sensor 1074, an iris sensor 1075, an ultrasonic sensor 1076, a radar 1077, and a microphone 1078. ) and the camera system 1079, and sensing information obtained from the corresponding sensor can be provided to the electric wheelchair control device 1010. The electric wheelchair control device 1010 includes sensing information received from at least one of an illumination sensor 1071, a proximity sensor 1072, a touch sensor 1073, a fingerprint sensor 1074, an iris sensor 1075, and a microphone 1078. Alternatively, operations such as illumination recognition, gesture recognition, touch recognition, user authentication, and user voice recognition may be performed based on the voice signal.

Additionally, the sensor board 1070 may be connected to the camera system 1079, receive image signals captured by at least one camera, and provide the received image signals to the electric wheelchair control device 1010. The electric wheelchair control device 1010 may configure a view screen based on the image signal received from the sensor board 1070 and transmit the configured view screen to the display board 1060 and/or the communication board 1050.

Additionally, the sensor board 1070 may also be connected to a SPAS sensor (not shown). The sensor board 1070 may provide sensing information received from the SPAS sensor to the electric wheelchair control device 1010. As an example, the electric wheelchair control device 1010 may detect a short-distance obstacle based on sensing information from the SPAS sensor.

The sensor board 1070 may perform a calibration operation for connected sensors and the camera system 1079 upon initialization.

The illuminance sensor 1071 may measure the illuminance around the electric wheelchair control device 1010 and/or around the LED 1061. The electric wheelchair control device 1010 can dynamically adjust the brightness of the LED 1061 screen using the illuminance sensing information received from the sensor board 1070.

The steering board 1080 is connected to the steering stick 1081, can detect a user's steering command, and can control a steering control device (not shown) provided in the electric wheelchair 1000 according to the detected steering command.

The driving board 1090 is connected to the motor 1091 and can control the driving speed of the motor 1091.

The electric wheelchair control device 1010 may include a main controller 1020, a memory 1020, various recognition engines, and a user authentication unit 1040.

Here, the recognition engine includes a voice recognition engine 1031 that recognizes user control commands by processing the user's voice signal into natural language, and a facial recognition engine 1032 that recognizes user control commands by recognizing the user's facial movements from images captured by a camera. , a gesture recognition engine 1033 that recognizes and identifies the user's gesture, recognizes a user control command corresponding to the identified gesture, and identifies the user's touch action on the touch screen and controls the user corresponding to the identified touch action. It may be configured to include at least one of a touch recognition engine that recognizes commands.

The user authentication unit 1040 may include at least one of a voice identification unit 1041, an iris identification unit 1042, a fingerprint identification unit 1043, a face identification unit 1044, and an authentication level determination unit 1045. You can.

The voice identification unit 1041 extracts user voice characteristics from the user-input voice signal and compares the extracted user voice characteristics with voice characteristic data pre-registered in the memory 1020 to perform user identification and authentication.

The voice identification unit 1041 extracts characteristic data for the user's voice based on the user input voice signal, compares the extracted voice characteristic data with previously learned voice template data, and provides an authentication weight for the voice characteristic data. can be decided.

The voice identification unit 1041 can improve voice recognition sensitivity by removing noise included in the voice signal.

The iris identification unit 1042 extracts user iris characteristics from the iris image received from the iris sensor 1075 and compares the extracted user iris characteristics with iris characteristic data pre-registered in the memory 1020 to identify and authenticate the user. It can be done.

The iris identification unit 1042 extracts characteristic data for the user's iris based on the iris sensing information, compares the extracted iris characteristic data with previously learned iris template data, and determines an authentication weight for the iris characteristic data. You can.

The fingerprint identification unit 1043 extracts user fingerprint characteristics from the fingerprint image received from the fingerprint sensor 1074 and compares the extracted user fingerprint characteristics with fingerprint characteristic data pre-registered in the memory 1020 to identify and authenticate the user. It can be done.

The fingerprint identification unit 1043 extracts characteristic data for the user's fingerprint based on the fingerprint sensing information, compares the extracted fingerprint characteristic data with previously learned fingerprint template data, and determines the authentication weight for the fingerprint characteristic data. You can.

The facial identification unit 1044 extracts user facial characteristics from the user's facial image captured by the camera, and compares the extracted user facial characteristics with facial characteristic data pre-registered in the memory 1020 to perform user identification and authentication. there is.

The face identification unit 1044 extracts characteristic data about the user's face based on the face sensing information, compares the extracted facial characteristic data with previously learned facial template data, and determines an authentication weight for the facial characteristic data. You can.

The authentication level determination unit 1044 may dynamically determine the authentication level for the user based on the authentication weight determined by the identification unit(s) described above.

The authentication level determination unit 1044 may sort the characteristic data in order of highest authentication weight for each user and add up the authentication weights until the cumulative sum of the authentication weights of the sorted characteristic data reaches a predetermined threshold.

If the accumulated authentication weight is greater than or equal to the threshold, the authentication level determination unit 1044 may determine the authentication level based on the added characteristic data type and number.

For example, when the threshold value is reached with only the authentication weight corresponding to the voice characteristic data, the authentication level may be determined to be the first level. If the sum of the authentication weight corresponding to the voice characteristic data and the authentication weight corresponding to the iris characteristic data is greater than or equal to the threshold value, the authentication level may be determined as the second level. If the sum of the authentication weight corresponding to the voice characteristic data, the authentication weight corresponding to the iris characteristic data, and the authentication weight corresponding to the fingerprint characteristic data is greater than or equal to the threshold value, the authentication level may be determined as the third level. If the sum of the authentication weight corresponding to the voice characteristic data, the authentication weight corresponding to the iris characteristic data, the authentication weight corresponding to the fingerprint characteristic data, and the authentication weight corresponding to the facial characteristic data is more than the threshold, the authentication level goes to the fourth level. can be decided.

The memory 1020 can maintain various firmware and software required for the operation of the electric wheelchair control device 1010. Additionally, the memory 1020 can maintain a database of pre-registered characteristic data for each user for user identification and authentication. Additionally, the memory 1020 may maintain information on various parameters including the authentication application speaker mode. Characteristic data and authentication levels for each user may be stored and maintained in the memory 960. Additionally, pre-trained template data for each authentication type may be maintained in the memory 960.

The main controller 1020 can activate the LED 931 screen in the AOD operation mode by recognizing at least one of the user's gesture, voice command, facial movement, and touch operation according to the set operation control mode.

In addition, when the LED 1061 screen in the AOD operation mode is activated, the main controller 1020 configures a user interface screen for controlling the electric wheelchair 1000 and transmits the configured user interface screen to the display board 1060. It can be displayed on the LED (1061) screen.

Additionally, pre-trained template data for each authentication type may be maintained in the memory 960.

The electric wheelchair 1000 is not explicitly shown in FIG. 10 and may further include a rechargeable battery, a wheelchair frame and footrest, wheels, speakers, a lamp, a shock absorber, etc.

The sensor board 1070 according to the embodiment may be linked to a body pressure sensor (not shown) that measures the distribution of the passenger's body pressure on the seat.

The main controller 1020 may control air supply to a seat bolster (not shown) disposed on one side of the seat based on body pressure sensing information.

11 shows an example of an electric wheelchair according to an embodiment.

In Figure 11, drawing number 1110 is a left side view of the electric wheelchair, drawing number 1120 is a right side view of the electric wheelchair, drawing number 1130 is a front view of the electric wheelchair, and drawing number 1140 is a rear view of the electric wheelchair.

Referring to drawing number 1110, a left side camera 1111 for capturing an image of the left side of the electric wheelchair may be mounted on one outer side of the left handle 1116. In addition, one end of the support 1114 is connected to one side of the backrest 1115, and a facial camera 1113 for facial recognition, a touch display 1112, and a microphone 1117 for voice recognition are connected to the other end of the support 1114. can be installed. In an embodiment, the support 1114 may be implemented to be able to rotate left and right.

Referring to drawing number 1120, a right side camera 1121 and a steering lever 1122 for capturing images of the right side of the electric wheelchair may be mounted on one outer side of the right handle 1123.

Referring to drawing number 1130, a front camera 1131 is mounted on one side of the front of the right handle 1123 to capture images of the right side of the electric wheelchair, and a front camera 1131 is installed on one side of the bottom of the front of the electric wheelchair to detect nearby obstacles. An ultrasonic sensor 1132 and/or radar (not shown) may be provided.

The electric wheelchair control device 1010 and various boards can be mounted on one side of the electric wheelchair body.

The steps of the method or algorithm described in connection with the embodiments disclosed herein may be implemented directly as hardware, software modules, or a combination of the two executed by a processor. Software modules may reside in a storage medium (i.e., memory and/or storage) such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM.

An exemplary storage medium is coupled to a processor, the processor capable of reading information from and writing information to the storage medium. Alternatively, the storage medium may be integral with the processor. The processor and storage medium may reside within an application specific integrated circuit (ASIC). The ASIC may reside within the user terminal. Alternatively, the processor and storage medium may reside as separate components within the user terminal.

The above description is merely an illustrative explanation of the technical idea of the present invention, and various modifications and variations will be possible to those skilled in the art without departing from the essential characteristics of the present invention.

Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but rather to explain it, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be construed as being included in the scope of rights of the present invention.

Claims (10)

In the artificial intelligence-based electric wheelchair control method,
Performing a boot procedure upon power application to pair with a user device;
Obtaining user information from a server through the paired user device;
determining an operation control mode of the electric wheelchair based on the user information;
Recognizing a predefined start command through a sensor corresponding to the determined operation control mode while the AOD (Always On Display) screen is deactivated;
activating the deactivated screen according to the start command, outputting a user interface screen for controlling the electric wheelchair on a display, and then waiting for a user input signal;
Performing user identification and authentication by performing machine learning on the user input signal input through the sensor; and
Based on successful authentication, identifying a control command corresponding to the user input signal to control the operation of the electric wheelchair
A method for controlling an electric wheelchair, wherein the user information includes at least one of disability type information and health status information. According to paragraph 1,
Further including the step of setting the authentication application user mode,
The user authentication includes user-dependent authentication, which performs authentication only on preset specific users, and user-independent authentication, which performs authentication on all users registered in an authentication database. According to paragraph 1,
The operation control mode includes at least one of a manual control mode, a voice recognition control mode, a facial recognition control mode, a gesture recognition control mode, and an autonomous driving control mode. According to paragraph 3,
Based on the operation control mode being a voice recognition control mode, determining a view type corresponding to a recognized voice command;
determining at least one camera to receive an image corresponding to the determined view type;
Constructing a view screen based on the image obtained from the determined at least one camera; and
Displaying the view screen on the screen of the user device or the display
Further comprising, wherein the view type includes at least one of front view, left side view, right side view, rear view, left/right side view, front surround view, rear surround view, and bird's eye view, electric wheelchair control method. According to paragraph 4,
Further comprising detecting a nearby obstacle using at least one of a SPAS (Smart Parking Assistance System) sensor, an ultrasonic sensor, and a radar provided while the electric wheelchair is traveling, and the view screen based on the detected nearby obstacle. A method of controlling an electric wheelchair, characterized in that information about the detected nearby obstacle is displayed on one side of the. According to paragraph 1,
The sensors include microphones, cameras, and biometric sensors,
The biometric sensor includes at least one of a fingerprint sensor, an iris sensor, and a body pressure sensor,
Initiating the user registration process by selecting a predetermined user menu on the user interface screen,
The user registration process is,
extracting first characteristic data by recognizing the user's voice through the microphone;
extracting second characteristic data by recognizing a user's iris from an image captured through the camera;
extracting third characteristic data by recognizing the user's fingerprint through the fingerprint sensor;
extracting fourth characteristic data by recognizing the user's face from the image captured through the camera;
determining an authentication weight for each characteristic data based on first to fourth template data pre-learned corresponding to each of the first to fourth characteristic data; and
Determining the authentication level for the user based on the authentication weight determined for each characteristic data
Including, the extracted characteristic data for each user and the information regarding the determined authentication level are stored and managed in an internal memory, and the user authentication is performed according to the authentication level determined for each user. Electric wheelchair control method. In an artificial intelligence-based electric wheelchair control system for controlling an electric wheelchair,
A display board that displays a user interface screen on a display;
A sensor board that generates sensing information based on signals input from the sensor;
a communication board that communicates between the electric wheelchair and an external device;
a driving board that controls a motor provided in the electric wheelchair; and
Steering board for controlling the steering of the electric wheelchair
Peripheral board containing; and
An electric wheelchair control device that controls the overall operation of the electric wheelchair in conjunction with the peripheral board
Including,
The electric wheelchair control device performs a boot procedure according to power application to perform pairing with a user device, obtains user information from a server through the paired user device, and operates an operation control mode of the electric wheelchair based on the user information. Determine, recognize a predefined startup command through the sensor corresponding to the determined operation control mode while the AOD (Always On Display) screen is deactivated, and activate the deactivated screen according to the startup command, After outputting a user interface screen for controlling the electric wheelchair to the display, waiting for a user input signal, performing machine learning on the user input signal input through the sensor to perform user identification and authentication, and performing the authentication. Based on success, the operation of the electric wheelchair is controlled by identifying a control command corresponding to the user input signal,
The user information includes at least one of disability type information and health status information. In clause 7,
The electric wheelchair control device performs the user authentication in a preset authentication application user mode,
The user authentication includes user-dependent authentication, which performs authentication only for specific preset users, and user-independent authentication, which performs authentication for all users registered in an authentication database. In clause 7,
The motion control mode includes at least one of a manual control mode, a voice recognition control mode, a facial recognition control mode, a gesture recognition control mode, and an autonomous driving control mode. In clause 7,
The sensors include microphones, cameras, and biometric sensors,
The biometric sensor includes at least one of a fingerprint sensor, an iris sensor, and a body pressure sensor,
The electric wheelchair control device performs a user registration procedure according to user menu selection on the user interface screen,
The user registration process is,
extracting first characteristic data by recognizing the user's voice through the microphone;
extracting second characteristic data by recognizing a user's iris from an image captured through the camera;
extracting third characteristic data by recognizing the user's fingerprint through the fingerprint sensor;
extracting fourth characteristic data by recognizing the user's face from the image captured through the camera;
determining an authentication weight for each characteristic data based on first to fourth template data pre-learned corresponding to each of the first to fourth characteristic data; and
Determining the authentication level for the user based on the authentication weight determined for each characteristic data
Including, the extracted characteristic data for each user and the information regarding the determined authentication level are stored and managed in an internal memory, and the user authentication is performed according to the authentication level determined for each user. Electric wheelchair control system.