WO2022216232A1

WO2022216232A1 - Methods and systems for shared control of goal directed wheelchair navigation

Info

Publication number: WO2022216232A1
Application number: PCT/SG2022/050197
Authority: WO
Inventors: Wei Tech ANG; Neha Priyadarshini GARG; Zhen Lei; Bang Yi TAN; Lei Li; Ananda Ekaputera SIDARTA
Original assignee: Nanyang Technological University
Priority date: 2021-04-08
Filing date: 2022-04-06
Publication date: 2022-10-13
Also published as: CN117178165A

Abstract

Methods, systems and computer readable memory for shared control of goal directed wheelchair navigations are provided. According to another aspect of the present embodiments, a shared control of goal directed navigation system is provided. The shared control of goal directed navigation system includes a user control device and a shared navigation controller. The shared navigation controller coupled to the user movement control device and generates a set of user intentions for navigating a plurality of paths to a predetermined goal. The shared navigation controller also predicts one of the set of user intentions as a preferred path in response to computing a probability of various ones of the plurality of paths to the predetermined goal and uses the preferred path as a guide for a shared dynamic window approach (DWA) based local path planning, the shared DWA based local path planning recognizing user input received from the user movement control device. Further, the shared navigation controller determines a final control command based on the user input.

Description

METHODS AND SYSTEMS FOR SHARED CONTROL OF GOAL DIRECTED WHEELCHAIR NAVIGATION

PRIORITY CLAIM

[0001] This application claims priority from Singapore Patent Application No. 10202103600T filed on 08 April 2021.

TECHNICAL FIELD

[0002] The present invention generally relates to wheelchair navigation, and more particularly relates to methods and systems for shared control of goal directed wheelchair navigation.

BACKGROUND OF THE DISCLOSURE

[0003] With the population aging in many countries, the demand for wheelchairs is constantly growing. Manually controlling a wheelchair requires upper limb strength and fine motor control ability. While people with upper limb disability might find it extremely difficult to perform manual control, even healthy users can benefit from assistive wheelchairs through reduction in their workload. Robotized assistive wheelchairs aim to provide navigation assistance to humans through shared control by leveraging autonomous navigation capabilities.

[0004] A general framework for conventional autonomous navigation to navigate from a start position to a goal location consists of a global planner and a local planner. Given the environment map, a global planner computes a path between the start position and the goal location using heuristic search algorithms like A*. Then a local path planner like dynamic window approach (DWA) or timed elastic bands uses information from various sensors such as lidar or cameras to follow that path locally while satisfying kinodynamic constraints.

[0005] In a shared control framework for wheelchair navigation, the local path planner is modified to also take the user input into account. Some conventional frameworks only provide local obstacle avoidance to the user: in other words, they modify the user’s input such that the wheelchair does not collide with obstacles. Such systems either have no notion of the user’s final goal or assume that the user would be following a fixed path determined by the global planner. Thus, such systems fail to take account of human intention.

[0006] Without any notion of the final goal, it is difficult to assist in fine motor control tasks like making sharp turns or entering narrow doorways. Thus, without intention prediction, shared control systems can result in the user turning away from narrow doorways.

[0007] Assuming that the user would follow the path to the final goal determined by the global planner allows shared control algorithm to assist with fine motor control tasks but disadvantageously limits the control authority of the user as the user may have some other preferred path in mind. Various studies have shown that while being assisted, users still want to have control at the local level and expect the wheelchair to respond to their commands. However, providing assistance based on human intention prediction has several challenges such as defining a possible set of human intentions, keeping track of most likely human intentions based on control inputs provided by the user, and providing appropriate control to the user.

[0008] Most of the human intention prediction based shared controlled wheelchair systems do not fully address all these challenges. They have been developed for either very specific scenarios like navigating to predefined doors or for a fixed set of pre defined global goals with no control given to user over the choice of a local path. [0009] Thus, there is a need for methods and systems for shared control of goal directed wheelchair navigation which can assist with fine motor tasks and also allow user to follow their preferred path while addressing the challenges of defining a possible set of human intentions, keeping track of most likely human intentions based on control inputs provided by the user, and providing appropriate control to the user. Furthermore, other desirable features and characteristics will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and this background of the disclosure.

SUMMARY

[0010] According to at least one aspect of the present embodiments, a shared control method for a goal directed navigation system is provided. The method includes generating a set of user intentions for navigating a plurality of paths to a predetermined goal and predicting one of the set of user intentions as a preferred path in response to computing a probability of various ones of the plurality of paths to the predetermined goal. The method further includes using the preferred path as a guide for a shared dynamic window approach (DWA) based local path planning.

[0011] According to another aspect of the present embodiments, a goal directed navigation system with a shared control method is provided. The goal directed navigation system includes a user control device and a shared navigation controller. The shared navigation controller is coupled to the user movement control device and generates a set of user intentions for navigating a plurality of paths to a predetermined goal. The shared navigation controller also predicts one of the set of user intentions as a preferred path in response to computing a probability of various ones of the plurality of paths to the predetermined goal and uses the preferred path as a guide for a shared dynamic window approach (DWA) based local path planning, the shared DWA based local path planning recognizing user input received from the user movement control device. Further, the shared navigation controller determines a final control command based on the user input.

[0012] According to yet a further aspect of the present embodiments, a computer readable medium is provided. The computer readable medium includes instructions for a shared navigation controller to perform a method for shared control of goal directed navigation. The instructions cause the shared navigation controller to generate a set of user intentions for navigating a plurality of paths to a predetermined goal and to predict one of the set of user intentions as a preferred path in response to computing a probability of various ones of the plurality of paths to the predetermined goal. The instructions also cause the shared navigation controller to use the preferred path as a guide for a shared dynamic window approach (DWA) based local path planning, the shared DWA based local path planning comprises recognizes received user input. The instructions further cause the shared navigation controller to determine a final control command based on the user input.

BRIEF DESCRIPTION OF THE DRAWINGS [0013] The accompanying figures, where like reference numerals refer to identical or functionally similar elements throughout the separate views and which together with the detailed description below are incorporated in and form part of the specification, serve to illustrate various embodiments and to explain various principles and advantages in accordance with present embodiments. [0014] FIG. 1, comprising FIGs. 1A and IB, depicts processing of voronoi paths to be used as global paths form a given start point to an end goal in accordance with present embodiments, wherein FIG. 1A depicts the processing of the voronoi paths before contraction and FIG. IB depicts the processing of the voronoi paths after contraction.

[0015] FIG. 2, comprising FIGs. 2A to 2D, depicts progressive contraction and interpolation of a path using trajectory optimization while assuring the homotopy class of the path is not changed in accordance with the present embodiments, wherein FIG. 2A depicts computation of a straight line from a starting point S to a first point C, FIG. 2B depicts finding a point S’ on the straight line of FIG. 2 A, FIG. 2C computation of a straight line from the point S’ as a starting point to a second point C’, and FIG. 2D depicts finding a point S” on the straight line of FIG. 2C.

[0016] FIG. 3, comprising FIGs. 3A, 3B and 3C, depicts illustrations of the effects of exploring new homotopy classes in accordance with the present embodiments, wherein FIG. 3A depicts initial homotopy paths, FIG. 3B depicts homotopy paths after the wheelchair moves without exploration of new homotopy classes, and FIG. 3C depicts homotopy paths after the wheelchair moves with exploration of new homotopy classes in accordance with the present embodiments.

[0017] FIG. 4, comprising FIGs. 4A and 4B, depicts test scenarios for simulation experiments in accordance with the present embodiments, wherein FIG. 4A depicts a hospital scenario and FIG. 4B depicts a doorway traversal scenario.

[0018] FIG. 5 is a photograph of an experimental setup for a simulation in accordance with the present embodiments wherein the subject performs the experimental test with a weight on the non-dominant hand. [0019] FIG. 6, comprising FIGs. 6A and 6B, depicts two attempts by a first disabled subject for a hospital scenario using a conventional Shared-DWA method, wherein FIG. 6A depicts a first attempt by the first subject and FIG. 6B depicts a second attempt by the first subject.

[0020] FIG. 7, comprising FIGs. 7A and 7B, depicts two attempts by the first disabled subject for the hospital scenario using the method in accordance with the present embodiments, wherein FIG. 7A depicts a first attempt by the first subject and FIG. 7B depicts a second attempt by the first subject.

[0021] FIG. 8, comprising FIGs. 8A and 8B, depicts two attempts by a second disabled subject for the hospital scenario using the conventional Shared-DWA method, wherein FIG. 8A depicts a first attempt by the second subject and FIG. 8B depicts a second attempt by the second subject.

[0022] FIG. 9, comprising FIGs. 9A and 9B, depicts two attempts by the second disabled subject for the hospital scenario using the method in accordance with the present embodiments, wherein FIG. 9A depicts a first attempt by the second subject and FIG. 9B depicts a second attempt by the second subject.

[0023] FIG. 10, comprising FIGs. 10A and 10B, depicts two attempts by the first disabled subject for a doorway traversal scenario using the conventional Shared-DWA method, wherein FIG. 10A depicts a first attempt by the first subject and FIG. 10B depicts a second attempt by the first subject.

[0024] FIG. 11, comprising FIGs. 11A and 11B, depicts two attempts by the first disabled subject for the doorway traversal scenario using the method in accordance with the present embodiments, wherein FIG. 11A depicts a first attempt by the first subject and FIG. 1 IB depicts a second attempt by the first subject. [0025] FIG. 12, comprising FIGs. 12A and 12B, depicts two attempts by the second disabled subject for the doorway traversal scenario using the conventional Shared- DWA method, wherein FIG. 12A depicts a first attempt by the second subject and FIG. 12B depicts a second attempt by the second subject.

[0026] FIG. 13, comprising FIGs. 13 A and 13B, depicts two attempts by the second disabled subject for the doorway traversal scenario using the method in accordance with the present embodiments, wherein FIG. 13 A depicts a first attempt by the second subject and FIG. 13B depicts a second attempt by the second subject.

[0027] FIG. 14 depicts an illustration of a robotic wheelchair used for experiments in accordance with the present embodiments.

[0028] FIG. 15 depicts a photograph showing a real world wheelchair test scenario. [0029] And FIG. 16, comprising FIGs. 16A and 16B, depicts illustrations showing how, due to imprecise user input, the robotic wheelchair steers away from the door with only obstacle avoidance (Shared-DWA output) yet provides a correct action to help the robotic wheelchair steer through the door using the intention prediction based system in accordance with the present embodiments.

[0030] Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been depicted to scale.

DETAILED DESCRIPTION

[0031] The following detailed description is merely exemplary in nature and is not intended to limit the invention or the application and uses of the invention. Furthermore, there is no intention to be bound by any theory presented in the preceding background of the invention or the following detailed description. It is the intent of present embodiments to present methods and systems to provide assistance to a user in navigating to a user- specified goal while giving the control authority to the user over local path planning. In accordance with present embodiments, dynamic intermediate goals are generated using generalized voronoi diagrams, maintaining a probability distribution over them based on user inputs using principles of MaxEnt IRL and homotopy class of the goals, and give appropriate local control to the user through a shared dynamic widow local path planner which takes user input and most likely intermediate goal into account.

[0032] The methods and systems in accordance with the present embodiments advantageously lead to faster attainment of the user- specified goal as compared to a local obstacle avoidance system while complying with user authority as compared to a system with fixed predefined goals and no local shared control. The present embodiments provide methods and systems beneficial for users with upper limb disability, such as users with cerebral palsy. At the same time, the methods and systems in accordance with the present embodiments address the need to model joystick control ability of the user for better human intention prediction for people with upper limb disability.

[0033] In accordance with the methods and systems of the present embodiments, a shared control system is provided which can take into account a final goal of a wheelchair user for improved assistance while giving the user a high control authority over operation of the wheelchair. In addition, through simulations involving humans with induced disability, the methods and systems in accordance with the present embodiments have been quantitatively shown to perform better than a system which does not take into account a user’s goal. Further, it is shown as described hereinafter that cerebral palsy subjects using simulated control in accordance with the present embodiments are provided significantly better assistance while making turns into doorways as the methods and systems of the present embodiments take user control ability into account for better human intention intimation.

[0034] Conventional systems for shared control of robotic devices and wheelchairs can be broadly divided into two categories based on whether they predict human intention or not. Approaches that do not predict human intention either have no notion of final goal or assume a fixed global goal with a known path that has to be followed. For example, the reward function of the dynamic window approach (DWA) has been modified to give preference to velocities that are closer to user input. Similarly, the Vector Field Histogram approach has been modified to give some weight to the user input. These approaches, however, make it very difficult for the user to enter narrow doorways or to make sharp turns as obstacle avoidance control tries to turn them away. [0035] Knowing the user’s intention can help to avoid such behaviour. But user intention compensated solutions also have their disadvantage. For example, some solutions assume that a global goal and optimal path is known and they blend the user input with the output of a local path planner. The wheelchair thus tries to follow the optimal path to the goal determined by the robot while allowing the user to only make small changes at a local path planning level. While this helps the user to reach the goal, it also limits the user’s control authority and limits the user’s ability to choose between multiple ways of reaching the same goal.

[0036] Keeping track of all multiple ways to meet a goal and predicting which one is preferred by the user could lead to better shared control and several approaches have been proposed which assist users by predicting their intention. One main challenge for these approaches, however, is how to define a user’s intentions. Typically, intentions are defined as the shortest paths for manually identified global goals or very specific cases like automatically detected doorways. Requirements to manually define a set of all possible global goals limits the applicability of such approaches in a generic scenario.

[0037] In accordance with the present embodiments, it is assumed that a final goal of the user is known but the path that the user wants to take to reach that goal is unknown. A set of user intentions is automatically generated using generalized voronoi diagrams for a given goal. While voronoi diagrams have been used to compute various paths for reaching a goal, prior art systems and methods have only allowed a user to choose between those various paths but do not allow the user to control the wheelchair continuously. In accordance with the present embodiments, a shared-DWA based local path planner is used to give appropriate control to the user.

[0038] After the intentions are defined, next step in accordance with the present embodiments is to predict the path preferred by the user. Most conventional systems and methods rely on past commands issued by user to predict human intention. For example, some methods forward project a user’s joystick commands, some methods learn parameters of a hand-defined model for a specific user, and some methods use the ideas of MaxEntIRL to compute a probability of goals using a manually defined cost function for each goal. The methods and systems in accordance with the present embodiments utilize the ideas of MaxEntIRL to compute the probability of various paths as MaxEntIRL is a principled approach to compute probability of a user’s goal based on a cost function and is easy to specify.

[0039] After predicting a user’s intention, the next step of the methods and systems in accordance with the present embodiments is to compute final control command. Conventional methods either use outer loop blending or inner loop blending with the user command to compute the final control command. In outer loop blending, a user input is blended with the optimal action computed by a local path planner. While practical outer loop blending is easier to use and works well, a main disadvantage of outer loop blending is that many actions which are only slightly suboptimal and could have been better from a user point of view are discarded before blending with the user input. Also, outer loop blending may not be safe. In inner loop blending, a user input is considered along with various possible actions while calculating the best action for a given state. In accordance with the present embodiments, inner loop blending is used while calculating a reward for various possible velocities in a dynamic window.

[0040] Given only the final goal G of a wheelchair user in a static indoor environment with a known map, but no information about a preferred user path i_u, the question is what should be the control command a E A issued by the robot at every timestep t that results in a path which is as close as possible to the user’s preferred path i_u and also takes the user towards the goal G. Although, the preferred path i_u is unknown, the joystick commands u E U issued by the user are observed and can be used to predict the user’s preferred path.

[0041] To answer this question, a set of preferred paths I are computed in accordance with the present embodiments. Theoretically, the size of this set can be infinite as any path can be the user’s preferred path. Thus, the concept of homotopy classes is used in accordance with the preferred embodiments to restrict the size of this set of preferred paths I. For a given start location and goal location, two paths belong to a same homotopy class if they can be transformed into each other without crossing any obstacle. Thus, by the definition of the homotopy class, it is difficult for users to change to a path in a different homotopy class. Because of this property of homotopy classes, the size of the set of preferred paths I are restricted to the number of homotopy classes into which all the paths can be categorized. [0042] Therefore, in accordance with the methods and systems of the present embodiments, paths belonging to different homotopy classes are computed using generalized voronoi diagrams and the computed paths are used to represent each homotopy class. The user’s preferred path i_u would belong to one of these homotopy classes. A probability distribution is maintained over these paths i_u using principles of MaxEntIRL and the most likely path is used as a guide for DWA based local path planning. To allow more control to the user while following the most likely path among various homotopy paths, the local path planner also takes the user input into account while computing the final control command. A measure of how crowded an environment is (i.e., the crowdedness of the local environment) is used to adjust the weight between a user input and an action which takes the user towards the goal via most likely homotopy class. This adjustment results in a path which is close to the user’s preferred path i_u because when there is less free space, there are very few deviations possible from a path in a given homotopy class.

[0043] The first step is to process voronoi paths so that they can be used as global path plans and also to link the new paths after replanning to their previous homotopy class. This is shown in FIGs. 1A and IB where illustrations 100, 150 depict paths from a given start point 110 to an end goal 120. To generate paths belonging to different homotopy classes, the Voronoi diagrams are generated using Fortune’s sweep algorithm as shown in the illustration 100 of paths in different homotopy classes before contraction. The K shortest paths are then extracted from the start position 110 to the end goal 120 using Yen’s algortithm. The homotopy class is determined using known algorithms and, at each iteration of Yen’s algorithm, a homotopy class check is done to ensure that each of K shortest paths is in a unique homotopy class. The illustration 150 (FIG. IB) depicts the ten shortest paths (i.e., K=10) in different homotopy classes from the given start 110 to the end goal 120 after contraction.

[0044] The voronoi paths generated are very jagged and occasionally have unnecessary turns so, in accordance with the present embodiments, the voronoi paths are processed to obtain smooth trajectories that can be followed by a local path planner. For example, trajectory optimization can be used to generate trajectories from voronoi paths. However, it is necessary to make sure that the homotopy class of the paths is not changed while contracting and interpolating them to render them optimal. To do this, the path contraction and interpolation is done while checking that any obstacle is not crossed. An example of the path contraction and interpolation procedure in accordance with the present embodiments is shown progressively in the illustrations 200, 220, 240, 260 of FIGs. 2A to 2D. Referring to the illustration 200 (FIG. 2A), a nearest point (point A) is found on a path 210 which cannot be joined to the starting point (point S) using a straight line without colliding with an obstacle 214 and a straight line 216 is computed which joins the starting point S with a point before point A (i.e., point C). Referring to the illustration 220, a point S’ is found on the straight line 216 which can be connected to point A without colliding with the obstacle 214. Referring next to the illustrations 240, 260, the path between S and S’ is kept and the whole procedure is repeated with S’ as the starting point. That is, a nearest point A’ is found on the path 210 as shown in the illustration 240 which cannot be joined to the starting point S ’ using a straight line without colliding with an obstacle 244 and a straight line 246 is computed which joins the starting point S’ with a point C’ before the point A’. Referring to the illustration 260, a point S” is found on the straight line 246 which can be connected to point A’ without colliding with the obstacle 244. This results in taut paths around obstacles as shown in the illustration 260 (FIG. 2D). [0045] Once the wheelchair moves from its location, replanning of paths in different homotopy classes is required in accordance with the present embodiments. To be able to assign probability to the new paths based on past user actions, it needs to be assured that the new paths can be linked to their homotopy class. For this, the new position of the wheelchair is joined to each path in the list of paths inherited from the previous time step. It is known that this generates new collision free paths from the wheelchair’s current position because the wheelchair traversed the new segment added to each path. Thus, the new paths remain connected to their homotopy class. However, this may introduce an unnecessary turn in the new paths. So, in accordance with the present embodiments, the same contraction and interpolation procedure is applied that is used for path smoothing in order to smoothen out the unnecessary turn.

[0046] The initial paths with unique homotopy classes are generated for obstacles near the wheelchair. As the wheelchair approaches the end goal, many of these paths become infeasible because they are tied to the obstacles that were near the wheelchair’s starting position. Therefore, it is necessary to explore for new homotopy classes. At every replanning step, we find K’ homotopy paths from the wheelchair’s current position. If these paths have a different homotopy class than the paths already found, they are added to the list of homotopy paths. After this, the longest paths except the most likely path are removed from the list until the path list has only K paths left. Referring to FIGs. 3A, 3B and 3C, illustrations 300, 330, 360 depicts illustrations of the effects of exploring new homotopy classes in accordance with the present embodiments. The illustration 300 depicts initial homotopy paths while the illustration 330 depicts homotopy paths after the wheelchair moves without exploration of new homotopy classes. The illustration 360 depicts homotopy paths after the wheelchair moves with exploration of new homotopy classes in accordance with the present embodiments, advantageously providing additional options resulting from additional homotopy paths added to the list of homotopy paths.

[0047] After the set of possible paths I in unique homotopy classes have been generated, at every timestep t, maintaining a probability distribution p(iΙu_{o ...t}, E) over these paths indicates how likely the path i is after observing user actions u_{0 ...t} until time t. E consists of the wheelchair’s current position (ω_cω _y , θ_Z), velocity (ώ_x, ώ_y, θ_z) and environment map M which is assumed to be fully observable. User action u_t, is defined by the joystick input ( j_x,j_y )· The path i is represented by a set of equidistant waypoints, with each waypoint p defined by the position (p_x, p_y) on the map M. To calculate the probability distribution p(iΙu_{o ...t}, E ), a standard approach is to use Bayes rule and compute p(u_tΙi,E) o p(u_tΙi,u_{o ...t-1} , E), assuming user action at time t is independent of previous actions given the preferred path i_u and the environment E. The probability distribution is shown in Equation (1).

[0048] Next, p ( u_t Ιi,E ) is computed using the principles of MaxEntIRL which states that p(u_tΙi, E) is proportional to the exponential of the cost to reach the path i. The computation is shown in Equations (2) to (5):

[0049] The cost C_i(u_t) to reach the path i consists of two weighted components: the command cost Ccmd and the control cost Cctri- The command cost C_cmd is calculated by the difference between the direction from the user’s joystick input 9_joystick = tan ¹ - ^]j- and the orientation of a waypoint p 6L = tan —1 — Pv selected dynamically (as explained

Px later) along the path i.

[0050] The control cost Ca_rl denotes the difference between the current heading of the wheelchair ( q_z ) and the orientation q_r of the waypoint p. The temperature factor t allows adjustment of the rate of change of belief. Only the angular difference is examined because the distance is calculated from the waypoints selected dynamically along every path i e / based on the current linear speed of the wheelchair, i.e., the selected waypoint p on a certain path i is the closest waypoint which is at least at a distance s = ^_linear ^x t along the path i, with / as a time horizon factor and vunear is the speed of the wheelchair.

[0051] The way to calculate this cost function (ii_t) comes from the observation of the users’ operation of the wheelchair: whenever a user aims to move along a preferred path, intuitively he/she would like to steer the joystick towards the direction of that preferred path.

[0052] After having the probability distribution on the set of preferred paths, the probability is used to calculate the action a which takes the user towards the goal via user’s preferred path. Theoretically, this problem can be modelled as a partially observable Markov decision process (POMDP), a principled way to do planning under uncertainty, with the human preferred path being the hidden part of the state. However, solving POMDPs with continuous actions is currently not feasible. Also, for shared control of a robotic arm, information gathering actions are not preferred by users during shared control as they tend to push the robotic arm towards one goal which might not be the user’s preferred goal. Thus, the POMDP is solved as a QMDP using hindsight optimization. A QMDP (i.e., a Markov decision process where Q refers to Q-values) is the fully observable approximation of a POMDP policy and relies on the Q-values to determine actions. For grasping tasks, it is observed that a policy under goal uncertainty exists which takes the robotic arm towards a center of the goals. This policy works for the grasping task as there is a common path for all the goals.

[0053] However, for navigation tasks, going towards the center of all possible paths will be confusing for humans and, unlike grasping objects, a user might have to make considerable effort to turn towards the preferred path. Therefore, the QMDP is solved as a MDP by using a most likely path as the user’s preferred path to represent a system state along with environment E and calculate the best action for this state using the dynamic window approach (DWA).

[0054] The dynamic window approach (DWA) is a conventional method for local obstacle avoidance. The original DWA defines a goal G on the map and obtains an optimal action a from a set of actions A generated by a dynamic velocity window to minimize a cost function C as shown in Equations (6) and (7): a = arg mm G a (6)

C = w_c Clearance + w_h Heading + w_v Velocity (7) [0055] The cost function C consists of three components: Clearance , Velocity and Heading. The Clearance measures the distance to the closest obstacles, which indicates the spaciousness of the surroundings. The Velocity evaluates the cost on linear and angular velocities, where a faster speed, if allowed, is always preferred. The Heading calculates the difference between the current heading and the direction to the goal G, which denotes the progress towards the target location. And w_c, w_h, and w_v are the weightings assigned to each component. [0056] Because the original DWA has some drawbacks and is not integrated with the shared control framework, an improved Shared-DWA is used where the cost function C is computed as shown in Equations (8) and (9):

C = 1 — Clearance + Clearance Cost_cmd (8)

Cost_cmd = w_h Heading + w_v Velocity (9)

[0057] The distance values here in Clearance are normalized in [0, 1] considering the dynamic constraints and both linear and angular distances to obstacles, which handles the safety of the system. Thus, to ensure the safety, the Clearance always has the highest priority over the rest of the cost. The cost for user control Cost_cmd is further comprised of Heading and Velocity. To conform with the shared control framework in accordance with the present embodiments, the Heading function here measures how well the executed heading is aligned with the direction obtained from the user’s joystick input u. The Velocity function describes how close the executed linear speed is to the user’s desired linear speed, which again comes from the joystick input u.

[0058] Because the systems and methods in accordance with the present embodiments are additionally responsible for helping users to move along their preferred path and handle complicated tasks, like doorway traversal and sharp U-tums, using Shared- DWA alone is insufficient given its nature of only taking a user’s instantaneous input into account, especially for those who lack fine motor control, where a small mis-input might lead to some unwanted consequences.

[0059] Therefore, a method called goal-based Shared-DWA is proposed in accordance with the present embodiments which not only guarantees the user’s low level control over the system and secures the user, but also helps the user navigate along the desired path and deal with difficult tasks. To achieve this, the waypoint p along the most likely path is initially obtained. Then an extra component is added into the cost function for the computation of optimal action, where the added component is defined as Cost_p and is calculated as shown in Equation (10):

Cost_p = w_distance Distance + w_direction Direction (10)

[0060] The Distance measures the distance between the waypoint p and the rolling out trajectories generated from dynamic window, while the Direction measures the alignment with the orientation of the waypoint p when a trajectory is closest to p. By adding a function Cost_p, systems and methods in accordance with the present embodiments start to intervene in the control to help navigate along the most likely path instead of pure obstacle avoidance as in Shared-DWA. Then, we compute the weighted sum of Cost_cmd and Cost_p are computed to get a total cost as shown in Equation (11):

Cost_total — w_cmd Cost_cmd + W_goal Cost_p (11)

[0061] Tuning the values of w_cmd and w_goai enables adjustment of the extent of system intervention.

[0062] Finally, using similar computation as Equation (8), the best action for goal- based Shared-DWA in accordance with the present embodiments can be obtained as shown in Equations (12) and (13):

« = arg min C

(12)

C = 1 — Clearance + Clearance Cost_total (13)

[0063] There might be situations where the most likely path calculated is wrong. Observation are made that, before approaching areas with less free space like doorways, users give enough indication about their preferred homotopy class and the most likely path is generally correct when the user reaches a crowded environment. When the users are in space with relatively less obstacles, they want to roam around freely and also often change their preferred paths. So, to prevent a situation where a wrong prediction undesirably intervenes into control, the Clearance is used to dynamically adjust the weightings between Cost_cmd and Costp.

[0064] As mentioned above, the normalized Clearance is an indicator of safety and spaciousness. A value of 0 means the corresponding action leads to an inevitable collision, while a value of 1 means the action is assured to be safe. All the values in between represent an index of “dangerousness”, where a value closer to 0 indicates a more dangerous action and a value approaching 1 means a less risky operation. Based on the dangerousness index, the sum of the normalized Clearance for all actions in A actually suggests the spaciousness of the environment E given the current state. Hence, when the sum is high, a higher weighting will be assigned to Cost_cmd since the space is relatively safe to drive. And when the sum is low, Cost_p will be the dominant component as the space is likely to be confined and a system intervention will better assist the user to finish the navigation task. The weightings are computed as shown in Equations (14 and (15):

Wgoal — 1 ^cmd (15)

[0065] It is empirically observed as described hereinbelow that using Clearance to determine the weight of a user command in the reward function leads to the user expected behaviour most of the times. However, theoretically, the most likely path can still be wrong and lead to the wheelchair taking the user in an unwanted direction. Some conventional techniques address this issue by providing assistance only when the entropy of probability distribution over various paths is low or a probability of the most likely path is higher than a threshold. However, such solutions do not work in navigation scenarios where many paths initially follow a similar path segment but then diverge later. In such a scenario, no assistance will be provided to the user to enter the doorway as all the paths have almost equal probability.

[0066] To validate the methods and systems in accordance with the present embodiments, a series of experiments were conducted in simulations as well as on the real wheelchair as described below. The objective of the experiments was to verify the following hypothesis: The methods and systems in accordance with the present embodiments require less operating effort and are able to complete tasks more quickly as compared to Shared-DWA while giving the user control over the wheelchair comparable to that given in Shared-DWA.

[0067] Two test scenarios were used for simulation as shown in illustrations 400, 450 of FIGs. 4A and 4B, where the illustration 400 depicts an Amazon Web Services (AWS) hospital scenario and the illustration 450 depicts the doorway traversal scenario. Both scenarios were established in ROS and Gazebo and, in both scenarios, a starting point 410, 460 and a destination 420, 470 were fixed and a predefined path 430, 480 between the two was created, meaning the nominal “goal” and “preferred path” were all preset for easier comparison. The preferred path was known only to the user but unknown to the shared control method. The simulation test was conducted with eighteen healthy subjects (nine males and nine females) and two subjects with Cerebral Palsy (two males). All procedures were approved by the Institutional Review Board (IRB) of Nanyang Technological University.

[0068] Before their test, each subject could first drive the wheelchair in simulation for five minutes so that they were able to get familiar with the operation, speed, acceleration and even collision of the simulated wheelchair operation. To mimic hand impairment, all subjects were asked to perform the experiment with their non-dominant hand and with a 2kg weight worn on their non-dominant hand as can be seen in the photograph 500 in FIG. 5 to temporarily introduce stiff palm and fingers that limit fine motor control. As a reference, subjects were also asked to finish another set of experiments using their dominant hand but without the weight. For comparison with the conventional Shared-DWA method, the same experiments were conducted for all subjects using Shared-DWA.

[0069] During the test, the subjects were asked to follow the path as best as they could while navigating to the goal. Each subject was asked to complete the task twice in each scenario (i.e., the hospital scenario and the doorway traversal scenario) for each method (method in accordance with the present embodiments and the conventional Shared- DWA method) with each hand-weight combination (non-dominant hand with weight and dominant hand without weight), so every subject was supposed to finish sixteen trials in total.

[0070] As mentioned, an additional two subjects with a diagnosis of Cerebral Palsy were also recruited for the experiment. The test for these subjects was similar to the test with the healthy subjects except that these two subjects did not have to wear the 2kg weight on their non-dominant hand. Additionally, these subjects only needed to use their dominant hand. Therefore, each of these two subjects was asked to complete the task twice in each scenario (the hospital scenario and the doorway traversal scenario) for each method (method in accordance with the present embodiments and the conventional Shared-DWA method) with their dominant hand, i.e., each subject had to finish eight trials in total.

[0071] Real wheelchair experiments were conducted with the Cerebral Palsy subjects where they were seated on real wheelchair 1410 as shown in an illustration 1400 of FIG. 14. The wheelchair 1400 included LIDAR range scanners 1420 and a joystick input device 1430. The subjects had to provide user inputs via the joystick input device 1430 with their dominant hand. The real world test comprises a scenario as shown in a photograph 1500 of FIG. 15 which requires the user to exit and enter doors while making sharp turns. The start position, the goal position and a preferred path were marked with a dashed line of tape 1510 as a guide to the user. The real world test scenario is a simplified version of the hospital scenario in which cardboard boxes 1520 represent doorways and walls. The user has to first exit a “door” 1530, make a sharp U-turn 1540 to enter a next door 1550, and follow the dashed line of tape 1510 to finish the task.

[0072] To estimate and assess performance, we recorded and computed several parameters of every trial, including the average trial completion time t_{ta k}, and the dynamic time warping (DTW) distance dmw between the predefined path and the path finished by subjects. The t_task indicates how fast a user is able to complete the task, and the dmw shows how much a user is able to follow his/her preferred path.

[0073] After each trial for the healthy subjects, the subjects’ feedback about the trial regarding the feeling of control, preference of methods, dissatisfaction, and other factors was requested to solicit ideas and suggestions for future improvement. After the test with the disabled subjects, a semi- structured interview was also conducted, asking for suggestions and complaints from them and their caregivers about using wheelchairs.

[0074] The experiment results for healthy subjects are shown in Table I, where the data is broken down by algorithm (Shared-DWA and the method in accordance with the present embodiments labelled “Proposed Method”), scenario (doorway and hospital), and load (dominant hand without weight and non-dominant hand with weight). TA L f

[0075] FIGs. 6A and 6B depict two attempts for the first disabled subject of the Shared-DWA hospital scenario, where the dashed line represents the reference path, the rectangles represent doorway areas, and the solid line represents the user’s trajectory. In a similar manner, FIGs. 7A and 7B depict two attempts by the same subject for the hospital scenario using the simulated method and system in accordance with the present embodiments. FIGs. 8A and 8B depict two attempts for the second disabled subject of the Shared-DWA hospital scenario, while FIGs. 9A and 9B depict two attempts by the second subject for the hospital scenario using the simulated method and system in accordance with the present embodiments. The improvement in operation from the conventional Shared-DWA method and the method and systems in accordance with the present embodiments can clearly be seen for both subjects.

[0076] Similarity, FIGs. 10A and 10B depict two attempts for the first disabled subject of the Shared-DWA doorway traversal scenario, where the dashed line represents the reference path, the rectangles represent doorway areas, and the solid line represents the user’s trajectory. FIGs. 11 A and 1 IB depict two attempts by the same subject for the doorway traversal scenario using the simulated method and system in accordance with the present embodiments. FIGs. 12A and 12B depict two attempts for the second disabled subject of the Shared-DWA doorway traversal scenario, while FIGs. 13A and 13B depict two attempts by the second subject for the doorway traversal scenario using the simulated method and system in accordance with the present embodiments. In the doorway traversal scenario also, the improvement in operation from the conventional Shared-DWA method and the method and systems in accordance with the present embodiments can clearly be seen for both subjects.

[0077] FIGs. 16A and 16B depicts a photograph 1600 and an illustration 1650 showing how the method and system in accordance with the present embodiments helps one of the Cerebral Palsy subjects enter a narrow door in a situation where conventional Shared-DWA would have steered the user away. While both a Shared-DWA output 1610 and an imprecise user input 1620 would have steered the wheelchair 1410 away from the narrow doorway, the intention prediction based output 1630 in accordance with the present embodiments correctly steers the robotic wheelchair 1410 through the narrow doorway.

[0078] The experimental results for the two disabled subjects are summarized in Table II, where the data is broken down by algorithm (Shared-DWA and the method in accordance with the present embodiments labelled “Proposed Method”), and scenario (doorway, hospital and Real Test). s.

[0079] Using a repeated-measures Analysis of Variance (ANOVA), it was found that the healthy subjects completed the tasks within 34.28 seconds on average with the method in accordance with the present embodiments, which was reliably faster as compared to 37.88 seconds using the conventional Shared-DWA [F( 1,17) = 7.52 , p < 0.05]

[0080] Task performance using the 2kg weight limited fine motor control and increased time spent to complete the task. The healthy subjects spent on average 37.23 seconds with a weight, as compared to 34.93 seconds without the weight [F(l , 17) = 9:35, p < 0.01]

[0081] Moreover, the hospital scenario required longer completion time (47.69 seconds) than the doorway scenario (24.54 seconds) [F(l,17) = 75.62,/? < 0.0001], and both algorithms gave a similar completion time (no algorithm x scenario interaction, F(l,17) = 1.21, p > 0.1).

[0082] When looking into the way the users followed a given path, both algorithms were found to be comparable [F(l,17) = 0.93,/? > 0.1]. This suggests that the methods and systems in accordance with the present embodiments was at least as efficient as the Shared-DWA algorithm in allocating control authority to follow a user-preferred path. [0083] Throughout the experiment session, subjects’ operation of the joystick was observed and, as mentioned above, feedback was requested after each trial. Pertinent points are summarized hereafter.

[0084] (I) When subjects could complete all trials quite well, they tended to prefer Shared-DWA because they felt more control over the robot, even though statistic data showed the proposed method in accordance with the present embodiments was still slightly better. However, those subjects who could not exert fine motor control rated the proposed method higher because they could complete the trial faster and more easily.

[0085] (II) When subjects were approaching a narrow doorway in the hospital scenario, because of the narrow space, the intervention from the system increased. However, since the system took a shortcut to enter the door, which most subjects thought might be dangerous, subjects started to fight with the system, trying to pull the wheelchair back to their desired angle to enter the door.

[0086] (III) All the subjects admitted that the proposed method in accordance with the present embodiments did require less effort while completing the tasks so that they did not need too much concentration, especially during high-speed driving.

[0087] (IV) One of the subjects was not so satisfied with the proposed method in accordance with the present embodiments because she felt the method of path planning was too aggressive because of the nature of cutting short, and she preferred a gentler way to finish the task.

[0088] (V) Another subject expressed that when the system started intervening, he could obviously feel the wheelchair was not listening to him and was confused as to whether he gave a wrong input, so he said it would be better to be notified explicitly when the system intervened.

[0089] These points suggest that no matter how good or bad a user is at fine motor control, he/she always prefer to have more control over the wheelchair navigation. These points also convey that users lack a trust in the system, which leads to fighting the system when there is a clash between system and users. Thus, a user-friendly way may assist users and make the intervention as imperceptible as possible. Also, adding explicit feedback to inform and interact with users so that users are acquainted with the system intervention may improve operation of the methods and systems in accordance with the present embodiment.

[0090] From Table II, it can be seen that the task completion time in both the simulation scenarios is significantly shorter for both subjects when using our proposed method. Subject 1 was not able to finish the task twice in the hospital scenario when using shared-DWA because he kept pointing at the wall instead of the gap in the door. The intention prediction-based method in accordance with the present embodiments could correctly steer him towards the door and allow him to complete the task every time. Thus, the methods and systems in accordance with the present embodiments not only helped in completing the task faster for the Cerebral Palsy subjects, it also enabled them to perform fine motor control tasks. Similarly, with Subject 2 in the real wheelchair experiment, the shared-DWA stopped him while he was trying to exit a door while the method in accordance with the present embodiments helped him exit the door. This is reflected in around 30% longer completion time for Subject 2 with shared-DWA in the real test scenario. However, as can be seen from the users’ paths in FIGs. 6A, 6B, 7A, 7B, 8A, 8B, 9A, 9B, 10A, 10B, 11 A, 11B, 12A, 12B, 13A and 13B, both the subjects had ample deviation from the desired path due to their disability. This kind of deviation led to false intention prediction sometimes resulting in wrong assistance. Modelling the user’s disability, personalizing the human intention prediction, and planning under uncertainty using POMDPs can help address this issue.

[0091] Thus, it can be seen that the present embodiments provide a shared control system and methods which can use the knowledge of a final goal of a user to better assist the user in navigation and also provide appropriate control authority to the user. While powered wheelchair examples have been described and presented, the shared control of goal directed navigation in accordance with the present embodiments is equally applicable to other robotic or vehicular navigation control. The quantitative experiments done in simulation with the eighteen human subjects show that the system and methods in accordance with the present embodiments reduces the time and effort required to complete the task as compared to a baseline system which does not take user intention into account. However, the deviation from the reference path is comparable for both the systems which indicates that control authority is similar for both the systems. Subjective comments from users indicate that users whose disability is diminished after introducing weights prefer the system and methods in accordance with the present embodiments over the baseline system.

[0092] Experiments with cerebral palsy subjects in a hospital environment clearly demonstrate how the system and methods in accordance with the present embodiments can help users with upper limb disability as one of the subjects could complete the task with only the system and methods in accordance with the present embodiments.

[0093] It is expected that the system and methods in accordance with the present embodiments can be made adaptive to the specific disability of the user operating wheelchair and can take into account dynamic obstacles in the environment. Also, modifications can address unwanted assistance when the most likely path is different from the user’s intended path.

[0094] The power wheelchair market was at $ 1.1 billion in 2011 and is anticipated to reach $3.9 billion in 2018. The market growth comes in large part from demand for mobility from people who might otherwise be bedridden. The growing population of the elderly, particularly the groups over the ages of 65, 75 and 85 represent the primary growth driver for the wheelchair market and future growth in the market will be driven by the growing demand for powered wheelchairs supported by their superiority over manual wheelchairs in terms of enhanced efficiency and improved functionality. [0095] Powered wheelchairs require cognitive and physical skills that not all individuals possess. A survey indicates ten to forty percent of elderly who desired powered mobility could not be fitted with powered wheelchairs due to sensory impairments, poor motor function, or cognitive deficits made driving safely impossible with any of the existing controls. The trends of the wheelchair market are to incorporate latest and effective techniques such as advanced Human Robot Interface (HRI), navigation and obstacle avoidance technology, stairs climbing technology and adaptive assistance technology such as the techniques provide by the systems and methods in accordance with the present embodiments.

[0096] While exemplary embodiments have been presented in the foregoing detailed description of the present embodiments, it should be appreciated that a vast number of variations exist. It should further be appreciated that the exemplary embodiments are only examples, and are not intended to limit the scope, applicability, operation, or configuration of the invention in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient road map for implementing exemplary embodiments of the invention, it being understood that various changes may be made in the function and arrangement of steps and method of operation described in the exemplary embodiments without departing from the scope of the invention as set forth in the appended claims.

Claims

CLAIMS What is claimed is:

1. A shared control of goal directed navigation method comprising: generating a set of user intentions for navigating a plurality of paths to a predetermined goal; predicting one of the set of user intentions as a preferred path in response to computing a probability of various ones of the plurality of paths to the predetermined goal; and using the preferred path as a guide for a shared dynamic window approach (DWA) based local path planning.

2. The method in accordance with Claim 1 wherein generating the set of user intentions for navigating the plurality of paths to the predetermined goal comprises using homotopy classes to restrict a number of the plurality of paths.

3. The method in accordance with Claim 2 wherein generating the set of user intentions for navigating the plurality of paths to the predetermined goal further comprises using voronoi diagrams to compute ones of the plurality of paths belonging to each of the homotopy classes.

4. The method in accordance with Claim 2 or Claim 3 wherein computing the probability of various ones of the plurality of paths to the predetermined goal comprises maintaining a probability distribution over the various ones of the plurality of paths in the homotopy classes.

5. The method in accordance with Claim 4 wherein maintaining the probability distribution over the various ones of the plurality of paths in the homotopy classes comprises maintaining the probability distribution over the various ones of the plurality of paths in the homotopy classes using MAXEntIRL.

6. The method in accordance with any of the preceding claims wherein computing the probability of various ones of the plurality of paths to the predetermined goal comprises computing the probability based on a cost function.

7. The method in accordance with any of the preceding claims wherein the shared DWA based local path planning comprises recognizes user input, the method further comprising determining a final control command based on the user input.

8. The method in accordance with Claim 7 wherein determining the final control command for the goal directed navigation comprises determining the final control command for the goal directed navigation using inner loop blending.

9. The method in accordance with Claim 7 or Claim 8 wherein determining final control command for the goal directed navigation comprises determining final control command for the goal directed navigation while calculating a reward for various possible velocities in the shared DWA based local path planning.

10. The method in accordance with any of Claims 7 to 9 wherein determining a final control command based on the user input comprises determining a final control command based on a weighted user input, wherein a weighting factor for the weighted user input is determined in response to a measure of crowding of an environment of the preferred path.

11. The method in accordance with Claim 10 wherein the weighting factor for the weighted user input is further determined in response to an action which navigates towards the predetermined goal via a likely homotopy class.

12. A shared control goal directed navigation system comprising: a user control device; and a shared navigation controller coupled to the user movement control device and generating a set of user intentions for navigating a plurality of paths to a predetermined goal, predicting one of the set of user intentions as a preferred path in response to computing a probability of various ones of the plurality of paths to the predetermined goal, and using the preferred path as a guide for a shared dynamic window approach (DWA) based local path planning, wherein the shared DWA based local path planning comprises recognizes user input received from the user movement control device, the shared navigation controller further determining a final control command based on the user input.

13. The system in accordance with Claim 12 wherein the shared navigation controller generates the set of user intentions for navigating the plurality of paths to the predetermined goal comprises using homotopy classes to restrict a number of the plurality of paths and using voronoi diagrams to compute ones of the plurality of paths belonging to each of the homotopy classes.

14. The system in accordance with Claim 13 wherein the shared navigation controller computes the probability of various ones of the plurality of paths to the predetermined goal by maintaining a probability distribution over the various ones of the plurality of paths in the homotopy classes.

15. The system in accordance with any of Claims 12 to 14 wherein the shared navigation controller computes the probability of various ones of the plurality of paths to the predetermined goal by computing the probability based on a cost function.

16. The system in accordance with any of Claims 12 to 15 wherein the shared navigation controller determines the final control command for the goal directed navigation using inner loop blending.

17. The system in accordance with any of Claims 12 to 16 wherein the shared navigation controller determines the final control command for the goal directed navigation while calculating a reward for various possible velocities in the shared DWA based local path planning.

18. The system in accordance with any of Claims 12 to 17 wherein the shared navigation controller determines the final control command based on a weighted user input, wherein the shared navigation controller calculates a weighting factor for the weighted user input in response to a measure of crowding of an environment of the preferred path.

19. The system in accordance with Claim 18 wherein the shared navigation controller calculates the weighting factor for the weighted user input further in response to an action which navigates towards the predetermined goal via a likely homotopy class.

20. The system in accordance with Claim 18 wherein the shared navigation controller is a wheelchair navigation controller.

21. A computer readable medium comprising instructions for a shared navigation controller to perform a method for shared control of goal directed navigation, the instructions causing the shared navigation controller to: generate a set of user intentions for navigating a plurality of paths to a predetermined goal; predict one of the set of user intentions as a preferred path in response to computing a probability of various ones of the plurality of paths to the predetermined goal; use the preferred path as a guide for a shared dynamic window approach (DWA) based local path planning, wherein the shared DWA based local path planning comprises recognizes received user input; and determine a final control command based on the user input.