KR20080005798A - A cognitive and conduct disorder rehabilitation therapy systems using mothion tracking technologies and augmented reality - Google Patents

Abstract

A system for supporting cognitive and conduct disorder rehabilitation is provided to offer an interactive rehabilitation support system by using interactive HCI(Human Computer Interface) technique and game technique through moving pictures. A system for supporting cognitive and conduct disorder rehabilitation comprises an input part(1), an image process part(2) and an output part(3). The system is composed of a cognitive disorder rehabilitation support system for measuring concentration power, reactive speed, and accuracy of cognitive-disorder patients, and a conduct disorder rehabilitation support system for measuring concentration power and hand motion of conduct-disorder patients. The cognitive disorder rehabilitation support system consists of a visual reaction function module, a visual differential reaction function module, and a visual reaction and move function module. The conduct disorder rehabilitation support system consists of a visual track and target function module and a visual spatial and motor function module.

Description

A Cognitive and Conduct Disorder Rehabilitation Therapy Systems using Mothion Tracking Technologies and Augmented Reality}

1 (a) and (b) is a view for explaining the concept and actual hardware configuration of the rehabilitation support system according to the present invention,

2 is a flowchart schematically showing a process of a rehabilitation support system according to the present invention;

3 is a flow chart showing in detail the processing of the rehabilitation support system according to the present invention;

Figure 4 (a) shows the result of executing the CAMSHIFT used in the present invention, (b) shows the execution result of MeanShift,

5 is a flowchart illustrating a CAMSHIFT algorithm;

6 is a view showing the tracking results of the direction and X, Y coordinates and the horizontal and vertical size of the object using the color distribution of the object,

(A) and (b) and (c) and (d) of FIG. 7 illustrate a comparison between MeanShift and CAMSHIFT for changes in the size of an object seen on the screen when the object moves toward or away from the camera.

8 is a view showing the configuration and flow of the rehabilitation support system implemented using the CAMSHIFT algorithm according to the present invention,

9 is a flowchart illustrating a process of performing a Visual Reaction function by the Visual Reaction system;

10 is a screen shown by executing the Visual Reaction function by the Visual Reaction system,

11 is a screen in which a patient selects a rectangular area,

12 is a screen showing the situation after selecting the second button,

13 is a screen showing the Visual Reaction measurement data,

14 is a flowchart illustrating a process of performing a Visual Differential Reaction function by the Visual Differential Reaction system;

15 is a screen shown by executing the Visual Differential Reaction function by the Visual Differential Reaction system;

16 is a flowchart illustrating a process of performing a Visual Reaction and Move function by the Visual Reaction and Move system;

17 is a screen shown by performing the Visual Reaction and Move function by the Visual Reaction and Move system,

18 is a flowchart illustrating a process of performing a Visual Track and Target function by the Visual Track and Target system;

19 is a screen showing a path setting and an image button of the Visual Track and Target system,

FIG. 20 is a screen illustrating a path and a button layout of a screen of a visual track and target system that is displayed when each path is selected by the image button of FIG. 19;

FIG. 21 is a screen illustrating a state in which a full path is shown, a state in which no path is shown, and a state in which a partial path is displayed in the Visual Track and Target system;

22 is a flowchart illustrating a process of performing a Visual Spatial and Motor function by the Visual Spatial and Motor system;

23 is a view showing the width and speed control of the visible bush by the Visual Spatial and Motor system,

24 is a screen showing a balloon and a thorn bush by the execution of the Visual Spatial and Motor system,

25 is a view showing a process of recognizing and analyzing a pattern image;

26 is a graph showing the results of evaluation by the therapist's own experience,

27 is a graph showing the therapist opinion after applying the program of the present invention to a patient,

28 is a graph showing the results of a survey of the average time until the patient is mastered by the CogReHab program and the program implemented in the present invention;

29 is a diagram showing a systemic model of a motor portion of a conventional HCI;

30 is a view showing the interaction for rehabilitation training applied to the model of FIG. 29,

31 is an initial screen and execution screen of a conventional PSS CoReHab-Foundations I.

<Description of the code | symbol about the principal part of drawing>

1: input unit (camera), 2: image processing unit (computer),

3: Output part (display device).

The present invention relates to a real-time cognitive and behavioral rehabilitation support system using the CAMSHIFT algorithm, in particular the use of interactive human computer interface (HCI) technology, such as gesture recognition and motion tracking in the rehabilitation treatment of patients with cognitive and motor disorders In addition, the present invention relates to an interactive rehabilitation support system using video game techniques.

In recent years, advances in science and technology for medical care have resulted in lower mortality rates while significantly increasing survival rates. However, increased survival rates also mean that more people have to live with cognitive impairments throughout their lives. This is because most of the surviving patients have great difficulty living independently due to sequelae such as cognitive impairment and have limitations in returning to normal society.

The role of rehabilitation medicine in modern medicine is growing in importance over time. In the past, physicians thought they were responsible for treating patients only by diagnosing the pathology and performing or administering the surgery needed to remove the lesion, but current physicians have abandoned this approach and have made more observations with deep human attention. Attempts have been made to find that active efforts to solve patients' problems are needed, taking into consideration not only the diseases that patients have but also the physical illnesses and disabilities as well as psychological and social aspects.

Today, about 10% of the world's population consists of people with disabilities with many forms of disability, and these disability populations have limited activities in the communities in which they live. This started to raise the issue of rehabilitation for the disabled population and the development of rehabilitation medicine. Rehabilitation of the disabled has begun to be recognized as an important part of public health as part of comprehensive care.

Cognitive function refers to the ability to distinguish, select, accept, understand, and retain appropriate information, and to search for and apply information relevant to the situation in which it is needed. Therefore, when cognitive impairment occurs, the efficiency of such information processing decreases, the speed and continuity of cognitive function decreases, and the function for daily life decreases, and when a problem occurs, it is difficult to properly respond to it.

Computer information technology is increasingly penetrating the medical field. Rehabilitation patients have less ability to accept input from the physical world, less output to the physical world, ie voice or movement, and also allow novice users to control various functions on the computer screen for the convenience of novice users. In the past decades, tools have been used by humans and computers to interact with each other, such as the mouse and keyboard, a graphical user interface that has been developed as an alternative device for input to the computer and functions as interactive point-and-click functions. Has been. Software designers used the technology to create programs for several purposes. But people who couldn't use a keyboard or mouse could not use this program.

In FAce MOUSe [Nishikawa et al., 2003], surgeons can control laparoscopic movements by creating appropriate facial gestures without hand or foot switches or voice input. Focus is used as one of the diagnostic imaging techniques for selecting CT images by eye movement [Yanagihara et al., 2000]. Vision-based gesture capture systems have been proposed for manipulating windows and objects under a graphical user interface. Currently, a study by Graetzel et al. (Graetzel et al., 2004) has been conducted to reflect hand gestures as an interface between doctors and computers. They developed a computer vision system that uses hand gestures to allow the surgeon to move the pointer and click a button, a standard function of the mouse.

Patients with traumatic brain injury, structural damage to the brain, such as stroke and cerebral palsy, are accompanied by various brain disorders. Among these rehabilitation patients, not only motor impairment but also attention concentration, memory, judgment, problem Cognitive impairment, such as solving ability and planning ability, especially perceptual impairment, which is the ability to process and receive information from sensory organs.

This impairment of cognitive function and perceptual ability seriously impairs rehabilitation and rehabilitation ability after brain injury, and accurate and comprehensive evaluation should be prioritized for the treatment of these patients. Intensive treatment will be performed on the part. Basically, various medications, physical therapy, occupational therapy, psychotherapy, speech therapy, etc. are implemented. Recently, various diagnostic and treatment programs using advanced computer technology for cognitive and perceptual impairment have been developed and put into practical use. I have reached the stage. However, many computer cognitive therapy programs developed and distributed in English are difficult to apply due to language comprehension or communication problems, and domestic programs have recently been spreading but are still in low clinical utilization. to be.

In this regard, there is a method of technically using a virtual reality system to evaluate the performance of patients and provide a treatment environment. In particular, in the case of limited athletic ability, it is possible to safely remove various restrictions from the virtual reality space in 3D and receive treatment safely.

Such an example is widely known in the handicapped driving training and electric wheelchair adjustment training in 3D virtual space, and the part learned in the virtual reality space is known to be applicable in the real environment. In addition to the improvement of athletic ability and training program, the evaluation of various cognitive disorders and daily living behaviors can be evaluated and treated to more practical and specific parts through evaluation in the three-dimensional virtual reality space. It is expanding to. In other words, the advantage of the virtual reality approach over traditional neuropsychological tests is the ability to reproduce and reproduce the functions of everyday life.

In addition, the task can be repeated while adjusting the degree of difficulty at various and adjustable levels, and the errors and variability at the time of inspection are small. As mentioned earlier, safety is one of the advantages. However, there are also side effects of virtual reality, and early studies reported eye pain, loss of strength, loss of balance, etc., which was thought to be caused by delayed or distorted images on the computer display. Recently, this point is being supplemented by the development of image technology.

Recently, a program using a motion tracking system introduced in various game industries directly performs work using a subject or a body of a subject projected through a camera. In particular, in the case of patients with various perceptual impairments, such as unilateral disregard or cognitive impairment of the body, visual information about their body parts is used, and at the same time, specific tasks are used to improve the performance of the motion itself. It can be applied directly to the examination and treatment of visual-movement coordination. In addition, various cognitive therapy programs can attempt to treat new concepts that are different from existing two-dimensional or virtual reality treatments. do.

Hereinafter, the concept of cognition will be briefly described before explaining various cognitive rehabilitation training programs.

In other words, cognition refers to a series of processes included in knowledge, understanding, learning, perception, memory, judgment, and thought.

Such cognitive processes are not easy to separate completely in any situation, but may be the opposite of physical, behavioral or emotional processes. It is not easy to determine whether a person is training a cognitive process or some other process by simply observing behavior.

Many existing tests have been developed by psychologists for each type of cognitive impairment and have been objectiveized and standardized. However, it takes a lot of time to test one type of cognitive state, and it is very difficult to interpret test results and judge normality. It has a complicated structure.

The assessment of patients' cognitive function requires a brief examination to identify the type of cognitive impairment and assess the extent of it in order to develop an appropriate rehabilitation treatment plan, and to evaluate the effects of neurological and physical problems on the results of mental function tests. It should be able to interpret with emphasis.

The assessment of cognitive function required with the development of rehabilitation medicine should be able to distinguish between cognitive symptoms and personal psychiatric, psychological and emotional problems, identify the effects of cognitive dysfunction on functional disorders, treatment outcomes and cognition. The effect of treatment on the disorder should be followed.

In order to meet the above objectives, various methods of measuring cognitive function on mental, cognitive, behavioral and psychological problems are being developed in rehabilitation medically. Newly developed measuring methods according to clinical needs are a single test. We are searching for a test method that can distinguish the extent of impairment and measure quantitatively in a short time.

In addition, as the technological development of various measurement methods for cognitive function, in addition to the activation of rehabilitation medical cognitive function evaluation, treatment of the evaluated cognitive function is more actively performed, and thus, drug treatment, cognitive function training, and compensatory technique training. Treatment of comprehensive cognitive impairment such as

Cognitive dysfunction is largely observed from two perspectives. One is to measure the degree of cognitive dysfunction in terms of psychiatric function and to measure the extent of cognitive dysfunction. will be. The latter are specific cognitive functions such as finger cognitive impairment, Gerstmann syndrome, cognitive impairment such as face, road, or place, linguistic other skills that can be read but not read, speech impairment, or impaired linguistic function Observations of disability, etc., were of considerable theoretical and academic interest, with limited clinical value.

Next, a brief description of the various cognitive rehabilitation training programs is presented.

First, the concentration rehabilitation program which is one of the cognitive rehabilitation programs will be described.

Cognitive rehabilitation program refers to training to improve patient's information processing speed and response speed by using simple tasks. It is usually developed and applied as a computer program. For example, whenever a number appears on the computer screen, press a specific key to make it respond (the numbers appear randomly in the center, right, and left of the screen) [Ponsford et al., 1988] Each time this occurs, training involves pressing a specific key as quickly as possible (Gray et al., 1992). The second is a rehabilitation program in which the four components of concentration (attention, selective concentration, cross concentration, split concentration) are applied in a step-by-step manner to increase the contents of the training. There has been an increase in studies applying programs developed in the field [Gansler et al., 1991; Sturm et al., 1997; Novack et al., 1996.

At the lowest level of concentration, attention is focused on training the patient's attention in one place. Gansler [Gansler et al., 1991] applied training for attention-based rehabilitation, in which the patient concentrates on the passage of time and raises his hand when he or she thinks he or she has passed 30 seconds or 1 minute. Other training exercises used by Gray and Robertson [Gray et al., 1992], which include pressing the space bar as quickly as possible whenever a red circle appears on a computer screen, is also an exercise of attention.

Level 2 concentration training is selective intensive training. Selective concentration refers to the ability to focus on one characteristic of a task while suppressing spontaneous responses to other unrelated characteristics. In selective intensive rehabilitation, tasks that can be trained to respond selectively are applied, and tasks with a relatively low mental burden are used for training. The training used by Gansler [Gansler et al., 1991] for selective intensive training was to find out whether any of the randomly-ordered alphabetic letters were vertically or horizontally ordered and to speak the letters in those areas. This includes training used by Ponsford and Kinsella [Ponsford et al., 1988] to find a figure in the center of the screen that matches a figure that appears on the left or right side of the screen.

The next step is cross-intensive training. Cross concentration is the ability to withhold one's response until all the information needed to understand the situation is gathered. The training method used by Gansler [Gansler et al., 1991] reads vertically aligned numbers. It is a drill that reads only even numbers without subtracting odd numbers and changes the rules in the middle to ignore even numbers. Other trainings used by Ponsford and Kinsella [Ponsford et al., 1988], such as pressing the right button when the red rectangle appears on the computer screen and pressing the left button when the green rectangle appears, are also cross-focused. Gray and Robertson [Gray et al., 1992], which involves briefly showing four numbers of the same length on the screen and then disappearing them, then showing the other four numbers and choosing the same number as the one shown earlier.

The next step is split concentration. Segmentation is the highest level of concentration, often referred to as the dual task paradigm, where information flows simultaneously through two or more sensory organs (e.g., visual-hearing), or where different types of information flow simultaneously through one sensory organ. Refers to the ability to stay focused on each stimulus independently at time A-time B. The training used by Gansler [Gansler et al., 1991] for segmentation concentration was to train the patient to read a book, while at the same time allowing the trainer to call up a list of words and guess the words the trainer called, Gray and Robertson [Gray]. et al., 1992] used the red, yellow, green, and blue letters on the screen to show the letters one by one and then randomly list the letters that match the meaning and color of the letters and the letters that do not. This includes training to find a match between meaning and color.

As a result of providing a concentration rehabilitation program, some studies have reported that the intervention of the program was effective in improving the degree of concentration, while in some studies the effect was not significant. In other words, it was suggested that repetitive research is necessary through systematic research design.

Next, a memory rehabilitation program which is one of cognitive rehabilitation programs will be described.

Memory impairments eventually lead to learning disabilities. One of the most difficult problems patients suffer from in their daily lives is that they are unable to acquire new knowledge because of memory impairment. For this reason, more studies on memory rehabilitation were available than other cognitive disorders. For example, the memory rehabilitation programs applied in the studies have acquired the Mnemonic Technique to directly retrain the degraded memory, and the use of external compensation systems such as the Memory Diary and Neuro Pages in daily life. It has been used in two types, such as training to reduce disability.

Next, Human Computer Interaction (HCI) will be described.

One of the most intuitive HCI models is the Model Human Processor (MHP) [Card et al., 1983]. It is divided into three parts to handle the interaction: Perceptual, Cognitive and Motor. The first two are for processing information and making decisions, and the third is for interacting with the physical world.

FIG. 29 shows a systematic model of the motor portion of a conventional HCI, which is another traditional model [Chapanis, 1965; MacKenzie et al., 1995, which is composed of human, set of input devices, and computer.

Here, humans regarded the system as producing an output signal, i.e. a gesture. These gestures are useful for interacting. Traditionally useful signals are produced by the movement of the legs, arms, or fingers, or by the pressure on the touch panel. Other more sophisticated methods are eye movements and sounds. From the signal produced by humans, we create S (t) to be used as the input to the input devices, which change into the form of A (t) with binary elements of zeros and ones. The A (t) element describes the state of the input devices and is sampled by the computer from the port or driver. When the state of A (t) changes, an event is triggered by the computer. 30 depicts the application of the model of FIG. 29 to the field of rehabilitation training.

Information representing the state of a single-switch input device having one switch is represented by one bit. So a (t) is the size of one bit. The key here is the features of the input device that will be installed to accept user input. The input device must detect the human response expressed as a bit, such as a button click. They should also accept the signal with minimal effort to avoid discomfort to the patient. For example, the way of clicking a button can be tiring for the user, and the sensors tagged on the user's head can make the user feel uncomfortable.

Historically, computer-based biomechanical devices have been used to monitor the rehabilitation process. Greenleaf Medical Company has developed an "Eval" system for orthopedic evaluation.

The system provides the ability to easily collect and store data and tools to analyze patient information stored in a database. Lafayette Instrument Company provides software for monitoring and evaluating patients. The company's products store data and display patient reports.

Various systems of this kind have been developed and used in hospitals. For example, the system developed by MIT helped the patient to work out with the robot and restore the upper body's energy. The system has an actuator, and the Milano Politechnic system is portable and has been developed for patients with Parkinson's disease. However, the above mentioned systems have the difficulty and complexity of manipulating the robot, making it difficult for patients to use.

On the other hand, computer-based cognitive therapy was first reported in 1986 by Glisky et al. When computer-based memory training began.In 1997, Chen et al. Applied computer cognitive rehabilitation programs to 20 patients with traumatic brain injury. In 2000, Palmese and Raskin reported a 10-week application of Attention Process Training-II (APT-II) to three patients with strong traumatic brain injury, resulting in improved concentration and performance.

The PSS CogReHab program of the US Psychological Software Services, which has been widely introduced in Korea, is divided into four categories: Foundation to train attention, memory to train memory, visual spatial to train space-time perception, and problem solving to train problem solving. It consists of 45 individual programs, and the PSS CogReHab program attempts to increase the concentration and activity of the patient by taking several levels of foundations.

31 shows an initial screen (a: top screen) and an execution screen (b: bottom screen) of PSS CoReHab-Foundations I. As shown on the screen, red and blue squares on the screen are displayed on the screen in Foundations I. At the point of, the patient uses the keyboard to press the left arrow if it is a red square and the right arrow if it is blue, and the program is designed to improve the attention and hearing ability of the patient by hearing the sound according to the patient's response. It became.

However, these programs have inconveniences in operation for people with disabilities such as keyboard or mouse in rehabilitation therapy or cognitive and cognitive impairment.

In addition, since the rehabilitation treatment does not end in a short period of time, the patient feels bored in the process of rehabilitation treatment, and there is a disadvantage of reducing the patient's motivation for continuous rehabilitation treatment and consequently slowing the cure.

Hereinafter, a tracking algorithm, which is an existing technology related to the present invention, will be described.

▣ Tracking Algorithm

Tracking an object in a video image tracks the movement of the object in each frame of the video image. The result of tracking is to analyze the motion of the object. Tracking technology includes 2D tracking that estimates motion parameters in three-dimensional space, 3D tracking that tracks the position and direction of objects in three-dimensional space, and high-level tracking that tracks object deformation. Tracking often uses special markers or constraints on color and shape. There are two main approaches to tracking moving objects: motion-based and model-based. The motion based method is a robust method for grouping visual movements over time. This method is fast but does not guarantee what the tracked area means. On the other hand, model-based methods can impart higher dimensional knowledge, but there are many difficulties in calculation because they need to cope with size, movement, direction, and deformation.

Both of these methods track an object using measurements provided by the geometric or local characteristics of the object being tracked. If we assume that there are some differences between the two images, this approach can yield very accurate results for estimating the object. However, if these assumptions are not maintained, you must rely on other methods to track the object. Edge-based and region-based methods generally require a lot of resources to compute, making it difficult to identify objects in real-time applications. Blob-based tracking algorithms do not use local image information such as edges or regions, but use color, motion, and approximate shapes to segment objects from the background. These are efficient and robust in computation.

To track an object, the system needs not only to locate the object, but also to find the same object in subsequent images. Finding the same object in a series of images can be difficult because of changes in lighting, the movement of the object, or the environment in which other objects enter or exit the image.

In this regard, we will look at the Template, Active Contour Model, Active Shape Model, and MeanShift method.

1. Template.

The types of templates used to track an object in a video image include 2D templates and deformable templates. The 2D template is robust and fast, but this method requires initialization and learning and limits the range of motion of the tracking object.

Although the shape of an object changes from frame to frame, the overall structure of the object does not generally change. A deformable template model can use the proper prototype to determine the overall structure of an object.

2.Active Contour Model

The Active Contour Model is also known as Snake, where the internal force smooths the unnaturalness of the contour under the influence of the internal force from the curve itself and the external force calculated from the image data, while the external force draws the contour to important image features. It is. Snake's external force can be defined using color feature points, the contour of the head can be obtained using the segmentation method, and the contour's boundary can be tracked in the image of each frame.

This Active Contour Model is widely used in many applications including Edge Detection, Shape Modeling, Segmentation, and Motion Tracking. However, in the case of an object with concave boundary, it is difficult to proceed, and in the case of an image having a complicated contour, it is difficult to search the exact shape. In addition, the Active Contour Model takes a long time to process in real-time video.

3.Active Shape Model

In order to track an unstructured object in a continuous video image, it is basically necessary to extract the shape and position of the object. Most two-dimensional deformable models use information from boundaries to represent objects. In tracking deformable models, the Active Shape Model is one of the best approaches in terms of accuracy and efficiency, provided that prior information about the object to be tracked is applicable. The Active Shape Model uses dictionary information of the shape of the object outline to find.

This model requires a training set, which requires the user to construct various shape models by marking the reference points on the outside of the object from the training images. The reference point should also be able to represent the outer features of the object. For example, the corner point of the eye may be a good reference point. This method is faster, but it is difficult to find the exact shape of an object in the case of an unclear image, like the Active Contour Model, because it needs to find the exact part of the object when looking for an object within a new image. There is this.

In addition, iterative calculation is necessary to match the contour of the object to the model every frame, and it is difficult to realize the real shape for using the Active Shape Model in a commercialized system because it is difficult to predict the movement information of the contour of the object of the next frame. There are disadvantages.

4. MeanShift

The MeanShift algorithm finds the center of the tracking target based on the color information while fixing the size of the result obtained in the previous frame. The tracking method using the MeanShift algorithm is simple to implement and has a relatively small computational complexity because it uses a single candidate. However, due to the convergence characteristic in local mode, tracking may fail in situations such as overlap between objects.

Accordingly, the present invention intends to develop a computer system that recognizes and recognizes a human motion using image information acquired through a camera, and displays a corresponding response through a monitor such as a TV.

In other words, the company intends to develop a computer system in which a computer and a user interact with a video input through a camera by avoiding cumbersome and complicated data input methods using a conventional touch input method such as a body capture of a motion capture sensor.

In addition, by incorporating interactive game elements into repetitive and long-term treatment activities of rehabilitation patients, the patient's interests and fun are stimulated, resulting in psychological and physical motivational effects on rehabilitation treatment. To improve the ability of perception.

The present invention has been made in view of the above, by introducing a non-contact game technique through video that can cause the interest and fun of the patient to enhance the treatment effect and to collect and analyze the data on the treatment status of each patient The purpose of the present invention is to provide an interactive rehabilitation support system using augmented reality technique and CAMSHIFT algorithm to enable more systematic rehabilitation treatment.

The real-time cognitive and behavioral disorder rehabilitation support system of the present invention for solving the above problems has an input unit, an image processing unit, and an output unit, using an augmented reality technique and a CAMSHIFT algorithm, and a square that appears on the screen for the cognitive disorder patient Visual Reaction function, Visual Differential Reaction function, and Visual Reaction and Move function to train the visual discrimination function to measure the patient's attention, reaction speed and accuracy by selecting the area and the same color as the rectangular area. It consists of a modular cognitive rehabilitation support system and a behavioral disorder rehabilitation support system consisting of a Visual Track and Target function and a Visual Spatial and Motor function module that measure the attention and hand movement of a patient for a behavioral disorder patient.

(Example)

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

1 (a) and (b) is a view for explaining the concept and the actual hardware configuration of the rehabilitation support system according to the present invention, as shown in the rehabilitation support system is the subject of rehabilitation treatment, such as the entire body or part of the patient The image processing unit includes a camera 1, which is an input unit focused on an image processing unit, an image processing unit 2 that includes a program and interactive game elements according to each type of disorder (cognitive impairment, motor impairment, etc.), and an image processing unit. And a display device 3 which is an output unit for displaying the result.

2 is a flowchart illustrating an operation of a rehabilitation support system according to the present invention. A computer which is an image processing unit by recognizing a motion of a moving object by analyzing an image acquired from the camera 1 as an input unit and recognizing a corresponding gesture. In (2), the correspondence by the interactive game made by the application program is shown on the screen of the display unit 3, which is an output unit such as a TV. These flows interact with each other and repeat in response to actions.

Figure 3 is the input unit 1, the processing unit 2 and the output unit 2 constituting the rehabilitation support system according to the present invention according to the type of patient (ie, cognitive impairment, motor impairment and perceptual impairment, etc.) The process of rehabilitation of the patient is shown step by step.

On the other hand, the algorithm used to implement the rehabilitation support system of the present invention used a Continuously Adaptive Mean Shift (CAMSHIFT) algorithm to process the color probability distribution dynamically changing from each video frame.

As described above, the disadvantage of the MeanShift algorithm is that it does not update the size of the search window and tends to converge to the Local Maximum.

This disadvantage has been improved by the following procedure in the CAMSHIFT algorithm used in the present invention.

For example, the size and position of the initial search window are specified, the MeanShift algorithm is executed, and the horizontal and vertical size and rotation angle of the search window are newly set with the second moment.

The fast computational power of the CAMSHIFT algorithm used in the present invention solves the drawback that Particle Filter takes a lot of time to find a probabilistic motion model to obtain a post-distribution that outlines an object while maintaining a number of assumptions.

4 (a) shows the result of executing the CAMSHIFT used in the present invention, and (b) shows the result of executing the MeanShift described above.

As shown in (b) of FIG. 4, MeanShift does not know the rotation angle of the object and does not change the size of the search window.

On the other hand, as shown in (a) of Figure 4 in the CAMSHIFT can be known not only the position of the object but also the rotation angle.

5 shows a CAMSHIFT algorithm. As can be seen from FIG. 5, the CAMSHIFT algorithm detects an object through rapid learning, continuously predicts and detects the position of the object in an input image, and then detects the center of the object. Find out.

As the result of the CAMSHIFT algorithm, the object to be tracked by the user appears on the image, and as shown in FIG. 6, the direction of the object, the X and Y coordinates, and the horizontal and vertical size of the object are determined using the color distribution of the object. .

The algorithm focuses on the part to be tracked using a color-based probabilistic model and keeps track of how that part changes in the entire image, thus minimizing prior knowledge when tracking a user's desired object in real time. Because it is suitable for use in HCI applications.

Comparing and analyzing the CAMSHIFT algorithm and other algorithms used in the present invention are as follows.

In the field of computer vision, Snake is used to track the contours or objects using the Active Shape Model. These methods have the disadvantage that it is difficult to find the exact shape of the object in the case of an image whose contour is not clear. It is very difficult to use in real-time processing where it takes a long time to calculate the tracking of an object in a constantly changing video and needs to process 30 frames per second.

In addition, it takes a long time to remove the noise from the input image or calculate the contour as well as the complexity of the algorithm, making it difficult to process in real time.

Moreover, templates that require initialization and learning have limitations on the movement of the tracking object.

On the other hand, color information may be efficient feature information in various applications for detecting a face or tracking a hand.

Thus, color information will be an efficient tool for identifying features if the color model is properly adjusted to different lighting environments.

The above-described MeanShift and CAMSHIFT algorithms, which use such color information, can perform real-time processing with relatively low computational complexity.

On the other hand, as shown in (a) and (b) of Figure 7, the MeanShift algorithm has a disadvantage that it does not update the search window for tracking the object.

For example, when the object moves toward the camera as shown in (a) or moves away as shown in (b), the size of the object shown on the screen changes, but the search window cannot be updated, so that the center of the object cannot be found.

However, as shown in (c) and (d) of FIG. 7, the CAMSHIFT algorithm used in the present invention can update the search window in real time to track the movement of irregular objects. You can keep track of the object you were tracking unless it overlaps 100%.

In addition, because the HSV model uses Hue, it is robust to light changes by ignoring too high or low brightness pixels.

As described above, in the present system, the cognitive rehabilitation system is designed using the CAMSHIFT algorithm which tracks the object using color information for fast processing and irregular object movement.

That is, the CAMSHIFT algorithm used in the system of the present invention uses the MeanShift algorithm to track the object in real time, and updates the size and rotation angle of the tracked object using the secondary moment.

In addition, the system of the present invention analyzes the image acquired from the camera using the CAMSHIFT algorithm to recognize the motion of the moving object, and recognizes the corresponding gesture to screen the screen of a monitor or TV for application interaction with the patient. Is shown through.

The cognitive and behavioral disorder rehabilitation support system implemented in the present invention for rehabilitation training using the CAMSHIFT algorithm as described above is shown in FIG. 8.

On the other hand, the system of the present invention must satisfy requirements such as the function of tracking the behavior of the patient, the function of managing the rehabilitation data, the function of managing the information of the patient, and the system's response to the patient's behavior.

In order to satisfy this requirement, as shown in Fig. 8, the cognitive and behavioral disorder rehabilitation support system 2 of the present invention transmits a signal S (t) on the behavior of the patient to the image camera 1; It is configured to display the system's response (A (t)) on a computer monitor or TV screen by analyzing the image using the CAMSHIFT algorithm.

In addition, the patient's information can be managed individually, the time the patient performs each task, and the rehabilitation data can be maintained to determine the degree of rehabilitation.

In addition, the present invention uses the CAMSHIFT algorithm, which is an evolution of the MeanShift algorithm using the color probability distribution, 1) selecting the size and position of the initial search window, 2) applying the MeanShift algorithm, and 3) calculating the moment. 4) calculate the position and size of the search window, and 5) repeat steps 2) and 3) until they converge.

Therefore, in the present invention, it is possible to determine the size and rotation angle of the tracking target by using the moment as well as tracking the rapid processing and the movement of irregular objects in real time.

In addition, the system of the present invention measures the rehabilitation of the patient by measuring the time the patient performs one task.

On the other hand, cognitive and behavioral disorder rehabilitation support system (2) of the present invention can be divided into two large cognitive and behavioral disorder rehabilitation support program.

For example, as shown in FIG. 8, the cognitive impairment program for the cognitive impairment program is to select a rectangular area appearing on the screen and the same color as the rectangular area so that the patient's attention, reaction speed, and accuracy are improved. It consists of Visual Reaction, Visual Differential Reaction, and Visual Reaction and Move functions that train visual discrimination.

In addition, as shown in FIG. 8, the behavioral disorder program for the behavioral disorder patient includes a Visual Track and Target function and a Visual Spatial and Motor function module for measuring the attention and hand movement of the patient.

Each program maintains the basic information of the patient and the rehabilitation data of the patient, and provides the function for outputting the rehabilitation data and the measured time indicating the degree of rehabilitation in the form of a graph.

Hereinafter, the Visual Reaction function, Visual Differential Reaction function, Visual Reaction and Move function, Visual Track and Target function, Visual Spatial and Motor function performance, experiment and evaluation of the cognitive and behavioral disorder rehabilitation support system according to the present invention Explain.

First, the Visual Reaction function, the Visual Differential Reaction function, and the Visual Reaction and Move function of the cognitive impairment rehabilitation support system will be described.

One. Visual Reaction

Awakening, the most basic aspect of cognitive function, refers to the state of overall response to stimuli from the surroundings. Concentration is a form of awakening and training to focus the patient's attention in one place.

9 is a flowchart illustrating a process of performing a Visual Reaction function by the Visual Reaction system. The Visual Reaction system is the lowest level of concentration to allow a patient to select various colored square points appearing at random times on a computer screen. It is a system for training.

The white square that appears in the center of the screen the first time you run the program is the initial measurement area for setting which interface the patient will use to run the program. Click the Settings button to set up the details for the patient.

On the other hand, setting the details of the patient is registered by the doctor or nurse, and the reference measurement frequency is the part of setting the number of experiments to the patient to produce the results. Once enrolled, a patient can be searched by name to read previously set data from the database.

As shown above, the database is searched to check how many times a particular patient has executed this program and shows the current number of executions, and the total number of measurements is shown by calculating how many times the patient has executed this program as a reference measurement. .

As soon as the setting is completed as above, you can enter training. The doctor or nurse conducting the training pays attention to the patient using the hand or other tool in the initial white square and then clicks the Run button to conduct the training.

10 is a screen shown by executing the Visual Reaction function as described above, the circle on the screen shows the result from the CAMSHIFT algorithm.

Afterwards, since the patient has executed this program using his hand, he clicks on the rectangular area shown at the bottom of the screen of FIG. 10 to proceed to the next process.

For example, when the patient selects the rectangular area, a screen as shown in FIG. 11 is displayed. As described above, when the patient selects the rectangular button, the CAMSHIFT algorithm determines whether the center of the tracking object has selected the rectangular area.

As described above, the method for determining whether the rectangular area is selected is divided into two methods.

First, the CAMSHIFT algorithm checks whether the perimeter of the object being tracked spans a rectangular area. The method proceeds in the following order.

① Use the CAMSHIFT and the degree of rotation to find the coordinates of the circumference.

X coordinate around circle = center of circle + radius * cosθ

Y coordinate around circle = center of circle + radius * sinθ

② Check that the coordinates are included in the rectangular area.

③ If it is included, it is judged as selected.

Secondly, we check when the CAMSHIFT area grows due to the sudden change of the screen so that the rectangular area enters the CAMSHIFT area. The method is as follows.

① Find the circle that fills the square area.

② Find the distance between the circle and the circle drawn by the width of CAMSHIFT.

③ If the distance obtained above is smaller than the sum of the radius of the circle filling the rectangular area and the width of the CAMSHIFT, that is, the radius, it is considered to be included.

An empty square area generated in the screen of FIG. 11 is an area generated after a random time passes after selecting a rectangular area at the bottom of the screen. The reaction until the patient keeps focusing on the screen and selects a rectangular area that may not come out when By measuring the time, the patient's attention and concentration can be improved.

In addition, to improve the patient's athletic performance, the patient should be trained at a distance of about 3 meters from the input camera, and the rectangular area created after selecting the area at the bottom of the screen should be upward from one third of the top of the screen. Randomly generated.

When two areas are created, the patient selects the area by recognizing the occurrence of the second rectangular area. 12 is a screen illustrating a situation after selecting a second rectangular area (button).

This Visual Reaction program shows the user's selection on the screen for about two seconds when the patient selects the second rectangular area.

When the second area is selected, it starts measuring the time from the second rectangular area after selecting the first area until the second area is selected.

The measured time is managed for each patient to see the development of the patient. When the reference number set by the doctor or nurse is reached, the patient shows the total time and average time taken for the reference number. It also shows a dialog box asking if you want to do this again so you can decide if you want to continue.

In order to view the change state of the patient, click the statistics button to set the search conditions, and then search for the patient's data (see FIG. 13), and the patient's rehabilitation training record can also be viewed graphically.

2. Visual Differential Reaction

FIG. 14 is a flowchart illustrating a process of performing a Visual Differential Reaction function by the Visual Differential Reaction system. The Visual Differential Reaction system is used to identify which rectangle is the same color as the bottom of the screen whenever a colored rectangle appears on the screen. It is a system for measuring patient's attention, reaction speed and accuracy.

In addition, since the basics before executing this program are the same as the above-mentioned Visual Reaction program, it abbreviate | omits description.

In this system, when the program starts, as shown in FIG. 15, one of the right square and the left square appears on the screen at random time, and the patient is waiting at the center of the screen, and when the square appears, Select the same colored square that appears and measure the response time from when the colored square appears to the left and right squares.

For example, in addition to measuring the response time during each training session, the patient selects a square of the same color as the color of the square displayed on the screen. Can be measured.

At this time, in order to remind the patient of success and failure, the sound of success and failure is heard and the patient's attention is focused.

3. Visual Reaction and Move

16 is a flowchart illustrating a process of performing the Visual Reaction and Move function by the Visual Reaction and Move system. When the patient moves his or her selected square to the square of the initial starting position, the system is ready to measure the patient's rehabilitation again. do.

If the patient does not bring the selected rectangle to the initial starting position, the system will continue to draw the selected rectangle at the position of the patient's hand, the object being tracked. In other words, the system is for improving the attention and response time of the patient.

As shown in FIG. 17, when the rectangle appears at an arbitrary position on the screen, the patient selects the rectangle displayed on the screen, and the system measures the response time of the patient.

Next, the Visual Track and Target function and the Visual Spatial and Motor function of the behavioral disorder rehabilitation system according to the present invention will be described.

On the other hand, Visual Track and Target, a system that is necessary for a series of actions in which several movement units and several muscles cooperatively coordinate contraction and relaxation to achieve desired movement in smooth motion, and to measure the attention and hand movement of the patient We implemented a Visual Spatial and Motor program that measures how long a patient can move a balloon in a thorn bush without breaking as the thorn bush moves.

One. Visual Track and Target

18 is a flowchart illustrating a process of performing a Visual Track and Target function by the Visual Track and Target system. When the Visual Track and Target system starts with a program for measuring the attention of the patient and the movement of the hand, the screen is displayed. There is a square on the screen. If you take the tracking tool there, you can see the path the patient will follow, click on the squares along the path to reach the last square, and the program will measure how quickly the patient follows the path. Will be recorded.

The Visual Track and Target system allowed the patient to test six paths. For the convenience of users, in order to easily distinguish paths, each path can be selected by placing an image button such as a path on the screen and a rectangular layout on the right side.

In addition, in order to know how fast the response time to the program is progressed according to the memory of the patient, a full path view and a partial path view are provided as menus for measurement. The basics before running this program are the same as the Visual Reaction program, so the explanation is omitted.

An example of this exercise is to walk 100 meters and then map the place you walked to, and to remember the objects, people, and events you saw while walking, and to mark the places where you witnessed them [Lawson et al. , 1989]. The Visual Track and Target system allows you to remember paths by following the squares one by one, which helps patients find that the most difficult problem in their daily lives is that they cannot acquire new knowledge due to memory impairment. I think it can be.

20 is a view illustrating a path and button arrangement of a screen that is displayed when each path is selected by the image button of FIG. 19.

The six paths set in FIG. 19 had only one path when the system was first created, but the path was further added because the patient could easily lose interest when performing only one path while implementing the system.

21 is a visual track and target system, if the correct operation is repeated while performing the programmed operation from the beginning to the end, the operation is learned and stored can be easily executed with less concentration than the first.

Thus, it is a screen showing the state showing the entire path, the state not shown, and the state displayed partial paths so that you can test what you have learned through repeated training of the patient.

By showing each path without showing all the paths as described above, the patient can concentrate on the rehabilitation program, and the doctor or nurse who checks the degree of rehabilitation of the patient can check the path to which the patient can proceed. As the memory of the patient about the path not given increases, it can be an opportunity to encourage the patient's motivation.

2. Visual Spatial and Motor

22 is a flowchart illustrating a process of performing a Visual Spatial and Motor function by the Visual Spatial and Motor system. The Visual Spatial and Motor system is a program for measuring the attention and hand movement of a patient. Measure how long you can move a balloon without bursting as the prickly bush moves.

In this Visual Spatial and Motor system, many patients have difficulty in concentrating on one task. Therefore, in order to be immersed in the task for a certain period of time, training is organized in the form of play to stimulate the curiosity and interest of the patients.

In the Visual Spatial and Motor system, as shown in FIG. 23, the doctor or the nurse may adjust the width and speed of the thorn bush around the balloon according to the degree of rehabilitation of the patient.

Starting with this Visual Spatial and Motor program, the patient will have a balloon in the center of the interface tool used by the patient, as shown in Figure 24, and a red triangle that is considered visible around the balloon.

The patient sees the balloon and the thorn bush go along for about three seconds to help him figure out what he is doing at the beginning of the program.

The Visual Spatial and Motor program maintains the patient's data by measuring the patient's rehabilitation status by measuring the time the patient has maintained without popping the balloon.

On the other hand, in the present invention, based on the existing augmented reality (Augmented Reality) based technology has been carried out to further improve the system, such as real-time tracking of the pattern that has been a problem, inaccuracy of camera parameter correction and further development of 3D content.

That is, FIG. 26 illustrates a process of defining and recognizing a pattern.

As shown in FIG. 26, an image input through a camera, which is an input device, selects an effective image through sampling, and digitalizes analog image data through quantization.

Thereafter, the digitized data is analyzed to analyze the position of the pattern and the shape of the pattern.

On the other hand, the image analysis process first performs a preprocessing process such as binarization, noise removal, thinning of the acquired image.

As described above, feature information is extracted from a preprocessed image and defined as a model. The construction of such a model is repeatedly performed through analysis and testing of various and rich image information, and classified into several models. Constructed as a knowledgeled model. Then, the input image is recognized as an object according to this model information.

Next, the experiment and evaluation of the cognitive and behavioral disorder rehabilitation support system according to the present invention will be described.

On the other hand, the present inventors and rehabilitation medicine professors and therapists applied to the actual patients, the request of rehabilitation medicine has been added to the function to manage the patient information and data and to give a response to the patients.

In addition, in order to measure the degree of rehabilitation of patients in the system, the measurement data is displayed in a graph because it is inconvenient to measure and manage the time trained by the patient and to show only a simple value.

In order to find out the difference between the system of the present invention and the existing PSS CogRehab program, a survey was conducted on the therapists and patients in comparison with the experience of the therapist utilizing the existing program PSS CoRehab, After applying the system of the present invention to the treatment of patients, 11 therapists were surveyed, and therapists measured the average time until the patients were mastered while using the system.

In addition, after applying the cognitive and behavioral disorder rehabilitation support system according to the present invention to a small number of patients, a questionnaire was conducted to examine the therapist's satisfaction.

In addition, the survey concluded that Psychological Software Services Inc.'s PSS CogReHab system and the system implemented in the present invention completed the patient's daily treatment time of 20 to 30 minutes and were satisfied with five parts such as convenience, interest, and attention. It was divided into 5 stages: very good (5 points), good (4 points), normal (3 points), bad (2 points), and very bad (1 point).

On the other hand, in the light of the therapist's own experience, the graph of the result of comparing this program with the previous program before the rehabilitation training with this program directly to the patient is shown in FIG. 26.

This program was badly evaluated because the therapist had to make room in that it occupies space, unlike the CogReHab program, which uses only a keyboard, because the patient performed rehabilitation training in front of the imaging camera. However, in implementing the program to the therapists, it was highly evaluated for convenience, interest, applicability, and attention.

After applying the program to the patient, the results of the therapist's evaluation on the convenience and attention items are shown in FIG. 27.

As shown in FIG. 27, the patient received a better evaluation of attention than a conventional keyboard program.

This can be seen that the existing keyboard program was boring, but the patient was better at paying attention to the program.

On the other hand, the graph of the survey results for the average time until the patient is mastered by the CogReHab program and the program implemented in the present invention is shown in Figure 28.

As shown in FIG. 28, the results of the survey on the average time until the patient is mastered indicate that the program implemented in the present invention is faster than the CogReHab program.

This can be seen that the rehabilitation training improves the patient's program proficiency by allowing the patient to sit directly in front of the imaging camera and visually see the patient's actions.

As described above, the present invention is a PC-based rehabilitation training system that accepts images with an image camera to solve the inconvenience of keyboard or mouse operation in the field of rehabilitation treatment and to implement a cognitive and behavioral disorder rehabilitation system that is not boring It was developed.

That is, the system of the present invention acquires an image from the image camera, recognizes the motion and gesture of the patient in the cognitive rehabilitation training program, responds accordingly, and gives the patient a response through the screen.

The system developed through the present invention is directly utilized for cognitive rehabilitation and motion rehabilitation treatment, and the user can replace the repetitive rehabilitation therapy with the treatment using the system using vision techniques, thereby improving the psychological and physical motivational effects on the rehabilitation therapy. You can reap.

In addition, the cognitive and behavioral disorder rehabilitation system developed in the present invention uses a CAMSHIFT algorithm to recognize the hand of the patient or the tool used by the patient in real time, the CAMSHIFT algorithm is irregular object movement, noise, interference, light It is well suited for use in HCI applications because it is robust to changes in and uses minimal prior knowledge when tracking objects of interest in real-time images.

In addition, as described above, the present invention provides four programs for developing attention, reaction time, hand control, and visual perception to enhance physical or psychological achievement for patients with cognitive or behavioral disorders. The developed cognitive rehabilitation program includes Visual Reaction, which measures attention and reaction time, Visual Differential Reaction, Visual Track and Target, which measures attention and hand movement, and Visual, which measures attention and visual perception motor skills. It consists of Spatial and Motor.

Moreover, the cognitive rehabilitation system developed in the present invention has been proved to be an interesting tool and a useful tool for the rehabilitation of cognitive and behavioral disorders compared to the conventional treatments by conducting a comparative survey with the existing CogRehab system.

On the other hand, the present invention is not limited to the above embodiments, of course, it can be carried out in a variety of modifications within the scope without departing from the object and background of the present invention.

For example, the presently implemented system of the present invention has many development possibilities by having a minimum configuration. Therefore, in the future, we apply the current system to patients with behavioral and cognitive problems, and the common problem of optimization algorithms of traditional nonlinear models such as CAMSHIFT can fall into Local Maximum or Local Minimum. Apply Simulated Annealing MeanShift Algorithm, which compares the amount of energy of a given state with the next candidate state to determine whether to adopt or not adopt a new state according to the amount of energy. It can be developed into a system that is robust to changes in the experimental environment both internally and externally.

As described above, according to the present invention, the problem of verbal comprehension or communication can be solved through the use of gestures or motion, and the efficiency of diagnosis and treatment can be greatly improved by increasing patient convenience, interest, and concentration of understanding, and perception. In addition to disability, it is possible to safely and cheaply diagnose and treat behavioral disorders and to have easy and extensive scalability according to contents of contents.

Claims (1)

In the real-time cognitive and behavioral disorder rehabilitation support system having an input unit, an image processing unit, and an output unit using augmented reality technique and CAMSHIFT algorithm, Visual Reaction function that trains visual discrimination function to measure patient's attention, reaction speed and accuracy by selecting rectangular area on the screen for the cognitive impairment, and selecting the same color as the rectangular area. Cognitive disability rehabilitation support system composed of Reaction function and Visual Reaction and Move function module, Real-time cognitive and behavioral rehabilitation support system comprising a behavioral rehabilitation support system consisting of Visual Track and Target function and Visual Spatial and Motor function module for measuring the attention and hand movement of the patient for the behavioral disorder patients .

Images