KR20220125850A - Method for omnidirectional robot of face-to-face interaction with human and medical service assistant robot capable of remote monitoring thereof - Google Patents

Abstract

The present invention relates to a method by which a medical service robot performs face-to-face interaction with the human and a medical service robot system capable of performing remote monitoring by using the same and, more specifically, to a medical service assist robot, which can provide medical knowledge and measured medical information by controlling the position and creating a conversation so that a patient and a display unit mounted on the medical service assist robot can face each other or perform optimal recognition. According to the present invention, a medical service robot system capable of performing remote monitoring comprises: an autonomous driving unit (100) which extracts RSSI data for the ID of each of beacons installed in a hospital room and a patient and controls autonomous driving to a requested location; a position control unit (200) which estimates body information of the patient in a 3D space to locate the patient and controls the position so that the located patient and the display unit (32) mounted on the medical service assist robot (2) can face each other or perform optimal recognition; a conversation generating unit (300) which evaluates an observation service for the patient through conversations; an audio-video (A-V) recognizing unit (400) which recognizes the motion of the patient and the sounds generated by the patient according to content provided by the medical service assist robot (2) by using an A-V module; and a medical information providing unit (500) which measures predetermined medical information from the patient and provides the information so that medical staff can monitor the same. Therefore, a medical environment can be improved.

Description

A method for performing face-to-face interaction with humans of a medical service robot and a medical service robot system capable of remote monitoring using the same

The present invention relates to a method of performing face-to-face interaction with a human of a medical service robot and a medical service robot system capable of remote monitoring using the same, and more particularly, a display unit mounted on the subject and the medical service robot provides face-to-face or optimal recognition. It relates to an assistive robot for medical services prepared to provide medical knowledge and measured medical information by controlling a posture to perform and generating a dialogue.

Recently, service robots that emphasize social roles in providing services while coexisting with humans are rapidly growing. In particular, social interaction between humans and robots has recently emerged as the most active research field among robot technologies since social robots that can perform social interactions in the form of interacting with humans and sharing daily communication and communion appeared.

One of the skills needed to interact with humans is for robots to gaze and track the person they interact with in real time. However, most robot bases are two-wheel drive. This is because there are several limiting situations to follow while observing and tracking the subject in real time due to movement restrictions in the axial direction. Therefore, the present invention uses a robot capable of omnidirectional movement to perform continuous face-to-face interaction with humans. would like to propose

Meanwhile, to perform human-robot face-to-face interaction, the user's joint position data is extracted in 3D space through the RGB-D sensor installed in the robot or external environment, and coordinates are converted into the robot coordinate system. After extracting the body pose information that the user is facing through the converted coordinates, the position control of the base of the medical service robot 2 is performed so that the robot can move to a position where social interaction with the user can be performed. Here, in order to continuously maintain user-robot interaction, we propose a tracking mechanism that can perform continuous face-to-face and distance maintenance even when the user's posture changes in real time.

Korean Patent Laid-Open No. 10-2002-0015621 (2002.02.28)

The present invention has been devised to solve the above problems, and an object of the present invention is to provide a medical robot for assisting health care work due to an increase in work stress and a decrease in efficiency due to insufficient number of medical workers.

The technical problems to be solved by the invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the description below. will be able

A medical service robot system capable of remote monitoring according to the present invention,

An autonomous driving unit 100 for controlling autonomous driving to a requested location by extracting RSSI data for each ID of a beacon mounted in a hospital room and subject;

Position of estimating the subject's body information in three-dimensional space to confirm the position, and controlling the posture so that the identified subject and the display unit 32 mounted on the medical service robot 2 can perform face-to-face or A-V recognition control unit 200;

a dialogue generating unit 300 for observing and assisting the patient's condition through dialogue;

A-V recognition unit 400 for recognizing a subject's performance and sound generated by the subject according to the contents provided by the medical service robot 2 using an Audio-Video module; and

It is characterized in that it consists of a; medical information providing unit 500 that measures preset medical information from the subject, monitors the medical staff, and provides the medical staff to check the information of the subject.

The posture control unit 200,

a location data extraction unit 210 for extracting body information of a subject using RGB-D sensor data;

a position data correction unit 220 that converts the extracted joint data into distance information based on the coordinate system of the camera unit 31 and corrects it using a moving average equation;

a movement command unit 230 for performing a movement command using the corrected position coordinates;

a distance maintaining unit 240 for maintaining the face-to-face distance or angle between the subject and the medical service robot 2 at a preset value; and

It is composed of a face-to-face tracking unit that continuously gazes at the subject's face; characterized in that it controls the posture so that the display unit 32 mounted on the subject and the robot can perform face-to-face recognition.

The A-V recognition unit 400,

a video recognition unit 410 for recognizing the action performed by the subject according to the content provision; and

and an audio recognition unit 420 for recognizing the sound generated by the subject.

The medical service robot (2),

a face-to-face interaction unit 20 for adjusting the posture so that the subject and the display unit 32 can perform face-to-face or A-V recognition by the posture control unit 200;

a monitor unit 30 including the display unit 32 and a camera unit 31 providing a content image; and

and an omnidirectional movement unit 50 that travels to a position requested by the autonomous driving unit 100 .

In addition, the method for performing face-to-face interaction with a human of the medical service robot according to the present invention,

A first step of extracting the subject's joint data using RGB-D sensor data;

a second step of converting the extracted joint data into distance information based on the coordinate system of the camera unit 31 and then correcting it using a Moving Average formula;

a third step of performing a movement command using the corrected position coordinates;

a fourth step of maintaining a face-to-face distance or angle between the subject and the medical service robot (2) at a preset value;

It is characterized in that it is performed using; a fifth step of continuously tracking the subject's face.

The moving average (MA) formula is characterized in that it is performed by the following [Equation 1].

[Equation 1]

(Where P _d is the human coordinate value, and n is the size of the number of measurements).

Executing the movement command in the third step is characterized in that the position is controlled using the following [Equation 4].

[Equation 4]

(Here, R is the distance from the center of the medical service robot 2 to the center of the wheel, r is the radius of the wheel, v _i is the linear velocity of each wheel, w _i is the angular velocity of the center of the medical service robot 2, θ is the global The angle the x _rob axis rotates counterclockwise about the coordinate system x axis).

In the fifth step, the subject's face tracking is,

It is characterized in that the tracking is performed using the upper body gaze angle estimation module as shown in Equation 5 below.

[Equation 5]

(Here, Ψ is the rotation angle of the medical service robot).

By means of solving the above problems, the present invention can provide an IoT-linked nursing medical robot that assists in healthcare work in a hospital environment.

In addition, the present invention can provide a medical service medical service robot (2) that can improve the medical environment and improve the satisfaction of patients and medical personnel.

In addition, the present invention is a robot with HRI (human-robot interaction) technology. Based on the questionnaire, personal information or observation tasks provided by the user can be recorded using a conversation manager, and depending on the situation, the robot can interact with the patient. It is possible to provide a medical service medical service robot 2 that can remotely connect medical staff in real time and provide necessary biosignal information at the same time.

In addition, according to the present invention, the robot can extract patient vital signals and medical environment (falls, bedsores, air pollution, temperature and humidity, etc.) information from the IoT device, measure the status of fluids, and can provide an automatic alarm for replacement time .

In addition, the present invention can preemptively respond to periodic patient visit check and call-bell work, and by using the fall risk measuring tool and the pressure sore measuring tool, the importance of nursing work can be adjusted for each high-risk patient, so that safe nursing can be performed.

1 is a block diagram of a medical service robot system capable of remote monitoring according to the present invention. 2 is a photograph showing a medical service robot 2 according to an embodiment of the present invention. 3 is a flowchart illustrating a method of performing face-to-face interaction with a human of the present invention, a medical service robot. 4 is a diagram showing an experimental configuration of face-to-face interaction according to an embodiment of the present invention. 5 is a view showing the wheel structure of the medical service robot according to an embodiment of the present invention. 6 is a diagram illustrating an interaction area classification according to a distance between a subject and a robot according to an embodiment of the present invention. 7 is a graph showing experimental results according to a human body posture angle according to an embodiment of the present invention. 8 is a control configuration diagram of a medical service robot system capable of remote monitoring according to an embodiment of the present invention. 9 is a block diagram illustrating an execution method of the schedule registration unit 510 performed by the medical information providing unit 500 according to an embodiment of the present invention. 10 is a flowchart illustrating an execution method of the schedule registration unit 510 performed by the medical information providing unit 500 according to an embodiment of the present invention. 11 is a flowchart illustrating an execution method of the schedule execution unit performed by the medical service robot 2 according to an embodiment of the present invention. 12 is a flowchart of a work command of the schedule registration unit 510 and the schedule execution unit according to an embodiment of the present invention. 13 is a detailed embodiment according to the sequence of work commands of the schedule registration unit 510 and the schedule execution unit of FIG. 12 according to an embodiment of the present invention. 14 is an embodiment in which the autonomous driving unit 100 recognizes a position of a beacon to control autonomous driving according to an embodiment of the present invention. 15 is a flowchart illustrating a method of performing autonomous driving in the autonomous driving unit 100 according to an embodiment of the present invention. 16 is a block diagram illustrating an execution method performed by the dialog generating unit 300 according to an embodiment of the present invention. 17 is a flowchart illustrating a method performed by the dialog generating unit 300 according to an embodiment of the present invention. 18 is an embodiment in which the dialogue generating unit 300 provides a speech message according to the distribution of the total score after the evaluation of the psychological state according to an embodiment of the present invention. 19 is a block diagram illustrating an execution method performed by the video recognition unit 410 according to an embodiment of the present invention. 20 is a flowchart illustrating a method performed by a spirometry unit according to an embodiment of the present invention. 21 is a flowchart illustrating a step of reproducing spirometer content according to an embodiment of the present invention. 22 is a flowchart illustrating a method performed by the upper extremity muscle strength measurement unit according to an embodiment of the present invention. 23 is a view showing the joint position at which the upper extremity muscle strength measurement unit recognizes the upper extremity direction in the upper extremity direction calculation step S431-3 of FIG. 22 . 24 is a view showing the joint position for the upper extremity muscle force measurement unit to recognize the coordinate system of the upper extremity joint in the local coordinate system generation step (S431-5) of FIG. 22 . 25 is a view showing the joint position at which the upper extremity muscle strength measurement unit recognizes the forward lifting, lifting, side lifting, and arm folding in the forward lifting determination step (S432-1) to the operation number determination step (S438) of FIG. . 26 is a flowchart illustrating a step of reproducing upper limb muscle strength content according to an embodiment of the present invention. 27 is a flowchart illustrating a method performed by a gait motion measurement unit according to an embodiment of the present invention. 28 is a flowchart illustrating a step of reproducing walking motion content according to an embodiment of the present invention. 29 is a block diagram illustrating an execution method performed by a cough recognition unit according to an embodiment of the present invention. 30 is a flowchart illustrating a method performed by the cough recognition unit according to an embodiment of the present invention.

Terms used in this specification will be briefly described, and the present invention will be described in detail.

The terms used in the present invention have been selected as currently widely used general terms as possible while considering the functions in the present invention, but these may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. Therefore, the term used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than the name of a simple term.

In the entire specification, when a part “includes” a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

Hereinafter, with reference to the accompanying drawings, the embodiments of the present invention will be described in detail so that those skilled in the art can easily carry out the embodiments of the present invention. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein.

Specific details including the problem to be solved for the present invention, the means for solving the problem, and the effect of the invention are included in the embodiments and drawings to be described below. Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings.

Hereinafter, the present invention will be described in more detail with reference to the accompanying drawings.

The present inventor's medical service robot system capable of remote monitoring, as shown in FIG. 1, includes an autonomous driving unit 100, a posture control unit 200, a dialogue generating unit 300, an A-V recognition unit 400, and a medical information providing unit ( 500) consists of In addition, the method for performing face-to-face interaction with a human of the medical service robot of the present invention is performed by the steps shown in FIG. 3 .

That is, according to the present invention, a medical service robot system capable of remote monitoring can be executed by a method of performing face-to-face interaction with a human of the medical service robot. The entire system of the present invention and the EMR-associated server may be configured as shown in FIG. 8 .

First, the method for performing face-to-face interaction with a human of the medical service robot is performed by the first step (S100) to the fifth step (S500), as shown in FIG. 3 .

First, the first step (S100) extracts the subject's joint data using RGB-D sensor data. More specifically, joint data is extracted from the OpenPose recognizer using the RGB-D sensor data of the configured system.

Next, in the second step (S200), the extracted joint data is converted into distance information based on the coordinate system of the camera unit 31, and then corrected using a moving average equation. More specifically, joint data extracted using depth information is converted into distance information based on the camera coordinate system. Then, in order to minimize the position error of the joint data required for estimating the user's body pose, the following [Equation 1] is corrected through the moving average (MA) equation.

(Where P _d is the human coordinate value, and n is the size of the number of measurements).

Next, in the third step ( S300 ), a movement command is performed using the corrected position coordinates. More specifically, the base of the medical service robot 2 performs a movement command from the corrected estimated position to the target position coordinates for maintaining social interaction.

The wheel structure of the medical service robot is shown in FIG. 5 . 5 shows the coordinate system and speed of the three-wheeled omnidirectional mobile robot. where R is the distance from the center of the medical service robot (2) to the center of the wheel, r is the radius of the wheel, V and ω are the linear and angular velocity of the center of the medical service robot (2), v _i is the linear velocity of each wheel, w _i is the angular velocity of the center of the medical service robot (2), Φ is the angle of the linear velocity V based on the x _rob axis, and θ is the angle at which the x _rob axis rotates counterclockwise with respect to the X axis of the global coordinate system.

At this time, position control is performed by kinematics as shown in [Equation 2] to [Equation 4] below. Since each wheel of the omnidirectional moving part 50 of the medical service robot is 30°, 150° and 270°, [Equation 2] can be expressed as follows.

The speed P' of the robot with respect to the coordinate system is expressed as [Equation 3] through the Jacobian matrix J.

By combining [Equation 2] and [Equation 3], the angular velocity of each wheel can be obtained as shown in [Equation 4], and through this, the desired linear velocity and angular velocity in the global coordinate system can be obtained.

(Here, R is the distance from the center of the medical service robot 2 to the center of the wheel, r is the radius of the wheel, v _i is the linear velocity of each wheel, w _i is the angular velocity of the center of the medical service robot 2, θ is the global It is the angle by which the x- _rob axis is rotated counterclockwise with respect to the x-axis of the coordinate system).

Next, in the fourth step (S400), the face-to-face distance or angle between the subject-medical service robot 2 is maintained at a preset value. More specifically, as shown in Figure 6, Hall. According to E (1966), it is defined as an area with friendliness at 0.46-1.2 m among social distances. Accordingly, the moving target coordinates are selected to maintain a certain distance in order to perform social interaction with the subject based on the user's body pose information.

Next, in the fifth step (S500), the subject's face is continuously tracked. More specifically, while the omnidirectional movement unit 50 of the medical service robot reaches the target position, the upper body gaze angle estimation module performs horizontal (Pan) and vertical (Tilt) functions to continuously track the face of the interaction target. It is classified as information, and when the omnidirectional moving part cannot maintain a certain distance or height with the user, it is designed to approach the user at a certain distance or height through the upper body link structure. Since the omnidirectional movement unit 50 of the medical service robot (2) moves in parallel with rotation, the horizontal gaze module corrects the difference in horizontal rotation angle between the medical service robot (2) and the human, and the vertical gaze module provides medical service Based on the robot's height information, it moves to a position for face-to-face gaze that maintains a certain distance from the subject's relative distance and height information from the medical service robot.

The upper body gaze angle estimation module is preferably estimated using the following [Equation 5].

(Here, Ψ is the rotation angle of the medical service robot).

The autonomous driving unit 100 controls autonomous driving to a requested location by extracting RSSI data for each ID of a beacon mounted in a hospital room and a subject.

More specifically, as shown in FIG. 15 , in order to control autonomous driving, the autonomous driving unit 100 first extracts RSSI data for each beacon ID (S110), performs Gaussian filtering (S120), and obtains distance information. It is performed through the step of converting (S130) and the step of estimating the three positions based on trilateration (S140).

The posture control unit 200 determines the position by estimating the subject's joints in a three-dimensional space, and controls the posture so that the identified subject and the display unit 32 mounted on the medical service robot 2 face each other. .

More specifically, the posture control unit 200 includes a position data extraction unit 210 , a position data correction unit 220 , a movement command unit 230 , a distance maintaining unit 240 , and a face-to-face tracking unit 250 . The posture is controlled so that the subject and the display unit 32 mounted on the robot face each other.

The position data extraction unit 210 extracts the joint data of the subject by using the RGB-D sensor data.

The position data correction unit 220 converts the extracted joint data into distance information based on the coordinate system of the camera unit 31, and then corrects it using a moving average equation. More specifically, the Moving Average (MA) equation in the position data correction unit is preferably performed by the following [Equation 1].

[Equation 1]

(Where P _d is the human coordinate value, and n is the size of the subset).

The movement command unit 230 performs a movement command using the corrected position coordinates. More specifically, it is preferable to control the position by using the following [Equation 4] to perform the movement command in the movement command unit.

[Equation 4]

(Here, R is the distance from the center of the medical service robot 2 to the center of the wheel, r is the radius of the wheel, v _i is the linear velocity of each wheel, w _i is the angular velocity of the center of the medical service robot 2, θ is the global It is the angle by which the x- _rob axis is rotated counterclockwise with respect to the x-axis of the coordinate system).

The distance maintaining unit 240 maintains the face-to-face distance or angle between the subject-medical service robot 2 as a preset value.

The face-to-face tracking unit 250 continuously stares at the subject's face. More specifically, it is preferable to track the face-to-face tracking unit using the upper body gaze angle estimation module as shown in Equation 5 below.

[Equation 5]

(Here, Ψ is the rotation angle of the medical service robot).

Next, the dialogue generating unit 300 assists in observing the patient's condition through dialogue. More specifically, the conversation generating unit 300 evaluates the psychological state by scoring the subject's responses according to preset questions such as the patient's consciousness, biorecord, intake/excretion, and psychological state, and then calculates the total score, and the It is preferable to provide speeches according to the distribution of the total scores.

As shown in FIG. 17 , the conversation generating unit 300 performs a question order determination step (S311), a question output step by voice or screen (S312), a patient response step (S313), an answer step (S314), QA list recording Step S315, determining whether there are remaining questions (S316), diagnosing a psychological conversation (S317), and outputting the diagnosis by voice or screen (S318) are performed.

In particular, as shown in FIG. 18 , the conversation generating unit 300 scores the response of the subject according to a preset question, evaluates the psychological state, calculates a total score, and provides a speech message according to the score distribution of the total score do. In one embodiment, if the total score is 0-2, ‘I think the patient is in a good mood. We will do our best to make it better.” If the above total score is 3-6, ‘The patient is under some hospitalization stress. Would you like to take a walk with me to refresh yourself? Or would you like me to play some relaxing music?’ If the above total score is 7-10, ‘The patient is under significant hospitalization stress. I will deliver it to the teacher in charge.'

Next, the A-V recognition unit 400 recognizes the subject's performance according to the content provided by the medical service robot 2 and the sound generated by the subject using the Audio-Video module.

More specifically, the A-V recognition unit 400 includes a video recognition unit 410 and an audio recognition unit 420 . The video recognition unit 410 recognizes the operation of the subject according to the provision of the content. The audio recognition unit 420 recognizes the sound generated by the subject.

In an embodiment according to the present invention, the video recognition unit 410 may include a spirometry unit, an upper extremity muscle strength measurement unit, and a gait motion measurement unit. The video recognition unit 410 is a structure that can pose a position where the upper body joint structure can be recognized as a video.

The spirometry unit measures the lung capacity of the subject according to the spirometry content provided by the medical service robot (2). 20, the spirometry unit color recognition (S411), content preparation signal transmission (S412), content start signal reception (S413), signal waiting (S414), yellow ball, red ball, two of the blue ball Marker position determination (S415), respiration recognition (S416), lung capacity transmission (S417), content termination (S418) and the result signal transmission (S419) is performed through the steps.

In addition, the spirometer content is provided by the display unit 32, and as shown in FIG. 21, module execution, repetition number designation (S421), Balls and Markers color detection (S422), content preparation topic transmission (S423), content Start topic reception (S424), content start (S425), Balls and Markers real-time location recognition (S426), Balls movement determination (S427), real-time lung capacity topic transmission (S428), final result output and content end (S429) and module It is performed through the end topic reception (S430) step.

The spirometer content algorithm converts the BGR image input through the RGB camera into HSV format to designate the color recognition range, recognizes two markers pre-attached above and below the spirometer in the HSV image to calculate the height of the spirometer, inside the spirometer Calculates the current height by recognizing the three colors of the ball, calculates the spirometry (cc) based on the distance between the two markers, that is, the height of the spirometer and the position of the three balls, outputs real-time results, and repeats spirometry for a specified number of times and outputs the final result and end the content.

The upper extremity strength measurement unit measures the upper extremity muscle strength of the subject according to the contents of the upper extremity strength provided by the medical service robot (2). As shown in Figure 22, the upper extremity muscle strength measurement unit joint recognition (S431), center joint recognition confirmation (S431-1), central position confirmation (S431-2), upper extremity direction calculation (S431-3), content preparation signal transmission ( S431-4), local coordinate system creation (S431-5), content start signal reception (S431-6), reception standby (S431-7), content execution (S431-8), current operation number = 0 (S431-9) Calculating the angle after executing the steps of lifting forward (S432-1), lifting up (S432-2), lifting sideways (S432-3), and bending the elbow (S432-4) respectively (S434), Determining whether the action matches and outputting the accuracy (S435), ending the content or achieving more than 80% accuracy (S436) When the action number exceeds 3 after performing the current action number +=1 (S437) (S438), the content end signal is transmitted (S439) is performed.

23 is a joint position at which the upper extremity muscle force measurement unit recognizes the direction of the upper extremity in the step of calculating the direction of the upper extremity (S431-3), and FIG. 24 is the upper extremity joint in the step (S431-5) of the local coordinate system of FIG. It is a diagram showing the joint position for recognizing the coordinate system of In addition, Fig. 25 shows the joint position at which the upper extremity muscle strength measurement unit recognizes the forward lifting, lifting, side lifting, and arm folding in the forward lifting determination step (S432-1) to the operation number determination step (S438) of FIG. It is a drawing.

In addition, the upper extremity muscle strength content is provided by the display unit 32, and as shown in FIG. 26, module execution (S441), joint recognition (S442), pelvis recognition (S443), center alignment (S444), upper body Coordinate system generation (S445), real-time direction calculation (S446), content preparation topic transmission (S447), content start topic reception (S448), content start (S449), current content operation index determination of front, up, side, bend (S450) ), direction, roll, and pitch angle calculation (S451), motion target angle-based accuracy percentage calculation (S452), real-time accuracy percentage topic transmission (S453), end motion recognition and execution of next motion content (S454) (repeat S449~S452) , the final result output and content end (S455) and module end topic reception (S456) are performed through the steps.

The upper extremity muscle strength content algorithm recognizes three-dimensional points of each joint of a human object recognized through a recognition model and a joint recognition program learned from the image input through the RGB camera and depth sensor, and the left and right shoulder and neck (center of the clavicle) Calculate the cross product of the joint vectors and calculate the average of the two cross products as the direction vector of the upper body. After comparing the Euler angle of the joint and the Euler angle of the exemplary motion, the motion accuracy is output in real time, and when the current motion is finished, it moves to the next motion and repeats motion measurement (current 4 motions: forward lift, up lift, side lift, arm fold) After performing all operations, output the final result and end the content.

The gait motion measurement unit recognizes the gait motion of the subject according to the gait motion content provided by the medical service robot (2). 27, joint recognition (S461), center joint recognition confirmation (S461-1), central location confirmation (S461-2), lower extremity direction calculation (S461-3) and content preparation signal transmission (S461-4) do. After the calculation of the lower limb direction (S461-3), the local coordinate system is generated (S461-5), the content start signal reception is determined (S461-6), and the reception waits (S461-7) or the content is executed (S461-8). When the content execution step (S461-8) starts, the walking motion determination is executed (S462), the number of steps is output (S463), and the content end determination is made (S464). When it is finished, the result signal is transmitted (S465).

However, the step of determining the walking motion (S462) is a step of comparing the position of the leg joint and the pelvic joint (S462-1) more specifically, both legs in front of the pelvis (S463-1) and one of the two legs in front of the pelvis (S464). -1), the position is determined by determining whether both legs are behind the pelvis (S465-1).

In addition, the gait motion content is provided by the display unit 32, and as shown in FIG. 28, module execution (S471), joint recognition (S472), pelvis recognition confirmation (S473), center alignment (S474), lower body Coordinate system generation (S475), real-time direction calculation (S476), content preparation topic transmission (S477), content start topic reception (S478), content start (S479), walking motion (front leg measurement and leg joint cross recognition) determination (S480) ), real-time stride length, steps, topic transmission (S481), final result output and content termination (S482), and module termination topic reception (S483).

The gait motion content algorithm recognizes three-dimensional points of each joint of a human object recognized through a recognition model and a joint recognition program learned from an image input through an RGB camera and a depth sensor, and the left and right pelvis and spine joint vectors After finding the external product, the average of the two external products is calculated as the direction vector of the lower body. Compare the directionality of the lower body with the direction in which the joints of the two legs are stretched, and project the vectors of the joints of the two legs on the vector of the lower body to determine which leg is currently ahead, then recognize the motion in real time to identify and record the intersection of the two legs and walk The number is measured, and the gait motion measurement is repeated for a specified number of times, outputting the final result and ending the content.

In addition, according to an embodiment of the present invention, the audio recognition unit 420 may include a cough recognition unit for checking whether or not coughing among the sounds generated by the subject. 30 , the cough recognition unit performs sound input (S491), voice waveform/frequency analysis (S492), cough? (S493), determining whether content is running (S494), and transmitting the recognition result (S495) is performed by

Next, the medical information providing unit 500 measures preset medical information from the subject, monitors the medical staff, and provides the medical information so that the medical staff can check the patient's information.

In an embodiment according to the present invention, the medical information providing unit 500 may include a schedule registration unit 510 . As shown in FIG. 9 , the schedule registration unit 510 may check patient information by interworking with a Public Health Information System (PHIS) to register a schedule according to the patient information, and the medical service robot ( 2) Request to be able to carry out this schedule.

10 is a flowchart illustrating an execution method of the schedule registration unit 510 performed by the medical information providing unit 500 according to an embodiment of the present invention. A PHIS schedule registration step (S511), a work command transmission step (S512), Job result confirmation step (S513), step of determining whether the corresponding job has been completed (S514), step of storing DB result (S515), step of outputting a notification of completion by voice or text (S516), determining whether all schedules have been completed It is performed by the step (S517).

11 is a flowchart illustrating an execution method of the schedule execution unit performed by the medical service robot 2 according to an embodiment of the present invention, in which the operation command reception step (S521), the robot movement step (S522), the medical service execution step ( S523), the operation result transmission step (S524), the step of determining whether all contents are finished (S525), and the step of determining whether all schedules are completed (S526) are performed.

12 is a flowchart of a work command of the schedule registration unit 510 and the schedule execution unit according to an embodiment of the present invention, and FIG. It is a specific example according to the order.

On the other hand, the present invention, a medical service robot system capable of remote monitoring is performed by the medical service robot (2). As shown in FIG. 2 , the medical service robot 2 is composed of a locking part 10 , a face-to-face interaction part 20 , a monitor part 30 , a handle part 40 , and a forward movement part 50 . can be

As shown in FIG. 2 , the locking part 10 is composed of an infusion holder 11 that can hold an infusion and a length adjustment part 12 that can adjust the height of the infusion hanger 11 . .

The face-to-face interaction unit 20 includes a display unit 32 and a camera unit 31 . More specifically, the display unit 32 outputs a content image including the spirometry content, upper extremity muscle strength content, gait motion content, and the like, and the camera unit 31 receives the content from the video recognition unit 410 . Measure so that the subject's performance can be recognized according to the provision.

The handle portion 40 is composed of a handle portion 40 for manually moving the medical service robot (2).

The omnidirectional movement unit 50 is provided with three-wheeled omnidirectional wheels to travel to a position requested by the autonomous driving unit 100 . Each wheel of the autonomous driving unit 100 is preferably provided at 30°, 150° and 270°.

Below, an experimental example in which the subject's pose accuracy and robot position and orientation accuracy were measured was performed.

The system for carrying out Experimental Examples 1 and 2 consisted of an omnidirectional robot platform equipped with Orbbec's Astra camera and Pan-Tilt module having an angle of view of 60H*49.5V*73D. The camera position (x ₁ , y ₁ , z ₁ , θ, Ψ) in the indoor environment is based on the map coordinate system of the medical service robot (0, 0, 1.88m, -1.57, -0.35rad), and the subject (x _h , y _h , z _h ) is located at (2, 0., Height)m and faces the front and ±45°.

In addition, an upper body gaze angle estimation module that can be controlled at 180° horizontal and vertical angles to continuously gaze at the user at the head position (0, 0, 1.26 m) of the mobile robot was mounted.

Experimental Example 1: Subject Pose Accuracy

Experimental Example 1 according to the present invention confirmed the pose accuracy of the interaction subject by analyzing the error of the angle extracted through the joint coordinate value when the subject took a pose at a predetermined angle. This Experimental Example 1 extracts 1,500 pieces of data (5 people * 2 poses * 3 locations * 50 pieces per coordinate).

The measured data for each pose is shown in FIG. 7 .

Experimental Example 2: Robot Position and Direction Accuracy

Next, Experimental Example 2 for verifying the position and direction accuracy of the medical service robot aims to move to an agile and accurate position with respect to the human posture changing in real time. At this time, the medical service robot 2 is positioned at a distance of 0.8 m from the human to gaze at the user. Here, in Experimental Example 2, 3,000 pieces of data (3 poses * 2 locations * 100 pieces per coordinate * 5 times) are extracted. The success or failure of each experiment is to be located within ±5° of the angular error, and in the case of Experimental Example 2, it aims to be located within ±0.1m under the distance maintenance condition.

When performing face-to-face interaction, the standard deviation of the angle at which the medical service robot 2 looks at humans was located within 5°, but it was confirmed that the angle error was located relatively low with a success rate of 66.7%. In the case of a large error, when the distance between the human and the robot increases, it is determined that the medical service robot 2 moves and the slip occurs due to the frictional force of the wheels.

By means of solving the above problems, the present invention can provide an IoT-linked nursing medical robot that assists in healthcare work in a hospital environment.

In addition, the present invention can provide a medical service medical service robot (2) that can improve the medical environment and improve the satisfaction of patients and medical personnel.

In addition, the present invention is a robot with HRI (Human-Robot Interaction) technology, which can provide information based on the questionnaire and record related contents using a dialog manager, and the robot can communicate with patients and medical staff in real time depending on the situation. It is possible to provide a medical service medical service robot 2 that can connect remotely and simultaneously provide necessary biosignal information.

In addition, the present invention can extract the patient vital signal and medical environment (air pollution, temperature and humidity) information using the IoT device, and can provide an automatic alarm for measuring the state of the infusion, Yurinbag and replacement time.

In addition, the present invention can preemptively respond to periodic patient visit check and call-bell work, and can control the importance of nursing work for each high-risk patient by using a fall risk measuring tool and a pressure sore measuring tool, so that safe nursing can be performed.

As such, those skilled in the art to which the present invention pertains will understand that the above-described technical configuration of the present invention may be implemented in other specific forms without changing the technical spirit or essential characteristics of the present invention.

Therefore, the embodiments described above are to be understood as illustrative and not restrictive in all respects, and the scope of the present invention is indicated by the following claims rather than the above detailed description, and the meaning and scope of the claims and their All changes or modifications derived from the concept of equivalents should be construed as being included in the scope of the present invention.

1. Medical service robot system 100. Autonomous Driving Department 200. Posture control unit 210. Location data extraction unit 220. Location data correction unit 230. Move command unit 240. Distance Maintenance Department 300. Conversation Generator 400. A-V recognition unit 410. Video Recognition Unit 420. Audio Recognition Unit 500. Department of Medical Information 510. Schedule Register 2. Medical service robot 10. catch 11. Sap holder 12. Length adjustment part 20. Face-to-face interaction 30. Monitor unit 31. Camera part 32. Display unit 40. Handle 50. Forward moving part S100. The first step of extracting the subject's joint data using RGB-D sensor data S200. The second step of converting the extracted joint data into distance information based on the coordinate system of the camera unit 31 and then correcting it using a Moving Average formula S300. A third step of performing a movement command using the corrected position coordinates S400. The fourth step of maintaining the face-to-face distance or angle between the subject and the medical service robot (2) at a preset value S500. Step 5 to continuously track the subject's face

Claims (11)

Autonomous driving unit 100 for controlling autonomous driving to the requested location by extracting RSSI data for each ID of beacon mounted on the patient's room and subject; Confirming the location by estimating the subject's body information in three-dimensional space, and Posture control unit 200 for controlling the posture so that the display unit 32 mounted on the subject and the medical service robot 2 can perform face-to-face or A-V recognition; Conversation generating unit for observing and assisting the patient's condition through conversation (300); A-V recognition unit 400 for recognizing a subject's performance and sound generated by the subject according to the content provided by the medical service robot (2) using an Audio-Video module; and a medical information providing unit 500 that measures preset medical information from the subject, monitors the medical staff, and provides the medical information so that the medical staff can check the subject's information. According to claim 1, The posture control unit 200, Position data extraction unit 210 for extracting the subject's body information using RGB-D sensor data; The extracted joint data to the camera unit 31 coordinate system After conversion into distance information as a reference, the position data correction unit 220 that corrects using a moving average formula; The movement command unit 230 that performs a movement command using the corrected position coordinates; Subject- a distance maintaining unit 240 for maintaining a face-to-face distance or angle between the medical service robots 2 at a preset value; and a face-to-face tracking unit that continuously gazes at the subject's face; a remote monitoring capable medical service comprising: a display unit 32 mounted on the subject and the robot to control the posture to perform face-to-face recognition robot system. According to claim 1, The dialogue generator 300, Scores the response of the subject according to the preset question, evaluates the state of the subject, and then calculates a total score, and generates an utterance according to the score distribution of the total score. Medical service robot system capable of remote monitoring, characterized in that it provides. According to claim 1, The A-V recognition unit 400, A video recognition unit 410 for recognizing the operation of the subject according to the provision of the content; and an audio recognition unit 420 for recognizing the sound generated by the subject. According to claim 4, The video recognition unit (410), Spirometry unit for measuring the lung capacity of the subject according to the spirometry content provided by the medical service robot (2); Upper limb provided by the medical service robot (2) An upper extremity muscle strength measuring unit for measuring the upper extremity muscle strength of the subject according to the muscular strength content; A gait motion measuring unit recognizing the gait motion of the subject according to the gait motion content provided by the medical service robot (2); characterized by comprising: Medical service robot system. The medical service robot system of claim 4 , wherein the audio recognition unit 420 includes a cough recognition unit that checks whether the subject coughs among the sounds generated by the subject. According to claim 1, The medical service robot (2), Face-to-face interaction unit ( 20); a monitor unit 30 including the display unit 32 and a camera unit 31 for providing a content image; And Medical service robot system comprising a; omnidirectional movement unit (50) that travels to the position requested by the autonomous driving unit (100). According to claim 5, The spirometer content, The height of the spirometer is calculated by recognizing two marks previously attached to the top and bottom of the spirometer in the image, and the current height is calculated by recognizing the colors of three balls inside the spirometer, A medical service robot system, characterized in that it calculates the spirometry (cc) based on the height of the spirometer and the positions of the three balls. The upper body according to claim 5, wherein the upper extremity muscle force content recognizes three-dimensional points of each joint of the object, holds the direction vector of the upper body as the z-axis, and calculates the Euler angle and direction of each joint by substituting a rotation matrix, and then the upper body A medical service robot system, characterized in that the motion accuracy is determined by comparing the Euler angle of the joint with the Euler angle of the exemplary motion. According to claim 5, The gait motion content, Recognizes three-dimensional points of each joint of the object, holding the direction vector of the lower body as the z-axis and substituting a rotation matrix to determine the directionality of each joint of the lower body and the distance between the two ankles After calculation, the direction of the joints of the lower body and the direction in which the joints of the two legs are stretched are compared, and the vector of the joints of the two legs is projected on the vector of the lower body to determine which leg is in front, and then the motion is recognized in real time to determine the movement of the two legs. Medical service robot system, characterized in that it identifies the intersection point. A first step of extracting the subject's joint data using RGB-D sensor data; After converting the extracted joint data into distance information based on the coordinate system of the camera unit 31, using a Moving Average formula A second step of correcting; A third step of performing a movement command using the corrected position coordinates; A fourth step of maintaining the face-to-face distance or angle between the subject-medical service robot 2 at a preset value; The subject's face A method of performing face-to-face interaction with a human of a medical service robot, characterized in that it is performed using a fifth step of continuously tracking

Images