KR20230154362A - System and method for providing drug transport service using robot based on xtended reality - Google Patents

Abstract

A system and method for providing medicine transport services using an extended reality-based robot is disclosed. According to one aspect of the present invention, a medicine transport service providing system using an extended reality-based robot includes at least one scenario regarding the order of transporting medicine and arrangement position coordinate values, and vision information and movement information of a robot performing each scenario. A server that learns a motion model for It includes a user system that displays and transmits control commands according to the user's voice and movement to the server, where the server acquires vision information or movement information of the robot according to the control command based on the operation model and sends a movement control signal. It can generate and transmit motion control signals to the robot system.

Description

System and method for providing pharmaceutical transport services using extended reality-based robots {SYSTEM AND METHOD FOR PROVIDING DRUG TRANSPORT SERVICE USING ROBOT BASED ON XTENDED REALITY}

The present invention relates to a system and method for providing a medicine transport service using an extended reality-based robot. More specifically, the present invention relates to a system and method for providing medicine transport service using a robot in a medical institution, so that the user can control and monitor the robot using extended reality technology when transporting medicine using a robot in a medical institution. It relates to a system and method for providing medicine transport services using an extended reality-based robot.

Narcotic drugs such as morphine, which are used as painkillers in hospitals, are prevented from being stolen by using locking devices such as locks, and are managed so that they can only be brought in and out by a specific person in possession of an identity verification tool such as an access card. When the amount exported was falsely stated or more medicines than needed were exported, there was a problem in that it was difficult to accurately determine the fact of such illegal export and the quantity.

In order to solve this problem, there are cases where a separate manager is assigned to monitor the import and export of medicines. However, in this case, it is difficult to determine the quantity of medicines in real time, so not only is the expiration date and inventory management of medicines not done smoothly, but it is also necessary. Even in cases where medicines other than those prescribed by the law were taken out or more than the amount used was taken out, it was difficult for the exporter to immediately recognize this, so there was a risk that a medical accident could occur due to misuse of the exported medicines.

Therefore, there is a need for a method that can automatically manage the handling process, including the distribution process, of pharmaceuticals and at the same time check and manage the handling process at any time or in real time.

The background technology of the present invention is disclosed in 'Drug management system and drug management method thereof' in Korean Patent Publication No. 10-2014-0011726 (published on January 29, 2014).

The present invention was brought out to improve the above-mentioned problems, and the purpose of the present invention is to provide an extended reality-based system that allows users to control and monitor the robot using extended reality technology when transporting medicines using robots in medical institutions. To provide a system and method for providing pharmaceutical transport services using robots.

The problem to be solved by the present invention is not limited to the problem(s) mentioned above, and other problem(s) not mentioned will be clearly understood by those skilled in the art from the description below.

According to one aspect of the present invention, a medicine transport service providing system using an extended reality-based robot includes at least one scenario regarding the order of transporting medicine and arrangement position coordinate values, and vision information and movement information of a robot performing each scenario. A server that learns a motion model for It includes a user system that displays an image and transmits a control command according to the user's voice and movement to the server, wherein the server provides vision information or movement information of the robot according to the control command based on the operation model. By obtaining the motion control signal, the motion control signal can be transmitted to the robot system.

In the present invention, the robot system includes a communication circuit, a photographing device that operates according to the movement of the extended reality display device to capture images, and a robot that carries the medicine to the corresponding placement position coordinate value according to the operation control signal. , and transmitting the image captured through the imaging device to the server through the communication circuit, and transmitting an operation control signal from the server to the robot or imaging device, so that the robot or imaging device responds to the operation control signal. It may include a control device that operates accordingly.

In the present invention, the user system includes a communication circuit, an extended reality display device worn on the user's head and displaying images captured through the photographing device in extended reality, a voice recognition device that recognizes the user's voice, and A haptic sensor that recognizes the user's tactile information, and the user's intention for the movement of the robot from the voice data recognized by the voice recognition device or the tactile information recognized through the haptic sensor, and according to the identified intention It may include a control device that transmits control commands for controlling the robot to the server and outputs images received from the robot system through the extended reality display device.

In the present invention, the server receives vision information or movement information of the robot performed according to the motion control signal from the robot system, and if the received vision information or movement information of the robot is unlearned information, the server receives the vision information or movement information of the robot performed according to the motion control signal. The motion model can be updated by learning the vision information or movement information of one robot.

According to another aspect of the present invention, a method of providing a medicine transport service using an extended reality-based robot includes a user system displaying images captured through an imaging device of the robot system through an extended reality display device worn on the user's head. When the user's voice data or motion information is obtained in this state, transmitting a control command including the voice data or motion information to a server, the server, a robot according to the control command based on a previously learned motion model It includes obtaining the coordinates of the components to generate a motion control signal, transmitting the generated motion control signal to a robot system, and performing an operation of the robot system transporting medicine according to the motion control signal. .

The present invention further includes the step of the server learning at least one scenario for the order of transporting medicines and arrangement position coordinates, and a motion model for vision information and movement information of a robot performing each scenario, After performing the operation of transporting the medicine, the server receives vision information and movement information performed by the robot according to the transport of the medicine from the robot system, and the received vision information and movement information of the robot are not learned. If the information is not available, the step of updating the motion model by learning the received vision information and movement information of the robot may be further included.

In addition to this, other methods for implementing the present invention, other systems, and computer programs for executing the methods may be further provided.

The system and method for providing a medicine transport service using an extended reality-based robot according to an embodiment of the present invention allows the user to control and monitor the robot using extended reality technology when transporting medicine using a robot in a medical institution. .

Meanwhile, the effects of the present invention are not limited to the effects mentioned above, and various effects may be included within the range apparent to those skilled in the art from the contents described below.

Figure 1 is a diagram for explaining a system for providing medicine transport services using an extended reality-based robot according to an embodiment of the present invention.
Figure 2 is a block diagram schematically showing the configuration of a user system according to an embodiment of the present invention.
Figure 3 is an example diagram illustrating a screen displayed through an extended reality display device according to an embodiment of the present invention.
Figure 4 is a block diagram schematically showing the configuration of a robot system according to an embodiment of the present invention.
Figure 5 is a diagram for explaining a method of providing a medicine transport service using an extended reality-based robot according to an embodiment of the present invention.

Hereinafter, a system and method for providing a medicine transport service using an extended reality-based robot according to an embodiment of the present invention will be described with reference to the attached drawings. In this process, the thickness of lines or sizes of components shown in the drawing may be exaggerated for clarity and convenience of explanation.

Additionally, implementations described herein may be implemented as, for example, a method or process, device, software program, data stream, or signal. Although discussed only in the context of a single form of implementation (eg, only as a method), implementations of the features discussed may also be implemented in other forms (eg, devices or programs). The device may be implemented with appropriate hardware, software, firmware, etc. The method may be implemented in a device such as a processor, which generally refers to a processing device that includes a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices such as computers, cell phones, portable/personal digital assistants (“PDAs”) and other devices that facilitate communication of information between end-users.

Additionally, the terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof. Terms such as first, second, etc. may be used to describe various components, but the components should not be limited by these terms. The above terms are used only for the purpose of distinguishing one component from another.

Throughout the specification, “extended reality (XR)” refers to virtual reality (VR), augmented reality (AR), and mixed reality (MR). VR technology provides objects and backgrounds in the real world only as CG images, AR technology provides virtual CG images on top of images of real objects, and MR technology provides computer technology that mixes and combines virtual objects in the real world. It is a graphic technology. MR technology is similar to AR technology in that it shows real objects and virtual objects together. However, in AR technology, virtual objects are used to complement real objects, whereas in MR technology, virtual objects and real objects are used equally.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, identical or corresponding components are assigned the same drawing numbers and duplicate descriptions thereof are omitted. I decided to do it.

FIG. 1 is a diagram illustrating a system for providing medicine transport services using an extended reality-based robot according to an embodiment of the present invention, and FIG. 2 is a block diagram schematically showing the configuration of a user system according to an embodiment of the present invention. , FIG. 3 is an example diagram for explaining a screen displayed through an extended reality display device according to an embodiment of the present invention, and FIG. 4 is a block diagram schematically showing the configuration of a robot system according to an embodiment of the present invention. .

Referring to FIG. 1, the extended reality-based medicine transport service providing system using a robot according to an embodiment of the present invention includes a server 100, a user system 200, and a robot system 300.

The server 100 may learn at least one scenario for the order of transporting medicines and the coordinate values of the placement positions, and a motion model for vision information and movement information of the robot 330 performing each scenario. Here, the motion model may include movement information such as grasping, pushing, and moving the medicine, and the motion information may be related to the components constituting the robot 330 (e.g., driving unit, body, link, end effector, etc.). It may include rotational force, rotation range, driving range, gripping force, adsorption force, etc. At this time, the server 100 can learn a motion model composed of an artificial neural network using data (expert data or big data) in which an expert performs a given motion.

The server 100 receives data (vision, movement information, more than 100 for each scenario) on the expert's action to transport medicine in an extended reality environment, and allows the actual robot 330 to perform the action to transport medicine. A motion model (artificial neural network model) including vision information (CNN), motion information (NN), etc. can be trained and the learned motion model can be used as an input to the robot 330.

The server 100 obtains movement information that repeats the medicine transport operation in an extended reality environment, and infers the arm movement and both arm movement movements based on the end hand movements of the user equipped with the extended reality display device 220, so that the user It is possible to determine the movement intention and learn a motion model that causes the robot 330 to operate according to the determined intention. At this time, the server 100 can learn the motion model using a model-based inverse reinforcement learning algorithm.

In order to perform tasks (grasping, pushing, etc.) to transport medicines using the force/tactile information detected through the haptic sensor 240 of the user system 200, it must be used together with the kinematic information of the robot 330, and the medicines For stable interaction, the balance of forces must be maintained. Accordingly, the server 100 can utilize the balance of power relationship in the policy and objective function model using the behavior cloning/inverse reinforcement learning algorithm.

The server 100 receives vision information and movement information performed by the robot 330 according to the transport of medicine from the robot system 300, and when the received vision information and movement information of the robot 330 are unlearned information. The motion model can be updated by learning the received vision information and movement information of the robot 330.

When receiving a control command from the user system 200, the server 100 acquires vision information and movement information of the robot 330 according to the control command based on the motion model, generates a motion control signal, and sends the motion control signal to the robot. It can be transmitted to system 300. Here, the motion control signal may include vision information and movement information of the robot 330.

Since the control command includes the user's vision information and movement information, and the vision information and movement information include coordinate values, the server 100 converts the user's vision information and movement information into the vision information and movement information of the robot 330. It can also be converted and transmitted to the robot system 300.

This server 100 may refer to a device that trains an artificial neural network using a machine learning algorithm or uses a learned artificial neural network.

Server 100 may be a computing device including at least one processor. Server 100 may also mean a cloud server. In one embodiment, the server 100 may be used interchangeably with a platform.

The user system 200 displays the image captured by the photographing device 320 through the extended reality display device 220 worn on the user's head, and transmits control commands according to the user's voice and movement to the server 100. You can.

The user system 200 can receive images captured through the imaging device 320 of the robot system 300 as VR/AR/MR images.

The user system 200 can transmit control commands for manipulating the movement/motion/movement, camera angle, etc. of the robot system 300 to the robot system 300 through the server 100, and the robot system 300 Movement/motion/movement is controlled according to the operation control signal corresponding to the control command received from the user system 200, and extended reality images can be transmitted in real time.

As shown in FIG. 2, this user system 200 may include a communication circuit 210, an extended reality display device 220, a voice recognition device 230, a haptic sensor 240, and a control device 250. there is.

The communication circuit 210 can provide a communication interface necessary to provide transmission and reception signals between the user system 200 and the server 100 in the form of packet data in conjunction with a communication network. In particular, the communication circuit 210 may receive images captured in the robot system 300 from the server 100 and transmit control commands for controlling the robot 330 to the server 100. . Additionally, the communication circuit 210 may be a device that includes hardware and software necessary to transmit and receive signals such as control signals or data signals through wired or wireless connections with other network devices. Additionally, the communication circuit 210 may be implemented in various forms such as a short-distance communication module, wireless communication module, mobile communication module, and wired communication module.

The extended reality display device 220 is worn on the user's head and can display images captured through the imaging device 320 of the robot system 300 in extended reality.

The extended reality display device 220 may be implemented in the form of a head mounted display (HMD) at a location corresponding to both eyes of the user. Additionally, the extended reality display device 220 may be implemented in the form of a virtual reality headset, HoloLens, AR/MR glasses, etc.

The extended reality display device 220 is equipped with at least one direction/gyro/angle/acceleration sensor to sense the direction/angle of the user's head, and transmits the sensed results to the robot system 300 through the control device 250. You can. Then, the imaging device 320 of the robot system 300 may operate in a direction/angle corresponding to the direction/angle of the extended reality display device 220. The extended reality display device 220 can display the image transmitted from the robot system 300 as an extended reality image.

Accordingly, the user can carry out the operation of transporting medicine while checking the image information captured by the imaging device 320 of the robot system 300 in real time through the extended reality display device 220.

The voice recognition device 230 can recognize the user's voice and transmit it to the control device 250.

The voice recognition device 230 may be detachably provided on one side of the extended reality display device 220 or may be implemented in the form of a microphone integrally provided in the extended reality display device 220.

The haptic sensor 240 can recognize the user's force/tactile sensation and transmit the recognized tactile information to the control device 250.

The control device 250 determines the user's intention regarding the movement of the robot 330 from voice data recognized by the voice recognition device 230 or tactile information recognized through the haptic sensor 240, and performs the operation according to the identified intention. Control commands for controlling the robot 330 can be generated and transmitted to the server 100.

Additionally, the control device 250 may output the image received from the robot system 300 through the extended reality display device 220. That is, the control device 250 can render the image received from the robot system 300 into a VR/AR/MR image and output it through the extended reality display device 220. At this time, the control device 250 can configure a plurality of virtual screens on the extended reality display device 220 to control and monitor the process of the robot 330 transporting medicine.

For example, the control device 250 can display five virtual screens on the extended reality display device 220 as shown in FIG. 3. When the user looks straight ahead, the front screen is displayed, when the user turns his head to the left, the left screen is displayed, when the user turns his head to the right, the right screen is displayed, and when the user looks up, the top screen is displayed. It can be displayed, and the bottom screen can be displayed when the user looks down.

In this way, the control device 250 can link the environment (visual information) seen by the robot 330 with the extended reality environment used by the user.

In addition, the control device 250 acquires the end hand movements of the user equipped with the extended reality display device 220 through the haptic sensor 240, and performs arm movements and both arm movement actions based on the acquired hand movements of the user. Inferring, the user's intention can be determined based on the inferred arm movement and both arm movements, and a control command according to the identified user's intention can be generated and transmitted to the server 100.

The robot system 300 is equipped with an imaging device 320 and can perform an operation of transporting medicines according to an operation control signal from the server 100.

This robot system 300 may include a communication circuit 310, an imaging device 320, a robot 330, and a control device 340, as shown in FIG. 4.

The communication circuit 310 can provide a communication interface necessary to provide transmission and reception signals between the robot system 300 and the server 100 in the form of packet data in conjunction with a communication network. In particular, the communication circuit 310 may receive an operation control signal from the server 100 and transmit an image captured through the photographing device 320 to the server 100. Additionally, the communication circuit 310 may be a device that includes hardware and software necessary to transmit and receive signals such as control signals or data signals through wired or wireless connections with other network devices. Additionally, the communication circuit 310 may be implemented in various forms such as a short-distance communication module, wireless communication module, mobile communication module, and wired communication module.

The photographing device 320 can capture images by operating according to the movement of the user (i.e., the extended reality display device 220). In other words, the imaging device 320 can link the vision information seen by the robot 330 with the extended reality environment used by the user. This imaging device 320 may be implemented as, for example, a PTZ camera.

The robot 330 may perform an operation of transporting medicines to the corresponding placement position coordinates according to an operation control signal from the server 100.

The robot 330 is equipped with a driving unit (not shown) including an actuator or motor and can perform various physical movements, such as moving the joints of the robot 330. In addition, the movable robot 330 includes wheels, brakes, propellers, etc. in the driving part, and can travel on the ground or fly in the air through the driving part.

The robot 330 may be equipped with a smart sensor (not shown). The smart sensor is mounted on the arm of the robot 330 and can link the tactile information of the robot 330 with the user's tactile information.

Meanwhile, although the embodiment of the present invention is described as a robot 330, the robot 330 may be a robot arm. Since the robot 330 and the robot arm have the same configuration as the conventional one, their description will be omitted.

The control device 340 transmits the image captured through the imaging device 320 to the server 100 through the communication circuit 310, and transmits the operation control signal from the server 100 to the robot 330 and the imaging device ( 320), the robot 330 and the imaging device 320 can be controlled to operate according to the operation control signal.

Figure 5 is a diagram for explaining a method of providing a medicine transport service using an extended reality-based robot according to an embodiment of the present invention.

Referring to FIG. 5, when the user's voice data or motion information is acquired (S510), the user system 200 transmits a control command including the voice data or motion information to the server 100 (S520).

When step S520 is performed, the server 100 acquires the coordinates of robot components according to the control command based on the motion model, generates a motion control signal (S530), and sends the generated motion control signal to the robot system 300. Transmit (S540). Here, the control command includes the user's vision information and movement information, and the vision information and movement information include coordinate values, so the server 100 combines the user's vision information and movement information with the vision information and movement of the robot 330. It can be converted into information, and a motion control signal including the converted vision information and movement information can be transmitted to the robot system 300.

When step S540 is performed, the robot system 300 performs an operation to transport medicine according to the operation control signal (S550).

Afterwards, the robot system 300 transmits the vision information and movement information performed by the robot 330 according to the transport of medicine to the server 100, and the server 100 learns the vision information and movement information of the robot 330. If the information is not available, the motion model can be updated by learning the vision information and movement information of the received robot 330.

As described above, the system and method for providing a medicine transport service using an extended reality-based robot according to an embodiment of the present invention allows the user to control the robot and use extended reality technology when transporting medicine using a robot in a medical institution. It can be monitored.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely exemplary, and those skilled in the art will recognize that various modifications and other equivalent embodiments are possible therefrom. You will understand. Therefore, the true technical protection scope of the present invention should be determined by the scope of the patent claims below.

100: Server
200: user system
210, 310: Communication circuit
220: Extended reality display device
230: Voice recognition device
240: Haptic sensor
250, 340: Control device
300: Robot system
320: filming device
330: robot

Claims (6)

A server that learns at least one scenario for the order of transporting medicines and arrangement position coordinates, and a motion model for vision information and movement information of a robot performing each scenario;
A robot system equipped with an imaging device and performing an operation of transporting medicine according to a motion control signal; and
It includes a user system that displays images captured by the photographing device through an extended reality display device worn on the user's head and transmits control commands according to the user's voice and movement to the server,
The server is,
Based on the motion model, the vision information or movement information of the robot according to the control command is acquired, the motion control signal is generated, and the motion control signal is transmitted to the robot system. A pharmaceutical transportation service provision system used.
According to paragraph 1,
The robot system is,
communication circuit;
a photographing device that operates according to the movement of the extended reality display device to capture images;
A robot that carries out the operation of transporting medicines to the corresponding placement position coordinates according to the operation control signal; and
The image captured through the imaging device is transmitted to the server through the communication circuit, and an operation control signal from the server is transmitted to the robot or imaging device, so that the robot or imaging device operates according to the operation control signal. A system for providing medicine transport services using an extended reality-based robot, characterized in that it includes a control device that allows the drug to be transported.
According to paragraph 1,
The user system is,
communication circuit;
an extended reality display device worn on the user's head and displaying images captured through the photographing device in extended reality;
A voice recognition device that recognizes the user's voice;
A haptic sensor that recognizes the user's tactile information; and
The server determines the user's intention for the movement of the robot from voice data recognized by the voice recognition device or tactile information recognized through the haptic sensor, and sends a control command to control the robot according to the identified intention. A system for providing a medicine transport service using an extended reality-based robot, comprising a control device that transmits the image received from the robot system and outputs the image received from the robot system through the extended reality display device.
According to paragraph 1,
The server is,
Vision information or movement information of the robot performed according to the motion control signal is received from the robot system, and if the received vision information or movement information of the robot is unlearned information, the received vision information or movement information of the robot A medicine transport service providing system using an extended reality-based robot, characterized in that the motion model is updated by learning information.
When the user's voice data or motion information is acquired while the user system is displaying an image captured through the imaging device of the robot system through the extended reality display device worn on the user's head, the voice data or motion information is Transmitting a control command including a control command to a server;
The server generates a motion control signal by acquiring coordinates of robot components according to the control command based on a previously learned motion model, and transmitting the generated motion control signal to the robot system; and
A step in which the robot system performs an operation of transporting medicine according to a motion control signal.
A method of providing medicine transport service using an extended reality-based robot including.
According to clause 5,
The server further includes a step of learning a motion model for at least one scenario for the order of transporting medicines and placement position coordinates, and vision information and movement information of a robot performing each scenario,
After performing the operation of transporting the medicine,
The server receives vision information and movement information performed by the robot according to the transport of the medicine from the robot system, and when the received vision information and movement information of the robot are unlearned information, the received vision information of the robot And a method of providing a medicine transport service using an extended reality-based robot, further comprising the step of updating the motion model by learning motion information.