KR102152940B1 - Medical practice contents cps(contents sercice platform) interworking interface - Google Patents

Abstract

According to an aspect, at least one virtual environment simulator connected to a server, the virtual environment simulator providing visual information to a user through a wearable display device; And a server that provides an evaluation simulation to the virtual environment simulator, obtains a simulation result from the virtual environment simulator and provides it to a terminal, and provides feedback information on the simulation result; and a virtual environment simulation providing system comprising: have.

Description

Medical practice contents CPS(Contents Sercice platform) interworking interface {MEDICAL PRACTICE CONTENTS CPS(CONTENTS SERCICE PLATFORM) INTERWORKING INTERFACE}

The present invention relates to a medical practice content CPS (Contents Sercice platform) interworking interface.

In the medical field, surgery is an important field that is directly connected to the patient's treatment and life. In these operations, doctors who perform surgery are usually trained through observation and practice.

In the case of training through conventional observation and practice, if the surgical situation does not occur, there is a problem that training for the surgery is limited because there are cases in which training for the situation does not occur.

In addition, in the case of a conventional surgical simulator, even if a surgical image is learned according to a predetermined procedure or even if it is demonstrated, it is not possible to receive an expert's feedback, and thus it is difficult to learn whether a correct simulation has been performed.

One task is to provide a medical practice content CPS (Contents Sercice platform) interworking interface that can simulate and learn surgery through a virtual environment.

Another task is to provide a medical practice content CPS (Contents Sercice platform) interworking interface that can receive feedback on simulation results.

According to an aspect, at least one virtual environment simulator connected to a server, the virtual environment simulator providing visual information to a user through a wearable display device; And a server that provides an evaluation simulation to the virtual environment simulator, obtains a simulation result from the virtual environment simulator and provides it to a terminal, and provides feedback information on the simulation result; and a virtual environment simulation providing system comprising: have.

It is possible to provide a medical practice content CPS (Contents Sercice platform) interworking interface that can simulate and learn surgery through a virtual environment.

It is possible to provide a medical practice content CPS (Contents Sercice platform) interworking interface that can receive feedback on the simulation result.

1 is an environment diagram showing a simulation system according to an embodiment.
2 is a block diagram showing a server according to an embodiment.
3 is a block diagram of a simulator according to an embodiment.
4 is a block diagram of a terminal according to an embodiment.
5 is a flowchart illustrating a method of providing a simulation according to an exemplary embodiment.
6 is a diagram illustrating a feedback UI according to an embodiment.
7 is a diagram illustrating acquisition of a simulation image according to an exemplary embodiment.

The above objects, features and advantages of the present invention will become more apparent through the following detailed description in conjunction with the accompanying drawings. However, in the present invention, various changes may be made and various embodiments may be provided. Hereinafter, specific embodiments will be illustrated in the drawings and described in detail.

In the drawings, the thicknesses of layers and regions are exaggerated for clarity, and in addition, an element or layer is "on" or "on" of another component or layer. What is referred to includes not only directly above another component or layer, but also a case where another layer or other component is interposed in the middle. Throughout the specification, the same reference numerals represent the same elements in principle. In addition, components having the same function within the scope of the same idea shown in the drawings of each embodiment will be described with the same reference numerals.

If it is determined that a detailed description of known functions or configurations related to the present invention may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted. In addition, numbers (eg, first, second, etc.) used in the description of the present specification are merely identification symbols for distinguishing one component from another component.

In addition, the suffixes "module" and "unit" for constituent elements used in the following description are given or used interchangeably in consideration of only the ease of preparation of the specification, and do not themselves have distinct meanings or roles.

According to an aspect, at least one virtual environment simulator connected to a server, the virtual environment simulator providing visual information to a user through a wearable display device; And a server that provides an evaluation simulation to the virtual environment simulator, obtains a simulation result from the virtual environment simulator and provides it to a terminal, and provides feedback information on the simulation result; and a virtual environment simulation providing system comprising: have.

In addition, the terminal may include a first terminal used by a user using the virtual environment simulator and a second terminal used by an evaluator providing feedback on the simulation result.

In addition, the feedback information may include an image selected through the terminal among images included in the evaluation result, a comment input through the terminal, and identification information of the terminal user.

In addition, the server may match and store the simulation result and the feedback information, and provide the evaluation result and the feedback information to a requesting terminal.

Further, the terminal may provide a predetermined feedback UI and obtain the feedback information through the feedback UI.

In addition, the feedback UI includes an image selection object for obtaining a selection for at least one image among a plurality of images included in the simulation result, a drawing tool providing a display on the image, and a comment for inputting a comment on the image. It may include an input window.

1 is an environment diagram illustrating a simulation system 10000 according to an embodiment.

Referring to FIG. 1, a simulation system 10000 may include a server 1000, at least one simulator 2000, and at least one terminal 3000.

The server 1000 may be connected to at least one simulator 2000 and at least one terminal 3000.

The server 1000 may provide a simulation to the simulator 2000.

According to an embodiment, the simulation may include a simulation image and an evaluation scenario.

The evaluation scenario may include information on an operation that a user must perform while performing a simulation, a guideline required to perform a simulation, assistance information such as movement induction, and information on an evaluation criterion.

The server 1000 may provide a feedback user interface (UI) to the terminal 3000.

The feedback UI may be for providing at least one of information on a simulation result and feedback on a simulation result.

The simulator 2000 may be connected to the server 1000.

The simulator 2000 may acquire a simulation from the server 1000.

Also, the simulator 2000 may provide a simulation to a user, and obtain a result of the user's simulation performance.

According to an embodiment, the simulator 2000 may acquire an image while a user performs a simulation.

For example, the simulator 2000 may acquire a simulation image including an image of an operation performed in a virtual environment.

The simulator 2000 may provide the simulation result to the server 1000.

The terminal 3000 may be connected to the server 1000.

The terminal 3000 may obtain a simulation result from the server 1000.

In addition, the terminal 3000 may visually provide a simulation result to a user through a feedback user interface (UI).

In addition, the terminal 3000 may obtain comment information from a user and provide it to the server 1000.

2 is a block diagram showing the server 1000 according to an embodiment.

Referring to FIG. 2, the server 1000 may include a server communication unit 1100, a server display unit 1200, a server input unit 1300, a server storage unit 1400, and a server control unit 1500.

The server communication unit 1100 may be connected to the simulator and the terminal through at least one communication method of wired communication and wireless communication.

For example, wireless communication is a Wi-Fi network, 3G, LTE network, 5G, mobile communication network such as LoRa, wave (Wireless Access in Vehicular Environment), beacon, zigbee, Bluetooth , Bluetooth Low Energy (BLE), etc. may be included.

In addition, wired communication may include a twisted pair cable, a coaxial cable, or an optical cable.

The server display unit 1200 may be provided as a monitor, TV, or display panel that outputs visual information.

In addition, when the server display unit 1200 is provided as a touch screen, the server display unit 1200 may perform the function of the server input unit 1300.

The server input unit 1300 may obtain a signal corresponding to a user's input.

The user's input may be, for example, pressing, clicking, touching, or dragging a button.

The server input unit 1300 may be implemented as, for example, a keyboard, a keypad, a button, a jog shuttle, and a wheel.

The server storage unit 1400 may store data.

The server storage unit 1400 may store information obtained from the outside.

In addition, the server storage unit 1400 may store information necessary for the operation of the server 1000.

For example, the server storage unit 220 includes magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and floptical disks. The same magneto-optical media, and ROM, RAM, flash memory, SSD, CD-ROM, DVD-ROM, or USB may be provided.

The server controller 1500 may oversee operations for the server 1000.

3 is a block diagram of a simulator 2000 according to an embodiment.

Referring to FIG. 3, the simulator 2000 may include a wearable display device 2000a, a controller 2000b, and a simulator controller 2000c.

In addition, the wearable display device 2000a includes a wearable display communication unit 2100a, a wearable display storage unit 2200a, a wearable display sensor unit 2300a, a wearable display control unit 2400a, and a wearable display screen output unit ( 2500a) and a wearable display sound output unit 2600a.

The wearable display communication unit 2100a may be connected to the simulator controller 2000c.

The wearable display communication unit 2100a may be connected to the simulator controller 2000c through at least one of wired communication and wireless communication.

The wearable display storage unit 2200a may store data.

The wearable display storage unit 2200a may store a program required for the operation of the wearable display device 2000a.

Also, the wearable display storage unit 2200a may store information obtained from the outside.

The wearable display sensor unit 2300a may acquire a state of the wearable display device 2000a and a signal corresponding to a user's input.

The wearable display sensor unit 2300a according to an embodiment may include a wearable display motion sensor module 2310a and a wearable display acoustic sensor module 2300a.

The wearable display motion sensor module 2310a may acquire a signal regarding the state of the wearable display device 2000a.

For example, the wearable display motion sensor module 2310a may obtain rotation information regarding rotation of the wearable display device 2000a.

As another example, the wearable display motion sensor module 2310a may acquire movement information regarding the movement of the position of the wearable display device 2000a.

Wearable display motion sensor module 2310a is an acceleration sensor, gyroscope, gyro sensor, MEMS, geomagnetic sensor, inertial sensor (IMIU), image sensor, light sensor, illuminance sensor, photo sensor, infrared sensor, color sensor, depth sensor. Alternatively, it may include an electromagnetic wave sensor or the like.

The wearable display acoustic sensor module 2320a may acquire a signal corresponding to a sound introduced from the outside.

For example, the wearable display acoustic sensor module 2320a may be a microphone.

The wearable display controller 2400a may oversee the operation of the wearable display device 2000a.

The wearable display controller 2400a may be exemplarily implemented as a microprocessor, a microcontroller, a DSP, a processor, an electrical unit for performing a controller function, or the like.

In addition, according to a software implementation, the wearable display controller 2400a may be implemented with a separate software module that performs at least one function or operation in embodiments such as procedures and functions. The software code can be implemented by a software application written in an appropriate programming language.

The wearable display screen output unit 2500a may output visual information to a user.

For example, the wearable display screen output unit 2500a may output an image for virtual reality. In addition, as another example, the wearable display screen output unit 2500a may output an image of 3D virtual reality.

The wearable display screen output unit 2500a may be provided as an image output device such as a liquid crystal display (LCD), electronic paper, an LED display, an OLED display, a curved display, a stereoscopic (a three-dimensional display using binocular parallax).

The wearable display sound output unit 2600a may output auditory information.

The wearable display sound output unit 2600a may be provided as a sound device such as a tuner, a playback device, an amplifier, and a speaker.

The controller 2000b may provide information on the location of the controller.

The controller 2000b may provide information on the location of the controller to the wearable display device 2000a.

3 shows that the controller 2000b is connected to the wearable display device 2000a so that the controller 2000b provides information on the location of the controller 2000b to the wearable display device 2000a, but is limited thereto. If the controller 2000b is connected to the simulator controller 2000c, the controller 2000b may provide information on the location of the controller to the simulator controller 2000c.

In addition, the controller 2000b may further provide a signal input through an input device provided to the controller 2000b.

The simulator control unit 2000c may oversee the operation of the components provided to the simulator 2000.

For example, the simulator control unit 2000c may be provided with an electronic device such as a PC, a tablet, or a mobile.

4 is a block diagram of a terminal 3000 according to an embodiment.

Referring to FIG. 4, the terminal 3000 may include a terminal communication unit 3100, a terminal display unit 3200, a terminal input unit 3300, a terminal storage unit 3400, and a terminal control unit 3500.

The terminal communication unit 3100 may connect the terminal 3000 to an external electronic device. For example, the terminal communication unit 3100 may connect the terminal 3000 to an external electronic device such as the server 1000.

In addition, the terminal communication unit 3100 may be a communication module supporting wired and/or wireless communication.

The terminal display unit 3200 may output visual information.

When the terminal display unit 3200 is provided as a touch screen, the terminal display unit 1200 may perform the function of the terminal input unit 3300. In this case, depending on the selection, a separate terminal input unit 3300 may not be provided, and a terminal input unit 3300 performing limited functions such as volume control, a power button, and a home button may be provided.

The terminal input unit 3300 may acquire a signal corresponding to a user's input.

The terminal input unit 3300 may be implemented as, for example, a keyboard, a keypad, a button, a jog shuttle, and a wheel.

In addition, the user's input may be, for example, pressing of a button, touching, or dragging.

When the terminal display unit 3200 is implemented as a touch screen, the terminal display unit 3200 may serve as the terminal input unit 3300.

The terminal storage unit 3400 may store data.

The terminal storage unit 3400 may be implemented as, for example, flash memory, RAM, ROM, SSD, SD CARD, and optical disk.

The terminal storage unit 3400 may store data necessary for the operation of the terminal 3000.

The terminal controller 3500 may oversee the operation of the terminal 3000.

5 is a flowchart illustrating a method of providing a simulation according to an exemplary embodiment.

Referring to Figure 5, the simulation providing method includes a step of requesting a simulation (S100), providing a simulation (S200), obtaining a simulation result (S300), and posting the simulation result (S400). I can.

According to an embodiment, the step of requesting to provide a simulation may be performed (S100).

The simulator 2000 may request the server 1000 to provide a simulation.

According to an embodiment, the simulator 2000 may obtain information on a desired simulation from a user.

For example, the simulator 2000 may obtain information on a simulation by receiving identification information on a desired simulation from a user.

According to an embodiment, the simulator 2000 may obtain user identification information from a user, and may request a simulation corresponding to the user identification information from the server 1000.

According to an embodiment, the simulator 2000 may request a simulation corresponding to the simulator identification information given to the simulator 2000 from the server 1000.

According to an embodiment, the step of providing a simulation may be performed (S200).

The server 1000 may provide a simulation to the simulator 1000.

According to an embodiment, the server 1000 may provide a simulation corresponding to the simulation identification information requested by the simulator 2000 among the previously stored simulations to the simulator 2000.

According to an embodiment, the server 1000 may provide a simulation corresponding to the user identification information acquired from the simulator 2000 to the simulator 2000.

The server 1000 may pre-store a simulation to be provided for each user. For example, the server 1000 may store a first simulation to be provided to a first user and a second simulation to be provided to a second user in advance.

The server 1000 may acquire a simulation corresponding to the acquired user identification information. For example, when the server 1000 acquires user identification information corresponding to the first user, the server 1000 may acquire a first simulation to be provided to the first user.

The server 1000 may provide the acquired simulation to the simulator 2000.

The server 1000 may provide a simulation corresponding to the acquired simulator identification information to the simulator.

The server 1000 may pre-store a simulation to be provided for each user. For example, the server 1000 may store a first simulation to be provided to a first user and a second simulation to be provided to a second user in advance.

Alternatively, a simulation to be provided for each simulator type may be previously stored in the server 1000.

For example, the server 1000 may be provided with a simulator type corresponding to the first simulator 2000a.

The simulator type may be, for example, an otolaryngology simulator, an orthopedic simulator, or the like, or may be a type for detailed surgery such as a hip joint surgery simulator and a tumor removal surgery simulator.

According to an embodiment, the step of obtaining a simulation result may be performed (S300).

The simulator 2000 may provide a simulation result to the server 1000.

According to an embodiment, the simulation result may be an image in which a user performs a simulation.

The user may perform the simulation by wearing the wearable display device 2000a provided in the simulator 2000 and carrying the controller 2000b.

The simulator 2000 may provide a simulation image to a user through the wearable display device 2000a.

The simulation image may include a background and a virtual object.

The background may be provided as a predetermined image.

In addition, the virtual object may include an object to be operated on and a virtual object corresponding to the controller.

The object to be operated on may be a virtual object corresponding to a region to be operated by a user using a controller. For example, the object to be operated on may be a virtual object corresponding to a tumor to be removed during surgery.

The object to be operated on may be changed by the user's manipulation. For example, when the user performs a predetermined operation through the controller 2000b, the object to be operated may be removed from the simulation image or replaced with another virtual object.

In addition, the simulator 2000 may provide an image of a tool corresponding to the controller to the simulation image based on coordinate information of the controller 2000b. For example, it may be provided at a location corresponding to the coordinates of the controller 2000b in the simulation image.

In addition, in the simulation image, a virtual object corresponding to the controller 2000b may be provided instead of the actual image of the controller 2000b. For example, in the simulation image, an image of a surgical tool corresponding to the controller 2000b may be provided.

The simulator 2000 may provide a simulation image to the server 1000.

In addition, the simulator 2000 may provide evaluation information on the simulation to the server 200.

The evaluation information for the simulation may be an evaluation for the user performing a predetermined operation required in the simulation.

For example, the simulator 2000 may evaluate that the controller 2000b has performed a predetermined operation when the controller 2000b is positioned at a predetermined coordinate. For another example, the simulator 2000 may evaluate a user's performance for a predetermined operation required in a simulation by subtracting a score corresponding to an error between the coordinates of the controller 2000b and a predetermined coordinate. For a more specific example, if the error between the coordinates of the controller 2000b and the predetermined coordinate is 5, the controller 2000 subtracts 3 points corresponding to the error from 10 points assigned to the predetermined operation, and obtains 8 points. It can be obtained as an evaluation score for a predetermined motion.

The simulator 2000 may acquire a simulation image provided during the execution of the simulation, and may provide the acquired simulation image to the server 1000.

According to an embodiment, the step of posting the simulation result may be performed (S400).

The server 1000 may obtain a simulation image as a simulation result.

In addition, the simulation result may include evaluation information for an operation predetermined in the simulation. In the simulation result, the evaluation information for the predetermined motion is not only the score for the predetermined motion, but also the coordinate information of the controller 2000b for the predetermined motion, the error between the coordinates for the predetermined motion and the coordinate information of the controller 2000b. It may further include information of.

The server 1000 may post online so that the terminal can check the simulation result through the feedback UI and provide and confirm feedback on the simulation result.

For example, the server 1000 may post online through a web platform, an online application, or the like.

6 is a diagram illustrating a feedback UI according to an embodiment.

Referring to FIG. 6, the feedback UI may provide a simulation result image and comment information on the simulation result.

In addition, the comment information on the simulation result may include identification information on the user who made the comment, the created comment, and an image corresponding to the comment.

In order to obtain such comment information, the feedback UI includes an image selection object for obtaining a selection for at least one image among a plurality of images included in the simulation result image, a drawing tool providing a display on the image, and a comment on the image. A comment input window for inputting may be provided.

In addition, according to an embodiment, when an image corresponding to a comment is selected, the feedback UI may be provided so that a section corresponding to the image corresponding to the comment is played in the simulation result image.

As an example, when the image corresponding to the comment is an image for the time point 1:00 in the simulation result image, the terminal 3000 may provide the image to be played from the time point 1:00 among the simulation result images.

As another example, when the image corresponding to the comment is an image for the time point of 1:00 in the simulation result image, the terminal 3000 may provide an image of the simulation result image to be played back for one minute from the time point 1:00.

In addition, according to an embodiment, the feedback UI may further provide information on at least one of a simulator that provided a simulation result image and a user who performed the simulation.

For example, the feedback UI may provide information such as identification information, type, etc. of the simulator that provided the simulation result image.

As another example, the feedback UI may provide information such as identification information, affiliation, number of times, major, etc. of the user who performed the simulation.

7 is a diagram illustrating acquisition of a simulation image according to an exemplary embodiment.

According to an embodiment, the server 1000 may acquire a simulation image based on an externally acquired surgical image.

7(a) is an exemplary diagram of a surgical image acquired from the outside by the server 1000, and FIG. 7(b) is an exemplary diagram illustrating acquiring a region to be operated on from the surgical image.

The server 1000 may perform a surgical image from the outside, and may determine the type of the acquired surgical image.

For example, the server 1000 may further acquire information on the type of the surgery image while acquiring the surgery image.

For a more specific example, while acquiring the surgical image, the server 1000 may further acquire information indicating that the corresponding surgical image is an image for a nose tumor removal operation.

The server 1000 may acquire a simulation image based on the surgical image.

The server 1000 may determine an area to be operated on from the operation image.

For example, when there is an area having a similarity or greater to a pre-stored surgery target sample in the surgery image, the server 1000 may determine the corresponding area as the surgery target area.

As another example, the server 1000 may determine a region that satisfies the condition as a region to be operated on based on a predetermined condition for determining the operation target.

For a more specific example, the predetermined conditions for determining an operation target may be conditions such as color and size, and the server 1000 may determine an area satisfying conditions such as color and size as the operation target area.

In addition, the server 1000 may obtain a simulation image by replacing the operation target area with a virtual object.

The server 1000 may provide an object to be operated corresponding to an area to be operated from among previously stored objects to be operated.

As an example, when the surgical image is the removal of tumor A, the server 1000 may provide a simulation image by providing an object to be operated on the tumor A among previously stored objects to be operated on to an area to be operated.

In the above, the configuration and features of the present invention have been described based on the embodiments according to the present invention, but the present invention is not limited thereto, and various changes or modifications can be made within the spirit and scope of the present invention. It will be apparent to those skilled in the art, and therefore, such changes or modifications are found to belong to the appended claims.

Claims (6)

At least one virtual environment simulator connected to a server-the virtual environment simulator provides visual information to a user through a wearable display device, and includes a controller;
A first terminal used by a user who uses the virtual environment simulator;
A server providing an evaluation simulation to the virtual environment simulator, obtaining a simulation result from the virtual environment simulator, and providing feedback information on the simulation result;
Including; a second terminal used by an evaluator providing the feedback information on the simulation result;
The simulation result includes an image in which the user performs a simulation, the image includes a virtual object that is a surgical simulation target, and the virtual object is the case when the user performs a predetermined operation through the controller of the simulator. It can be removed from the image or replaced with another virtual object,
The simulation result further includes evaluation information for a predetermined motion in the simulation, and the evaluation information includes a score for the predetermined motion, coordinate information of the controller for the predetermined motion, coordinates for the predetermined motion, and It may include error information with the coordinate information of the controller,
The simulator may provide the evaluation information to the server, and when the controller is located at a predetermined coordinate, it may be evaluated as performing the predetermined operation, and the controller corresponding to an error with the predetermined coordinate By subtracting a score, it is possible to evaluate the performance of the user for the predetermined operation required in the simulation,
When there is an area with a similarity or higher than a predetermined similarity to a sample stored in advance in the surgical image, the server determines the area with a similarity or higher as a target area for surgery, and obtains a simulation image in which the area for surgery is replaced with a virtual object In addition, the server may provide an object to be operated corresponding to the region to be operated from among previously stored objects to be operated,
The server matches and stores the simulation result obtained from the virtual environment simulator and the feedback information obtained from the second terminal, and stores the simulation result and the feedback to a first terminal requesting the simulation result and the feedback information. Providing information, and providing the feedback information on the simulation result to a web platform or application,
The second terminal provides a predetermined feedback UI, obtains the feedback information through the feedback UI, and the feedback information may include a comment input through the second terminal, and to obtain the comment The feedback UI provides an image selection object for obtaining a selection for at least one image among a plurality of images included in the image, a drawing tool for providing a display on the image, and a comment input window for inputting a comment on the image. In the case of selecting an image corresponding to the comment, the feedback UI provides a section corresponding to the image corresponding to the comment in the image to be reproduced.

delete The method of claim 1,
The feedback information further includes an image selected through the second terminal among images included in the evaluation information, and identification information of a user of the second terminal.
Virtual environment simulation delivery system.




delete delete delete

Images