KR101397522B1 - An intravenous injection simulator system and method using augmented reality - Google Patents

Abstract

The present invention relates to augmented reality based intravenous training simulator system and method.
To this end, an augmented reality-based intravenous training simulator system according to an embodiment of the present invention includes an image capturing unit, a model storage unit storing vein information and intravenous position information, A learning model generating unit for generating a learning model including an extracted target image and characteristic point information of the target image based on the real time image captured by the image capturing unit and the learning model, An augmented reality realizing the augmented reality by mapping the vein information and the intravenous scan position information stored in the model storage unit in the real time image including the detected target image region and outputting the mapped information to the output device Section.

Description

TECHNICAL FIELD [0001] The present invention relates to an augmented reality-based intravenous training simulator system and method,

The present invention relates to intravenous injection training, and more particularly, to an augmented reality based intravenous training simulator system and method capable of maximizing the educational effect.

Animal medicine training is an important training course for preliminary veterinary practitioners to practice and practice medical care for animals and learn how to treat animals directly.

However, there is a problem that the regulations on the animal protection laws for the animal subject to the experiment are expanded, and the limitation of the number of experiments on the animals used in the experiment limits the opportunity of animal medical training to the trainees.

In the patent document 10-2012-0077226, a technique related to a wiring education system based on an augmented reality is described.

The object of the present invention is to solve the above-mentioned problems, and it is an object of the present invention to provide a method and apparatus for generating a learning model for intravenous training, detecting a target image region in a real- Which can maximize the educational effect by realizing an augmented reality by mapping an image used in an augmented reality.

According to one aspect of the present invention, an augmented reality-based intravenous training simulator system according to an embodiment of the present invention includes an image capturing unit, a model storage unit storing vein information and intravenous position information, A learning model generation unit that extracts a target image from an input image input from the input unit and generates a learning model that includes the extracted target image and minutia information of the target image; And a controller for mapping the vein information and the intravenous scan position information stored in the model storage unit in the real time image including the detected target image area to the output device And an augmented reality implementing unit for implementing the augmented reality.

The augmented reality-based intravenous training simulator system according to an embodiment of the present invention calculates a three-dimensional positional information and a directional angle of a vein and generates a haptic intravenous syringe that indicates the success or failure of vein as it is operated for intravenous injection in the real- .

In the augmented reality-based intravenous training simulator system according to the embodiment of the present invention, the haptic intravenous syringe includes an interface for notifying completion of intravenous injection, and a notification unit for notifying success or failure of intravenous injection upon completion of the intravenous injection .

In the IVUS training simulator system based on an augmented reality according to an embodiment of the present invention, the learning model generation unit may include a feature point detection unit for setting a corner portion of the target image in the input image as a feature point, A patch generator for generating patch tree information by constructing a set of feature point candidates in a tree form; and a patch generator for normalizing the generated patch tree information using the patch tree information for each angle generated by rotating the target image by a predetermined angle And a model generating unit for generating the learning model using the set feature points, the patch tree information, and the table.

In the augmented reality-based intravenous training simulator system according to the embodiment of the present invention, the target-image detecting unit may detect the number of feature points of the target image and the number of feature points that are close to each other within a preset range through the Gaussian pyramid image transformation of the real- An alignment unit for removing an outlier by applying a "RANSAC" algorithm to the image selected by the image conversion unit; and a calculation unit for calculating a homography of the image from which the outlier is removed, A correction unit that corrects the image from which the outlier is removed using the calculated homography, and a detection unit that detects the target image region in the corrected image.

In the augmented reality-based intravenous training simulator system according to the embodiment of the present invention, the augmented reality implementing unit maps a target image used for generating the learning model to a target image region of the real-time image, .

According to another aspect of the present invention, an augmented reality-based intravenous scanning training simulator method for extracting a target image from an input image and learning the target image including minutia information of the target image A step of generating a model, a step of, when a real-time image is inputted, detecting a target image region in the real-time image based on the learning model, and storing the stored vein information and intravenous And mapping the position information to output to the output device to implement the augmented reality.

The step of generating the learning model in the augmented reality-based intravenous training simulator according to the embodiment of the present invention includes the steps of setting a corner portion of the target image in the input image as a minutiae and setting a minutiae candidate group, Generating patch tree information by constructing the feature points and the feature point candidates in a tree form to generate patch tree information; generating patch tree information for each angle by rotating the target image by a predetermined angle; Storing the generated patch tree information in a table, and generating the learning model using the set feature point, the patch tree information, and the table.

The step of detecting the target image in the augmented reality-based intravenous training simulator according to the embodiment of the present invention may include: calculating a number of feature points of the target image among the plurality of images generated through the Gaussian pyramid image transformation of the real- The method comprising: selecting an image having a number of feature points in close proximity within a predetermined range; applying an "RANSAC" algorithm to the selected image to remove an outlier; Correcting the image from which the outlier has been removed using the calculated homography, and detecting the target image region from the corrected image.

The step of implementing the augmented reality in the augmented reality-based intravenous training simulator according to the embodiment of the present invention may further include mapping the image corresponding to the target image to the target image region and outputting the mapped image to the output device do.

According to the present invention, by implementing a simulator that can assist intravenous injection training using the augmented reality technique, it is possible to maximize the effect of intravenous injection training which requires precise manipulation.

Further, the present invention can enhance the convenience of intravenous injection training by displaying the intravenous position and vein information.

The present invention creates a learning model for intravenous training, implements an augmented reality by mapping an image used in generating a learning model to a target image region after detecting a target image region in a real-time image based on a learning model, Object recognition is possible.

1 is a block diagram illustrating an augmented reality-based intravenous training simulator system according to an embodiment of the present invention;
2 is a view showing the appearance of a haptic intravenous syringe used in an embodiment of the present invention,
3 is a block diagram showing an internal structure of a learning model generation unit according to an embodiment of the present invention;
FIGS. 4A and 4B are diagrams for explaining a feature point detection process for generating a learning model according to an embodiment of the present invention;
5 is a diagram for explaining a process of generating patch tree information according to an embodiment of the present invention;
6 is a block diagram illustrating an internal structure of a target image detecting unit according to an embodiment of the present invention.
FIG. 7 illustrates an exemplary embodiment of an augmented reality according to an exemplary embodiment of the present invention,
8 is a diagram illustrating a work bench system applied to an embodiment of the present invention,
9A to 9B are views for explaining a display device mounted on a work bench system,
10 is an exemplary view illustrating an augmented reality implemented using a workbench system according to an embodiment of the present invention;

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims.

In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. The following terms are defined in consideration of the functions in the embodiments of the present invention, which may vary depending on the intention of the user, the intention or the custom of the operator. Therefore, the definition should be based on the contents throughout this specification.

Hereinafter, an augmented reality-based animal training simulator system and method will be described with reference to the accompanying drawings.

1 is a block diagram illustrating an augmented reality based intravenous injection training simulator system in accordance with an embodiment of the present invention.

1, an augmented reality-based intravenous training simulator system includes an input device 100, a 3D model storage unit 120, an augmented reality simulator device 140, an output device 160, and the like. .

The input apparatus 100 may include an image capturing unit 102 for capturing and providing an image for realizing an augmented reality, and a haptic intravenous syringe 104 for intravenous injection.

The image capturing unit 102 is a means for capturing an image for realizing an augmented reality, for example, a camera for recognizing an augmented reality.

The image capturing unit 102 may further include an infrared camera as well as a camera for recognizing an augmented reality. That is, the image capturing unit 102 may be equipped with one infrared ray camera for recognition of an augmented reality and an infrared ray camera at the upper end of the monitor, which is one of the output devices 160, and an infrared ray camera may be mounted on the side surface.

In the embodiment of the present invention, the appearance of the haptic intravenous syringe 104 is in the form of a 5CC intravenous syringe used for actual intravenous injection, and it is possible for the experimenter to handle various types of input operations occurring during intravenous injection insertion But may include an input interaction for determining the success or failure of the intravenous simulation.

The haptic intravenous injector 104 may include at least one infrared ray illuminator 104a, a gyro sensor 104b, a button 104c, and a notification unit 104d. Here, the at least one infrared ray illuminator 104a can be recognized by an infrared camera, for example, but not limited to, an LED. The three-dimensional position of the haptic intravenous syringe 104 can be recognized by the infrared camera.

The notification unit 104d of the haptic intravenous syringe 104 is a means for informing the user of information on the success or failure of intravenous injection from the augmented reality simulator apparatus 140, . That is, the feedback vibration motor provides different vibrations according to success or failure, thereby enabling the experimenter to know whether the experiment is successful or not.

In addition, the haptic intravenous syringe 104 may calculate the direction angle information of the haptic vein injector 104 at its three-dimensional position and its position, and then provide it to the output device 160. Such information can be provided at a sampling rate of 5 KHz.

The haptic intravenous syringe 104 may include an interface 104c for informing the augmented reality simulator device 140 of the completion of intravenous injection.

The haptic intravenous syringe 104 is connected to the augmented reality simulator device 140 in a wired or wireless manner and provides information to the augmented reality simulator device 140 via wired or wireless communication with the button 104c, The augmented reality simulator device 140 can receive the success or failure of the injection through the wired or wireless communication.

The 3D model storage unit 120 stores vein information. Here, the process of generating vein information will be described. First, a beagle dog is selected as a target animal, and a 3D volume image of the images obtained after the CT image is formed. Then, various 3D shape vein information can be extracted by performing a segmentation process on the vein part of the 3D volume image.

On the other hand, the 3D model storage unit 120 stores the vein scan position information for each vein information.

The augmented reality simulator device 140 is provided with a learning model generation unit 142 for generating a learning model, a position corresponding to a target image detection unit 144 for detecting a target image in the real time image based on the real time image and the learning model, An augmented reality implementing unit 146 for outputting a resultant image obtained by mapping vein information and intravenous information to a target image detected from a real time image through an output unit 160 to implement an augmented reality, And the like.

The learning model generation unit 142 generates a learning model of a two-dimensional image (hereinafter, referred to as a target image) from an externally input image or an image photographed by the image capturing unit 102. For this purpose, A patch generation unit 210, a normalization unit 220, a model generation unit 230, a storage unit 240, and the like.

The minutia detection unit 200 can search for a corner portion in the target image using the "Harris Corner Detector ", and set it as a minutia.

In addition, the feature point detection unit 200 may assign scores to candidates that can be feature points in the process of extracting feature points. That is, when a feature point is extracted by applying a "Harris Corner Detector " to a two-dimensional image as shown in FIG. 4A, that is, a target image, corner portions and feature point candidates are extracted as shown in FIG.

The patch detection unit 210 configures patch tree information by constructing the feature points and the feature point candidate groups in a tree form.

First, as shown in FIG. 5, the patch detector 210 expresses feature points obtained from the "Harris Corner Detector " as a red circle, and expresses extracted feature point candidates as a green circle and a blue circle. Then, we generate patch tree information with the red circle as the top node, the green circle as the bottom node, and the blue circle as the bottom node.

The normalization unit 220 detects the patch tree information for each angle while rotating the input image by a predetermined angle, and then normalizes the patch tree information generated by the patch generation unit 210 using the detected patch tree information for each angle . That is, the normalizing unit 220 calculates the patch tree information while rotating the input image from 0 degrees to 360 degrees in 1 degree increments, normalizes the patch tree information generated by the patch generating unit 210 based on the calculated patch tree information, (Not shown).

The patch tree information is calculated for each angle as described above, and then the patch tree information based on the feature points is normalized based on the patch tree information, thereby providing patch tree information that is invariant to the rotation of the image.

The model generation unit 230 generates a learning model using the detected feature points, information in the form of a patch tree, and a lookup table, and stores the generated learning model in the storage unit 240. Here, the learning model can be stored in the storage unit 240 in the form of a text file.

The target image detecting unit 144 can detect the target image region in the real time image provided from the image taking unit 102 of the input apparatus 100 using the learning model stored in the storage unit 240. [

The target image detecting unit 144 may include an image converting unit 300, an aligning unit 310, a correcting unit 320, and a detecting unit 330, as shown in FIG.

The image converting unit 300 converts a real-time input image provided from the image capturing unit 102 into a Gaussian pyramid image. Here, the process of converting to a Gaussian pyramid image will be described below.

First, the size of real-time input image is 1/2, 1/4, 1/8 ... The Gaussian pyramid image is generated by restoring the downsampled image back to its original size by using the Gaussian blur method.

The resulting Gaussian pyramid image does not show the remaining feature points other than the strong feature points because the image is blurred.

In the embodiment of the present invention, the image transform unit 300 extracts the number of feature points from the smallest sampled Gaussian pyramid image to the size of the original real-time input image, and calculates the number of feature points in the learning model stored in the storage unit 240, The Gaussian pyramid image having a similar number of feature points within a preset range is selected from the number of feature points stored in the storage unit 240 through comparison between the numbers.

As described above, when a feature point similar to a target image exists in a real-time input image by searching a feature point number through a Gaussian pyramid image, a problem of detecting a false target image in a false real-time input image can be solved.

The aligner 310 processes the outliers and processes the outliers through a RANSAC (Random Sample Consensus, hereinafter referred to as RANSAC) method. Here, It means processing positions of minutiae similar to the target image which is erroneously detected.

The correcting unit 320 calculates the homography to correct and recognize the distorted target image obtained from the real-time input image. That is, since the target image portion is distorted according to the rotation of the subject photographed by the image capturing unit 102 and the photographing angle of the image capturing unit 102, the target image to be searched in the real- 142 do not have the form of the target image used to generate the learning model. Therefore, the corrector 320 corrects the real-time input image through the homography calculation.

The homography calculation process will be described. First, the four-corner point (corner) of the two-dimensional target image is set as a reference point, and then the image capturing unit 102 is controlled so that all the four vertices of the target image are displayed. And the coordinates are obtained in the input image of the corner.

Then, the homography is calculated by correlating the coordinates of the target image of the real world with the coordinates of the corner of the target image output to the real-time input image. Here, the coordinates of the target image can be obtained by an operation of multiplying the internal parameters of the image capturing unit 102 and the external parameters of the real world. The internal parameters can be obtained by projecting a three-dimensional real image in two dimensions. The external parameters include how far the image capturing unit 102 is from the origin of the real world and how far it has rotated.

The correction unit 320 calculates the homography using the external parameters and the internal parameters, and corrects the real time input image using the calculated homography.

The detection unit 330 can detect the target image region in the corrected real-time input image.

The augmented reality implementing unit 146 maps the vein information and the vein scan position information provided from the 3D model storage unit 120 in the real-time input image including the target image, and displays the mapped information through the output unit 160.

The augmented reality implementing unit 146 may implement an augmented reality by mapping an image of a target image, for example, a two-dimensional image, used to generate a learning model to the target image region detected by the detecting unit 330. [

That is, as shown in FIG. 7, a two-dimensional image 410 corresponding to a target image used to generate a learning model is mapped to a portion of the real-time input image 400 where the target image exists, 420 and the vein position information 430. FIG.

The output device 160 is an interface device for implementing an augmented reality, and examples thereof include a display device and a speaker. Examples of the display device include, but are not limited to, an LCD monitor.

Such a display device may be capable of rotating up to 80-50 degrees and may be mounted on the augmented reality simulator device 140. The haptic work bench system in which such a display device is mounted on the augmented reality simulator device 140 will be described with reference to Figs. 8, 9A, 9B, and 10. Fig.

8, a haptic workbench system 800 includes a display device, for example, a 22-24 inch LCD monitor, and a camera for augmented reality and an infrared camera 810 are installed at an upper end, And one infrared camera 820 may be installed on the side portion.

A display device detachably attached to the haptic workbench system is rotatable between 80-50 degrees as shown in FIGS. 9A and 9B, and the experimenter can adjust the screen of the monitor through such rotation.

When the experimenter performs the intravenous injection experiment using the haptic work bench system, as shown in FIG. 10, the experimenter holds the small animal leg model on one hand and the haptic intravenous injector 104 of FIG. 2 on the other hand. To carry out the experiment.

This haptic workbench system can be interlocked with the haptic intravenous syringe 104 of FIG. 2. That is, the infrared camera 810 at the upper end recognizes the infrared light emitter 104a installed on the side of the haptic vein injector 104, Dimensional plane position of the syringe 104 and the side infrared camera 820 measures the height of the infrared light emitter 104a of the haptic intravenous syringe 104 by measuring the height of the infrared light emitter 104a of the haptic intravenous syringe 104, Dimensional position value of the augmented reality simulator device 140 and provide it to the augmented reality simulator device 140. [

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention. Accordingly, the scope of the invention should not be limited by the described embodiments but should be defined by the appended claims.

100: Input device 102:
104: Haptic intravenous syringe 120: 3D model storage unit
140: augmented reality simulator device 142: learning model generation unit
144: target image detecting unit 146: augmented reality implementing unit
160: Output device 200:
210: patch information generation unit 220: normalization unit
230: model generation unit 240:
300: image converting unit 310:
320: a correction unit 330:

Claims (10)

An image capturing unit,
A model storage unit for storing vein information and intravenous position information,
A learning model generation unit that extracts a target image from the input image input from the image capturing unit or from outside and generates a learning model including the extracted target image and the minutia information of the target image;
A target image detecting unit for detecting a target image region in the real time image based on the real time image taken by the image taking unit and the learning model;
And an augmented reality implementing unit for mapping intravenous information and intravenous scanning position information stored in the model storage unit in a real-time image including the detected target image region and outputting the intravenous information and intravenous scanning position information to an output device to implement an augmented reality
Augmented reality based intravenous training simulator system.
The method according to claim 1,
The simulator system includes:
Further comprising a haptic intravenous syringe for calculating the three-dimensional position information and the directional angle and notifying the success or failure of the intravenous injection operation according to the operation for intravenous injection in the real time image
Augmented reality based intravenous training simulator system.
3. The method of claim 2,
The haptic intravenous syringe includes:
An interface for informing the completion of the intravenous injection,
And an informing unit informing the success or failure of the intravenous scanning operation upon completion of the intravenous scanning
Augmented reality based intravenous training simulator system.
The method according to claim 1,
Wherein the learning model generation unit comprises:
A feature point detector for setting a corner point of the target image in the input image as a feature point and setting a feature point candidate group;
A patch generator for generating the patch tree information by constructing the feature points and the feature point candidates in a tree form;
A normalizing unit for normalizing the generated patch tree information using the generated patch tree information for each angle by rotating the target image by a predetermined angle,
And a model generation unit that generates the learning model using the set feature points, the patch tree information, and the table
Augmented reality based intravenous training simulator system.
5. The method of claim 4,
Wherein the target-
An image transformation unit for transforming the Gaussian pyramid image of the real-time image into an image having a number of feature points in the target image and a number of feature points in a predetermined range,
An alignment unit for applying an "RANSAC" algorithm to an image selected by the image conversion unit to remove an outlier;
A correcting unit for calculating the homography of the image from which the outlier has been removed and correcting the image from which the outlier has been removed by using the calculated homography,
And a detector for detecting the target video region in the corrected image
Augmented reality based intravenous training simulator system.
5. The method of claim 4,
The augmented reality implementing unit includes:
And maps the target image used for generating the learning model to the target image region of the real-time image, and outputs the mapped target image to the output device
Augmented reality based intravenous training simulator system.
Extracting a target image from an input image, generating a learning model including the extracted target image and minutia information of the target image,
Detecting a target image region in the real-time image based on the learning model when a real-time image is input;
And mapping the pre-stored vein information and intravenous scan position information in a real-time image including the detected target image to output to an output device to implement an augmented reality
Augmented reality based intravenous training simulator method.
8. The method of claim 7,
Wherein the step of generating the learning model comprises:
Setting a corner portion of the target image in the input image as a minutiae and setting a minutiae candidate group;
Generating patch tree information by constructing the feature points and the feature point candidates in a tree form;
Generating patch tree information for each angle according to rotation of the target image by a predetermined angle, normalizing the generated patch tree information using the generated patch tree information for each angle, and storing the information in a table,
And generating the learning model using the set feature point, the patch tree information, and the table
Augmented reality based intravenous training simulator method.
9. The method of claim 8,
Wherein the step of detecting the target image comprises:
Selecting an image having a number of minutiae of the target image and a number of minutiae adjacent to each other within a predetermined range among a plurality of images generated through the Gaussian pyramid image transformation of the real-
Applying an "RANSAC" algorithm to the selected image to remove outliers,
Calculating homography for the image from which the outlier has been removed;
Correcting the image from which the outlier has been removed using the calculated homography;
And detecting the target image region in the corrected image
Augmented reality based intravenous training simulator method.
8. The method of claim 7,
The step of realizing the augmented reality includes:
And maps the image corresponding to the target image to the target image area and outputs the mapped image to the output device
Augmented reality based intravenous training simulator method.

Images