KR102265702B1 - Real-time detection device of superficial vein and blood flow - Google Patents

Abstract

The present invention relates to a real-time detection device of superficial vein and vascular abnormalities using near-infrared rays that detects a shape and blood flow of a superficial vein by analyzing a reaction specificity of hemoglobin in a blood vessel in a specific wavelength band in a near-infrared (N-IR) irradiation environment, and applies a high-speed image processing algorithm to the shape of the superficial vein and whether or not the blood vessel is abnormal to project, in real-time, on the original skin surface where the blood vessel is located. The present invention comprises: a near-infrared light source module; a camera module; an image processor module; a digital light processing module; and a communication module.

Description

Real-time detection device of superficial vein and blood flow using near-infrared rays

The present invention detects the shape and blood flow of the superficial vein by analyzing the reaction specificity of hemoglobin in the blood in a specific wavelength band in the near-infrared (N-IR) irradiation environment, and the shape and vascular abnormality of the superficial vein It relates to a real-time detection device for superficial vein and vascular abnormalities using near-infrared rays that project (投射, Projection) on the original skin surface where blood vessels are located by applying a high-speed image processing algorithm.

IV injection is one of the most important medical procedures performed in many medical fields. In most cases, skilled medical personnel perform IV injection without too much strain, but sometimes it is difficult for obese patients or idiosyncratic patients to see blood vessels with the naked eye. In the case of patients and children, the IV injection procedure is difficult.

In particular, in the case of a patient hospitalized for a long time due to a serious disease, Ringer's solution must be administered for almost 24 hours, so the IV injection must always be in place. As a result, it is no longer possible to find a location for IV injection in the arm, hand, or ankle, and eventually, the drug is injected through the central venous line.

For this reason, if a drug is injected into a non-vascular place during IV injection, symptoms such as infiltration, hematoma, nerve damage, and acute compartment syndrome occur.

In the above, infiltration refers to the leakage of intravenous fluid into the surrounding tissues outside the vein, and hematoma refers to a strain formed by bleeding into the surrounding tissues at the intravenous injection site, and nerve damage. occurs when the nerve sheath is substantially invaded during needle insertion or when the nerve is compressed by an intravenous catheter for a long period of time. It increases to the extent that it leads to a decrease in blood flow, which means that tissue ischemia, necrosis, and functional damage occur.

On the other hand, in the case of plastic surgery / dermatology, it is often necessary to avoid blood vessels during cosmetic procedures, and in the case of filler (filling material), which is popular recently, it is necessary to perform the procedure on the face where a large amount of microvessels and nerves are distributed. Accidents due to incorrect procedures are common.

Therefore, there is an urgent need to develop a technology that can determine the exact location and condition of blood vessels.

Registered Patent 10-1494638

An object of the present invention is to identify blood vessel information using the reflection principle of near-infrared (N-IR), reconstruct the identified blood vessel information into an image in the visible region, and accurately project the measured blood vessel information to the measured in-situ in real time. (Projection) and indication of vascular abnormalities together, it becomes possible to perform IV injections (or fillers, etc.) while grasping the location and shape of blood vessels that cannot be discriminated with the naked eye as they are. It can dramatically reduce medical accidents and side effects that occur in plastic surgery, and reduce the burden on medical personnel, enabling active treatment for patients, and also helps in treatment by checking vascular abnormalities at the location in real time. It is to provide a real-time detection device for superficial vein and vascular abnormalities using near-infrared rays.

In addition, it is an object to provide a real-time detection device for superficial vein and vascular abnormalities that prevents distortion or noise from occurring in the superficial vein detection process so that an accurate and clear blood vessel image is output.

A feature of the present invention for achieving the above object is a near-infrared light source module using a plurality of 760-790nm LED elements and 800-950nm LED elements, respectively, and these are alternately arranged and controlled; a camera module configured to include at least one lens for acquiring an image of a subcutaneous blood vessel and a near-infrared (IR) image sensor on which an image obtained from the lens is formed to acquire subcutaneous blood vessel data; The subcutaneous blood vessel data transmitted from the camera module is image-processed, but in the processed image, a change in the state of blood flow at a plurality of points in the same blood vessel, that is, a change in blood flow or a difference between oxidized Hb and reduced Hb, is determined and the difference exceeds the reference value. an image processor module for judging a blood vessel abnormality in the case of a blood vessel abnormality, and processing and outputting an image according to the blood vessel abnormality so that the user can check the blood vessel abnormality; a digital light processing (DLP) module for outputting a blood vessel image output from the image processor module on a photographed skin region in real time; There is a real-time detection device for superficial venous and vascular abnormalities using near-infrared rays, including

According to the present invention having the above configuration, the blood vessel image is accurately projected in-situ in real time, thereby dramatically reducing medical accidents and side effects that occur in the IV injection process, and avoiding the blood vessels. By not injuring blood vessels when necessary, it can reduce the burden on medical personnel in IV injection or plastic surgery situations, enabling active treatment for patients. It works.

Accordingly, by helping to find the veins of patients who are difficult to find veins in the blood collection room, such as the elderly and children, it reduces the burden on the blood collector and increases work efficiency, and the patient and guardian can visually check the veins, which gives a sense of stability. It has the effect of shortening the blood collection time because blood collection can be easily performed even by an inexperienced blood collector.

In addition, when performing a procedure in a vascular malformation clinic, the location of the lesion is checked and marked using ultrasound, or the procedure can be performed while directly observing the skin under the skin rather than performing the procedure while watching the ultrasound, which is more intuitive and increases the operator's convenience. , it can contribute to increasing the success rate of surgery by conveniently discriminating blood vessels and tissues that are indistinguishable with the naked eye during surgery.

In addition, since the subcutaneous layer can be accurately distinguished during injection procedures such as intravenous injection, intramuscular injection, and subcutaneous injection in plastic surgery, side effects of injection can be prevented.

In addition, the relative change in blood flow at multiple points in the same blood vessel is identified, and the difference in the concentration ratio of oxidized and reduced hemoglobin at each measurement point is determined, and if there is an abnormality, it is displayed at the location of the blood vessel in real time to indicate a specific disease or lesion. It can be usefully used for inferring to expand the applied medical field, help in diagnosing vascular diseases, and can be used as a simple discriminant test at low cost, helping to prevent vascular diseases.

1 is a view showing a real-time superficial vein detection device according to the present invention;
2 is a view showing a blood vessel detection image for each LED wavelength
3 is a view showing the operation of the visible light filter;
4 and 5 are views showing the action of the visible light filter according to the present invention;
6 to 8 are diagrams illustrating an example of driving a user interface for a mobile device;
9 is a diagram illustrating an example of real-time detection of changes in blood flow
10 to 15 are diagrams showing a real-time detection process of superficial veins according to the present invention
16 is a view showing an example of real-time detection of superficial vein and blood flow abnormalities according to the present invention

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

As shown in FIG. 1, the apparatus for real-time detection of superficial vein and vascular abnormalities using near-infrared light according to the present invention uses a plurality of 760-790nm LED elements and 800-950nm LED elements, respectively, and a near-infrared light source module in which they are alternately arranged and controlled (100); a camera module 200 configured to include one or more lenses 210 for acquiring an image of subcutaneous blood vessels and a near-infrared (IR) image sensor 220 on which an image obtained from the lens 210 is formed; The subcutaneous blood vessel data transmitted from the camera module is image-processed, but in the processed image, a change in the state of blood flow at a plurality of points in the same blood vessel, that is, a change in blood flow or a difference between oxidized Hb and reduced Hb, is determined and the difference exceeds the reference value. an image processor module 300 for judging a blood vessel abnormality and processing and outputting an image so that a user can visually check the blood vessel abnormality; A digital light processing (DLP) module (hereinafter referred to as a DLP module) 400 for outputting a blood vessel image output from the image processor module 300 on a photographed skin region in real time; .

In the drawings, unexplained reference numeral 500 denotes a communication module for communication with an external device, and 600 denotes an input/output module such as a menu selection button such as power on/off and a connector for charging or data transmission.

In the measurement of blood flow state change, the concentrations of oxidized (Oxide) and reduced (Deoxide) hemoglobin (Hb) in blood that are detected in real time are respectively gradated (繼照化, Scalarize), and oxidized and reduced hemoglobin at each point is measured. By summing the concentrations of , comparing the blood flow at a plurality of points, and comparing the concentrations of oxidized and reduced hemoglobin at the plurality of points, respectively, the change in the state of blood flow is identified.

Accordingly, an LED having an optimal wavelength band is selected by analyzing the infrared absorption rate of hemoglobin.

De-Oxygenated Hb (reduced hemoglobin) shows the highest absorption in the 650nm band, but the 650nm band is visible ray band, so skin penetration is weak and the red light source affects the image to be detected. In the invisible ray band, 760nm NIR Although it shows high absorption in the band, there is no manufacturer that produces Peak 760nm LED, so Peak 770nm LED (Marktech, Germany) is used.

Oxygenated Hb (oxidized hemoglobin) shows the highest absorption in the 940nm NIR band, but uses a Peak 850nm LED (OSRAM) that is best projected when projected in infrared and has no difference in absorption.

The blood vessel images obtained using the 770 nm LED device 110 and the 850 nm LED device 120 selected as described above are the same as in FIG. 2 , and among them, the 850 nm LED device (diffusion angle of 45°) enables optimal image acquisition. In other words, it is used alone when detecting superficial veins.

Therefore, in the present invention, a plurality of 770 nm LED elements 110 and 850 nm LED elements 120 are used as the near-infrared light source module 100 (in the embodiment of the present invention, a total of 16 are used, 8 each), and these are alternately arranged Also, they are turned on alternately during lighting control.

That is, when the 770nm LED element 110 is turned on, the 850nm LED element 120 is turned off, and when the 770nm LED element 110 is turned off, the 850nm LED element 120 is turned on.

On the other hand, the output of the near-infrared light source module 100 is adjusted and the sensitivity of the near-infrared image sensor is adjusted accordingly to enable detailed image control. For example, if the output of the near-infrared light source module is strengthened, the clarity of blood vessels can be increased, but Because the detail disappears, micro-vessels cannot be seen, and if the output of the near-infrared light source module is weakened to see micro-vessels, there is a correlation that the clarity of large blood vessels is weakened. Accordingly, the user can adjust the output of the near-infrared light source module and the sensitivity of the near-infrared image sensor according to the intended use.

The lens 210 constituting the camera module 200 is a C-Mount type lens, and uses Non-IR Coating Lenses so that the IR band can be sufficiently transmitted from the lens, and even at a short distance. The minimum focal length is 15 cm or less so that an image can be formed on the infrared (IR) image sensor 220 , and the blood vessel image projected from the DLP module 400 interferes with the IR image sensor 220 (flickering, flickering) In order to prevent this, as shown in FIGS. 1 and 3 , a visible light filter 230 which is a separate 720 nm low-cut filter is provided on the lens 210 .

The visible ray filter 230 is coated with a visible ray blocking material on the rear surface of the lens. Conventionally, a method of attaching a filter to the front of the lens and LED light source was used, but in this case, the loss of output LED light amount due to the scattering effect of the filter itself and Since it causes a reduction in the resolution of the acquired image, in the present invention, a clear image can be obtained without loss of LED light quantity and resolution of the acquired image by coating and filtering a visible ray blocking material on the rear surface of the lens.

Therefore, when the visible light filter 230 is applied, it blocks the infinite repetition of light from ambient light sources such as sunlight or lamp light, flickering images and Self Image of Vein, and near-infrared images of blood vessels (N- Only the IR Image of Vein) can pass through.

In the above, flickering is a structural problem caused by the DLP module outputting each RGB light source of the image to a rotating mirror, and the self-image infinite repetition is a problem that occurs while reproducing the image at the measured original position, that is, the reproduced image is reproduced again. It is a problem that is introduced into the measurement process.

As a result of measuring the performance of the visible light filter 230 with an optical spectroscope, it can be seen that light interference of 720 nm or less is completely blocked as shown in FIG. 4 , and FIG. 5 shows before/after using the visible light filter It can be seen that the interference disappears after use by representing the image.

The image processor module 300 uses an I.MX6Quad processor, which is Freescale Semiconductor's top-level multimedia-only processor focusing on high performance and low power consumption, and high-end portable media players that require HD-level video processing (portable media players with It is a solution optimized for HD video capability) and is designed based on the ARM Cortex-A9 MPCore platform.

In addition, the image processor module 300 is configured to include a separate PMIC for battery type power management.

The DLP module 400 is composed of a driver part for adjusting the image projection light source and an engine part for outputting an actual image image.

Then, the barrel is manufactured so that the focal length of the projection can be adjusted to 15 cm or less, the port mapping of the video signal is changed and implemented on the DLP controller board, and a dedicated device driver is developed and installed on the controller board. Porting.

In the present invention configured as described above, in order to match the recognition area of the camera module 200 and the projection area of the DLP module 400, the optical axes of the camera module 200 and the DLP module 400 are shifted to recognize the camera module 200 The area and the projection area of the DLP module 400 are matched, and distortion caused by this is corrected by the image processor module 300 .

In another method, an optical mirror is mounted to match the optical axis in order to match the recognition area of the camera module and the projection area of the DLP module.

On the other hand, the superficial vein real-time detection device is further provided with a communication module 500 such as Wi-Fi and Bluetooth, so that the obtained superficial vein image is transmitted to an external device, that is, a mobile device such as a mobile phone, a terminal (desktop, laptop, etc.), a server, etc. In order to transmit and control the operation with a mobile device or a terminal in charge, an example of a user interface for a mobile device for this purpose will be described below with reference to FIG. 6 .

In the user interface for the mobile device, a menu that variously adjusts and outputs the battery level of the superficial vein detection device, enlargement and reduction of the image, and the output of the detected superficial vein image as shown in FIG. 7 , that is, the Universal menu is real-time blood vessel detection It is a streaming menu to which an algorithm is applied, and the Fine Detail menu is a streaming menu with increased resolution for precise analysis (the blood vessel detection algorithm is applied only to CLAHE level 2), but it is not real-time, and the Blood Flow menu is used to precisely observe the relative change in blood flow. It is a streaming menu with increased resolution, and it is an image obtained through the cross lighting of 770nm LED and 850nm LED, and is not a real-time image.

In addition, as shown in FIG. 8, the user interface for the mobile device includes a menu for displaying the blood vessels in black as it is in the image taken of the superficial vein image, or displaying the blood vessels brightly by inverting the image,

In addition, by adopting a natural air cooling method, the superficial vein detection device can be used in an operating room, etc., and can stream high-resolution processed blood vessel images to mobile devices such as tablets in real time, which can be used as detailed analysis data of superficial veins. It can be used for vascular malformation analysis, identification of internal organs in open surgery, and retinal vascular degeneration in ophthalmology.

In addition, the superficial vein image acquired through the superficial vein detection device is transmitted to the server, and the server applies an artificial intelligence (AI) learning technique specialized for medical image analysis, so that the optimal puncture is performed during IV injection. (穿刺, Centesis) position is recommended, but the optimal puncture position is suggested by weighting through the puncture image obtained during actual puncture and the puncture position recommendation of an expert, and the puncture position suggested by the server is the superficial vein detection device of the present invention It is transmitted to the skin and projected onto the skin through the DLP module.

A process of real-time detection of blood flow changes in the image processor module will be described below.

When the 760-790nm LED device and 800-950nm LED device obtained from the camera module are turned on, a blood vessel image including Hb concentration information for each LED device is acquired, and the obtained blood vessel image for each LED device is the same as shown in FIG. The change in the state of blood flow at a plurality of points in the blood vessel, that is, the relative change in blood flow or the difference in the concentration ratio of oxidized Hb to reduced Hb is determined, and the relative change in blood flow or the difference in the concentration ratio of each hemoglobin is the reference value (standard value to be determined as normal) If it deviates from , it is judged as a blood vessel abnormality, and the image is processed and output in a specific character or color that the user can immediately check.

On the other hand, in the measurement of blood flow state change in the above, 770 nm and 850 nm LEDs are cross-lit and the concentrations of Oxide and Deoxide hemoglobin (Hb) in blood flow detected in real time are gradated (繼照化, Scalarize), respectively. This is to understand the change in the state of blood flow.

And the blood flow can be inferred as the sum of the reduced and oxidized Hb, and the relative change between the two points is derived by adding the difference in the grayscale change at the point to be measured as a grayscale value using a 24-bit system using the RGB888 interface. , If the relative degree of rice is out of the range of the reference value (relative degree of change in normal cases), it is judged as a vascular abnormality.

On the other hand, in order to display the superficial vein in its original position, the image processor module corrects the optical distortion of the lens and corrects the projection angle error of the DLP. Pre-Processing: After the pre-processing, Geometric transformation process that performs geometric transformation including perspective transformation on the acquired image: Equalization is performed by setting a separate Contrast Limiting value in blocks by means of histogram equalization (increasing the contrast of the acquired image). Equalization [CLAHE (Contrast Limited Adaptive Histogram Equalization)] Process: Noise removal that removes noise that has increased after applying CLAHE [Median Blur] Process: Inverting process that inverts the contrast of darkened blood vessel image [Invert]: Image segmentation and binarization Through the shape detection [Binarization] process: is performed to detect the superficial vein and display it at the original position through the DLP module.

Optical distortion correction in the pre-processing process will be described.

An image acquired through a lens basically has a distortion phenomenon, and major distortions include radial distortion and tangential distortion.

Among them, Radial Distortion is caused by the refractive index of the convex lens, and the degree of image distortion is determined by the distance from the center (a straight line is distorted into a curve as it gets farther from the center of the image). is corrected by Equation 1.

Equation 1

Tangential distortion occurs when the camera lens and the image sensor are not horizontal during the camera manufacturing (assembly) process or the centering of the lens itself does not match. It is a distortion that can make some areas of the image appear closer than expected. It is corrected by Equation 2.

Equation 2

Other physical distortion factors are further corrected by applying the Brown-Conrady Model.

Equation 3

The projection angle error correction prevents image distortion caused by the lens angle. Before blood vessel measurement, a reference point (or reference line) is first irradiated to a location to be detected using the DLP module, and the coordinates of the reference point are obtained from the image data collected through the image sensor, and the camera module and the DLP module are used as reference coordinates. Sort and apply to blood vessel detection.

For example, as shown in FIG. 10 , after acquiring the coordinates of the reference point, that is, each of the four corner points (indicated by red dots), the checker board image projected by the DLP module projected by the DLP module is captured by the camera. , align the camera module and the DLP module with the obtained reference coordinates and apply it to blood vessel detection.

That is, in order to match the recognition area of the camera module and the projection area of the DLP module, the optical axis of the camera module is shifted by about 16° so that the blood vessel recognition area of the camera module and the projection area of the DLP module coincide with each other. The distortion is then corrected in the image processor module.

After making the recognition area of the camera module and the projection area of the DLP module match as described above, a geometric transformation process is performed on the 2D image obtained from the camera module, which occurs because the optical axes of the camera module and the DLP module do not match. As for correcting perspective deformation, the geometric deformation includes perspective, scaling, translation, rotation, and the like, and an example thereof is shown in FIG. 11 .

The equalization process is performed by the histogram equalization method. Histogram equalization is to increase the contrast of the 2D image as shown in FIG. 12, and the intensity of the histogram can be better dispersed, and due to this, the The image area of the contrast ratio becomes higher.

Thus, histogram equalization is useful for images with light or dark backgrounds and foregrounds, especially overexposed or underexposed images, although it can depict more detail, but since it is applied over the entire area, contrast is improved while the detail in the image is caused by excessive brightness. There is a problem in that area information is lost.

Therefore, in order to solve this problem, as shown in FIG. 13 , the histogram equalization technique is changed to the modified CLAHE technique and applied, and the CLAHE is repeated multiple times if necessary.

The CLAHE (Contrast Limited Adaptive Histogram Equalization) divides the image into small blocks (tiles) and then performs histogram equalization for each block. However, when there is a lot of noise in the image, Since noise is also amplified together, it is solved by setting a separate contrast limiting value. And artifacts occurring at the boundary of tiles are removed by applying bilinear interpolation.

Then, in order to remove the noise increased after CLAHE is applied, as shown in FIG. 14 , a noise removal process, Median Blur, is performed, which aligns the values of surrounding pixels and their own values according to their sizes, and the median value (Median). ) and selects it as its own pixel value.

Thereafter, as shown in FIG. 14 , an invert process is performed to invert the contrast of the image to brightly express dark blood vessels.

Lastly, the shape detection process (binarization), that is, dividing the image and detecting the shape of a desired part or object, scans the entire input image and turns the pixel value at the location into white color if the threshold value is greater than the pixel value. (1 or 255), and if the pixel value is smaller than the threshold value, it is set to black and output, and it is projected to the in-situ in real time. The image and the actual projected image are shown in Fig. 15, and when there is a blood vessel abnormality, as shown in Fig. 16, the color of the projection image is changed (eg, green is normal, red is blood flow abnormality), or whether there is a vessel abnormality in the projected image by text or indicated by symbols, etc.

Since the superficial vein is displayed at the original location in this way, the location of the puncture can be accurately identified during the procedure. In order to set a more accurate location of the puncture, the superficial vein image acquired through the real-time detection device for blood flow change is transmitted to the server, and in the server By applying artificial intelligence (AI) learning technique specialized in medical image analysis, the optimal puncture position is recommended during IV injection, but the puncture image obtained during actual puncture and the puncture position of a specialist are recommended The optimal puncture position is proposed through weighting, and the puncture position proposed from the server is transmitted to the image processor module and projected onto the skin through the DLP module.

100: near-infrared light source module
200: camera module
210: lens
220: near infrared (IR) image sensor
300: image processor module
400: Digital Light Processing (DLP) module

Claims (7)

a near-infrared light source module 100 using a plurality of 760-790 nm LED devices and 800-950 nm LED devices, respectively, and in which they are alternately arranged and controlled;
a camera module 200 configured to include one or more lenses 210 for acquiring an image of subcutaneous blood vessels and a near-infrared (IR) image sensor 220 on which an image obtained from the lens 210 is formed;
The subcutaneous blood vessel data transmitted from the camera module is image-processed, but in the processed image, a change in the state of blood flow at a plurality of points in the same blood vessel, that is, a change in blood flow or a difference between oxidized Hb and reduced Hb, is determined and the difference exceeds the reference value. an image processor module 300 for determining a blood vessel abnormality in the case of a blood vessel abnormality and displaying the blood vessel abnormality in an output image;
a digital light processing (DLP) module (hereinafter referred to as a DLP module) 400 for outputting a blood vessel image output from the image processor module 300 on a photographed skin region in real time;
a communication module for transmitting the blood vessel image acquired through the image processor module 300 to an external device in real time; consists of,
Before detecting the blood vessel image, the DLP module irradiates the skin with a reference point or reference line, and in the image obtained from the camera module, the image processor module processes the image based on the reference point or reference line. A real-time detection device for superficial vein and vascular abnormalities using near-infrared rays, characterized in that output. The method according to claim 1, wherein the lens 210 constituting the camera module 200 uses C-Mount type Non-IR Coating Lenses, and the minimum focal length is 15 cm or less. A real-time detection device for superficial vein and vascular abnormalities using near-infrared rays. [Claim 2] The apparatus for detecting superficial vein and blood vessel abnormalities using near-infrared rays according to claim 1, characterized in that a visible ray filter 230 is provided by coating a visible ray blocking material on the rear surface of the lens 210. The method according to claim 1, wherein the image sensor 220 uses an N-IR CMOS image sensor, changes the HDMI image output to 24 bits in parallel, and the camera image input is pin-mapped in parallel 12-bit gray scale. pin mapping), a real-time detection device for superficial vein and vascular abnormalities using near-infrared rays. delete According to claim 1, In order to match the recognition area of the camera module 200 and the projection area of the DLP module 400, the optical axis of the camera module 200 is shifted by about 16° to the blood vessel recognition area of the camera module A real-time detection device for superficial vein and blood vessel abnormalities using near-infrared rays, characterized in that the projection area of the DLP module matches, and the distortion caused by this is corrected by the image processor module (300). The method of claim 1, wherein the superficial vein image acquired through the real-time detection device for superficial vein and vascular abnormality is transmitted to a server, and an artificial intelligence (AI) learning technique specialized for medical image analysis is applied in the server, intravenous ( IV Injection) recommends the puncture position during the procedure, but suggests the puncture position by assigning weights through the puncture image obtained during actual puncture and the expert puncture position recommendation, and the puncture position suggested by the server is superficial. A real-time detection device for superficial vein and vascular abnormalities using near-infrared rays, which is transmitted to a vein detection device and projected onto the skin through a DLP module.

Images