CN115587981A

CN115587981A - Vein re-projection system method and terminal based on FPGA

Info

Publication number: CN115587981A
Application number: CN202211245617.0A
Authority: CN
Inventors: 崔笑宇; 李德亮; 项雯析; 周国伟; 孙加兵; 刘嘉巍; 柏群; 齐瑞群; 高兴华; 赵越; 陈洪铎
Original assignee: Northeastern University China; First Hospital of China Medical University
Current assignee: Northeastern University China; First Hospital of China Medical University
Priority date: 2022-10-12
Filing date: 2022-10-12
Publication date: 2023-01-10

Abstract

A vein reprojection system method and a terminal based on an FPGA solve the problems of unobvious blood vessel detection depth information and more noise interference in the prior art. The system method and the terminal utilize the parallel processing technology to realize the two-dimensional matched filtering algorithm and the image segmentation algorithm on the FPGA, thereby extracting the vein image shot by the infrared camera and providing the real-time response of the algorithm processing result on a small handheld portable device. Based on the parallelization characteristic of multiplication and addition in convolution operation and the pipeline design of a programmable logic gate array (FPGA), a hardware implementation framework can be divided into two modules of image data caching, parallel output and parallel pipeline convolution; the terminal is composed of three parts of image acquisition, image processing and image display. The image enhancement can be effectively realized, the image noise is reduced, the image segmentation effect is more ideal, and the processing efficiency is improved.

Description

Vein re-projection system method and terminal based on FPGA

Technical Field

The invention belongs to the technical field of vein imaging systems, and particularly relates to a vein reprojection system method and a vein reprojection terminal based on an FPGA (field programmable gate array) and capable of being used for image processing for extracting blood vessel information.

Background

Venipuncture and intravenous injection are common medical procedures in clinical treatment. However, in many cases, the blood vessels of some patients are not easily observed due to the difference in thickness or length of the blood vessels, and obesity, dark skin, and the degree of denseness of hair can be main factors. In addition, it is difficult for medical personnel to quickly and accurately perform injections while wearing protective clothing and goggles or while wearing surgical gloves. Therefore, research into venipuncture assistance devices has become a hotspot.

The optical characteristics of blood vessels indicate that hemoglobin in blood vessels has a better ability to absorb near-infrared light than surrounding tissue; based on this principle, researchers have made many efforts to develop practical blood vessel extraction devices and methods. Existing vein extraction algorithms are mainly classified into two categories: one is based on image filtering and pixel segmentation method, extracting vein area directly on image layer; another type is a line tracking method that gradually expands along the vascular structure starting from the seed point. For example: jinglinang Zhao et al proposed a chain tracking method based on superpixels to segment retinal vessels; amir Hajian et al used an improved repetitive line tracking method to extract vein structures. Moreover, besides these two main methods, there are also some other theoretical methods that have some effect; for example: the depth and thickness of the subcutaneous vein are measured by using three kinds of equal-wavelength diffuse reflection images, and finger veins and the like are effectively segmented from the images by applying an algorithm based on active contour and denoising.

Moreover, the identification and separation of blood vessels from the hand back vein image is the core step of the whole hand back vein imaging system; the former algorithm of these developments is mainly executed on the CPU. However, given the characteristics of medical image processing, i.e. the special size of the image, and the high requirements for portability in many scenarios such as emergency situations, software-based algorithms tend to be time consuming and not suitable for carrying around. Also, with the accelerated development of electronic technology, hardware-based image processing systems have become more and more popular, and particularly, the advent of FPGA-based image programming technology has promoted the development of a hardware solution with miniaturization, rapidness, and low power consumption. Therefore, there is a need for an improved method and apparatus for rapid extraction of blood vessel images for venipuncture.

Disclosure of Invention

Aiming at the problems of unobvious blood vessel detection depth information and more noise interference in the prior art, the invention provides the FPGA-based vein reprojection system method and the FPGA-based vein reprojection terminal, which can effectively realize image enhancement, reduce image noise, enable the image segmentation effect to be more ideal and enable the processing efficiency to be higher.

The technical scheme adopted by the invention is as follows: the vein reprojection system method based on the FPGA comprises the following steps:

firstly, a camera driving module acquires image data;

converting the serial input pixel data into parallel, and outputting the pixel data through an I/O module;

step three, sending the parallel image data to a matching library convolution filter;

step four, sending the output result of the step three to a data comparison module, finding out the maximum value in parallel, and keeping the maximum value as a filtering result;

step five, performing threshold segmentation on the image obtained in the step four to obtain a binary image;

step six, multiplying the binary image obtained in the step five by a corresponding matched filtering image, and further extracting blood vessel depth information;

and step seven, outputting the blood vessel depth information image obtained in the step six through a display module.

The method comprises the following steps that firstly, blood vessel structure detection is carried out after image data are collected, and images are processed by utilizing two-dimensional matched filtering; after filtering processing, the most useful target information can be separated from the original image; meanwhile, blood vessel-like templates representing different directions are designed to serve as filters to match the vein images of the back of the hand; observing the gray distribution diagram of the cross section of the blood vessels, finding that the intensity distribution of the blood vessels has the characteristic of inverted bell shape and can be approximated by a Gaussian function; the mathematical expression of the gaussian curve is:

wherein K (x, y) is a gaussian kernel function, and the coefficients define the intensity distribution range of the gaussian function along the x-axis coordinate center, i.e., the width of the blood vessel; l is the length of the vessel truncated along the y-axis.

And step two, generating a link structure buffer array corresponding to the image resolution by adopting a buffer, buffering the image data read in series by using an N-1 line buffer for an image window with the size of NxN, and converting the pixel values of the N lines into parallel output.

Step three, four rotated matched filters are used, the angular resolution is 45 degrees, and the specific calculation method is as follows: let p = [ x, y]Is a discrete point in the kernel, θ _i Is the direction in which the ith nucleus matches the vessel angle;

the matrix rotation is calculated assuming the Gaussian kernel is centered at the origin [0,0 ]:

one domain N is defined as:

wherein u and v represent the rotation weight value in the ith kernel, and are new coordinates after rotation;

defining truncation on a Gaussian curve infinitely extending in the positive and negative directions of the x axis, wherein the same coefficient is the width of the blood vessel, and L is the length of the blood vessel truncated along the y axis;

if the number of points in the domain N is A, the mean of the Gaussian kernel function can be calculated as:

the convolution templates finally used are:

wherein,

the coefficient sigma defines the deviation degree of the Gaussian function from the center of the x axis, and corresponds to the width of the image seed vessel;

l is the length of the vessel that is truncated along the y-axis.

And step four, the data comparison module adopts a parallel pipeline comparison mode.

And step five, the threshold segmentation adopts a maximum inter-class variance method.

A vein re-projection terminal based on FPGA comprises an image acquisition module, an image processing module and an image display module; the image acquisition module comprises 4 LED light sources, a dichroic mirror and a CMOS camera; the image processing module is composed of an FPGA core circuit board; the image display module is composed of a DLP projector. The near-infrared light source irradiates the skin, the camera captures a vein infrared image, and the vein infrared image is captured by the image sensor through the dichroic mirror; the FPGA core circuit board is communicated with the image sensor, and image data captured by the image sensor is read through the parallel port; then, the FPGA is used for carrying out image processing of the algorithm, and image caching is carried out inside the SDRAM; and finally, the FPGA controls the transmission of the video stream, and the processed image is transmitted to the DLP and then projected to the original position of the skin for displaying.

The mounting frame structure of the vein re-projection terminal is arranged in three layers, the first layer is a light source structure, and 4 LED light sources are arranged in a square shape with the camera as the center so as to achieve the effect of uniform illumination; the second layer is a dichroic mirror, and consists of a projection optical machine and an FPGA circuit board; the last layer consists of a camera and a fan; and, adopt non-coaxial design structure on the light path, namely: the projector is not on one axis with the image sensor.

Setting the environment of a darkroom to filter out image interference caused by visible light; the optical filter with double laminated layers is adopted, so that the problem of visible light interference caused by insufficient compactness of the optical filter is effectively solved; moreover, the DLP is equidistant from the image sensor optical path, and the image sensor position is adjustable. So as to ensure that the whole optical system can be real-time and complete, and reduce mutual interference between signals to the maximum extent.

The projector is provided with a mode structure controlled by keys, and can project green, yellow, blue or yellow enhanced images under the control of the keys; moreover, the blood vessels with different thicknesses can be processed by adjusting the threshold value through a key, so that the projected image is clearer; the free switching between the 850nm near-infrared light source and the 940nm near-infrared light source can be realized through a key structure; meanwhile, the projection position and direction can be controlled to be adjusted through the keys. The green mode can completely extract blood vessels and project a green image; the yellow mode increases the background, so that the blood vessel image is more obviously compared with the surrounding environment of the hand; the blue mode can more highlight the depth information of the blood vessel, so that the medical care personnel can conveniently perform venipuncture; a yellow enhancement mode, which can project a more detailed capillary image; is suitable for blood vessels of different people with obesity, dark skin and the like.

The invention has the beneficial effects that: the core of the system method and the terminal for vein re-projection based on the FPGA is that a two-dimensional matched filtering algorithm and an image segmentation algorithm are realized on a programmable logic gate array FPGA by utilizing a parallel processing technology, so that a vein image shot by an infrared camera is extracted, and real-time response of an algorithm processing result is provided on a small handheld portable device. Based on the parallelability of multiplication and addition in convolution operation and the pipeline design of a programmable logic gate array (FPGA), a hardware implementation framework can be divided into two modules, namely image data caching, parallel output and parallel pipeline convolution; the terminal is composed of three parts of image acquisition, image processing and image display, can effectively realize image enhancement, reduce image noise, and enable the image segmentation effect to be more ideal and the processing efficiency to be higher.

Drawings

Fig. 1 is a gray scale distribution diagram of a cross section of a vein of the back of a hand to which the present invention is applied.

Fig. 2 is a flow chart of the algorithm of the present invention.

Fig. 3 is a hardware configuration diagram of the present invention.

Fig. 4 is a diagram of an optical system of the present invention.

Fig. 5 is a graph showing the effect of the present invention.

Fig. 6 is an overall flow chart of the present invention.

Detailed Description

Aiming at the problems of unobvious blood vessel detection depth information, more noise interference and the like in the prior art, the invention provides a vein re-projection system method and a terminal based on an FPGA (field programmable gate array), which comprise blood vessel structure detection, two-dimensional matched filter algorithm design, image data caching and input/output, parallel pipeline convolution, and finally fast obtaining a clear blood vessel image and re-projecting the clear blood vessel image onto the skin through a small handheld terminal; therefore, the image is enhanced, the image noise is reduced, the image segmentation effect is more ideal, and the processing efficiency is improved.

The details of the present invention are explained. The vein reprojection system method based on the FPGA comprises the following steps:

step one, a camera driving module collects image data. And after image data are collected, detecting the blood vessel structure, wherein the blood vessel structure is detected to process the image for subsequent two-dimensional matched filtering. The essence of the matched filtering algorithm is to design an optimal filter to match the target shape in the region of interest in the image. After filtering processing, the most useful target information can be separated from the original image; meanwhile, in this algorithm, blood vessel-like templates representing different directions are designed as filters to match the vein image of the back of the hand. As shown in FIG. 1, the invention selects a Gaussian blood vessel template as a sign signal of a hand back vein image for matching and recognition according to the gray distribution diagram of the hand back vein cross section.

Observing the gray distribution diagram of the cross section of the blood vessels, finding that the intensity distribution of the blood vessels has the characteristic of inverted bell shape and can be approximated by a Gaussian function; the mathematical expression of the gaussian curve is:

Moreover, the camera driving module can extract image data according to the data output protocols of the CMOS sensors of different manufacturers, including but not limited to LVDS, MIPI and the like.

And step two, converting the serial input pixel data into parallel, and outputting the pixel data through the I/O module. And generating a link structure buffer array corresponding to the image resolution by adopting a buffer, buffering the image data read in series by using an N-1 line buffer for an image window with the size of NxN, and converting the pixel values of the N lines into parallel output.

And step three, sending the parallel image data to a matching library convolution filter. Since the angular resolution and size of the matched filter are related to the filtering performance and hardware resource cost, it is important to consider both factors. Through the verification of a preliminary experiment in MATLAB and a simulation experiment on FPGA, the performance of the invention is best when 4 matched filters are used. I.e. an angular resolution of 45. First the filter templates rotated at different angles should be calculated.

Four rotated matched filters are used, the angular resolution is 45 degrees, and the specific calculation method is as follows: let p = [ x, y]Is a discrete point in the kernel, θ _i Is the direction in which the ith nucleus matches the vessel's oblique angle.

the corresponding point of the point p in the rotated coordinate system is = [ u, v ] =, and both ends of the gaussian curve extend infinitely in the positive and negative directions of the x-axis, respectively, and therefore, the gaussian curve needs to be truncated. Thus, a domain N is defined as:

where u and v represent the rotation weight values in the ith kernel, which are also the new coordinates after rotation.

Truncation is defined for a gaussian curve infinitely extending in the positive and negative directions of the x-axis, the above coefficient is the width of the blood vessel, and L is the length of the blood vessel truncated along the y-axis.

Then the coefficient at the point [ x, y ] in the ith gaussian kernel matrix can be obtained as:

in the matching filtering, to avoid changing the original background gray characteristic of the image, the average value of the convolution kernel coefficients should be 0. The convolution templates ultimately used are therefore:

wherein,

the coefficient sigma defines the deviation degree of the Gaussian function and the center of the x axis, and corresponds to the width of the blood vessel of the image; l is the length of the vessel that is truncated along the y-axis.

Step four, sending the output result of the step three to a data comparison module, finding out the maximum value in parallel, and keeping the maximum value as a filtering result; and the data comparison module adopts a parallel pipeline comparison mode.

Step five, carrying out threshold segmentation on the image obtained in the step four to obtain a binary image; the threshold segmentation adopts a maximum inter-class variance method.

And step six, multiplying the binary image obtained in the step five by the corresponding matched filtering image, thereby extracting the blood vessel depth information.

And step seven, outputting the blood vessel depth information image obtained in the step six through a display module. The display module can convert the blood vessel depth information image into a data transmission protocol supported by displays and projectors of different manufacturers for output.

Two-dimensional matched filtering is a template-based image processing technique whose key operation is the convolution between pixels in an image window and a convolution kernel template. The convolution kernel templates are computed according to the algorithm described previously. The filter coefficients for these six filters were previously computed in MATLAB, and these pre-computed 8-bit convolution kernel weights were used as macro definitions in Verilog language in the Quartus II environment. Based on the huge parallel characteristic of multiplication and addition processes in convolution operation and the pipeline design of a programmable logic gate array (FPGA), the two-dimensional matched filtering algorithm realized on the FPGA can be divided into three modules: the device comprises an image data cache and I/O module, a convolution operation module and a threshold segmentation module. The test proves that the hardware execution process is simpler, more hardware resources are saved, and the efficiency is higher.

The image data caching and input/output module is responsible for processing pixel data, enabling the pixel data to be input into an image processing flow in parallel, and managing output of filtered pixel values. The present invention uses first-in-first-out (FIFO) as a line buffer for image data. For an image window of size N, the present invention uses N-1 line buffers to buffer the serially read image data and convert the N lines of pixel values to parallel outputs.

As shown in fig. 2, the algorithm flow of the present invention is as follows, in the convolution module, the parallel image data outputted by the input/output module will be outputted and converted by different filters at 0 °, 45 °, 90 ° and 135 °, respectively. For an image window of size N x N, the first N rows of 8-bit image pixel values will be loaded into the scan line buffer at the beginning of the convolution unit. Once the scan line buffer is full, all data is sent together to the next stream, the multiply module. The invention adopts an improved convolution multiplication and shift addition method to improve the resource utilization efficiency. And adding the multiplied results to obtain a result value of the matched filter in the corresponding direction. From the convolution result values in four filters, i.e., four directions, the largest one is selected as the result value of matched filtering in the current window. And traversing the whole image by using windows with the same filter size to complete the parallel convolution of the FPGA.

The invention also relates to a vein re-projection terminal based on the FPGA, and as shown in figure 3, the terminal consists of three modules, namely an image acquisition module, an image processing module and an image display module. Including MT9V034 camera, 4 LED light sources, OPD01M ray apparatus, dichroic mirror, FPGA core circuit board, fan, battery, DLP projection arrangement.

A near infrared light source 4 is irradiated on the skin, and a camera captures a vein infrared image, which is captured by an image sensor through a dichroic mirror 5. The FPGA core circuit board 1 is communicated with the image sensor 2, and then image data captured by the image sensor is read through the parallel port. And then, after the image processing of the algorithm is carried out by the FPGA1, image caching is carried out in the SDRAM 3. And finally, the FPGA1 controls the transmission of video stream, the processed image is transmitted to the DLP6 and then projected to the original position of the skin for imaging, thereby completing vein re-projection auxiliary venipuncture based on the FPGA.

The mounting bracket structure at vein reprojection terminal divides the three-layer setting, and the first layer is the light source structure, and 4 LED light sources use the camera to arrange as central square to reach the even effect of illumination. The LEDs are all at a radiation intensity of 50mw, and the light intensities of 850nm and 940nm are respectively controlled by a linear voltage stabilizer. The second layer is a dichroic mirror, and consists of a projection optical machine and an FPGA circuit board; the last layer consists of a camera and a fan. Furthermore, as shown in fig. 4, the design of the present invention on the optical path adopts a non-coaxial design, i.e. the projector 1 and the image sensor 2 are not on the same axis. The dichroic mirror 4 of the present invention is selected to cut off at 780nm, which not only allows light with wavelengths above 780nm to be almost completely transmitted, but also allows light with wavelengths below 780nm to be almost completely reflected.

Meanwhile, in order to ensure that the whole optical system can be real-time and complete and reduce mutual interference among signals to the maximum extent, the invention designs the following on the optical path: firstly, the environment of the darkroom 3 is designed to filter out image interference possibly caused by visible light; secondly, the optical filter with double stacked layers is designed, so that the visible light interference caused by insufficient compactness of the optical filter can be effectively solved; finally, the DLP1 and the image sensor 2 are equidistant in light path, and the position of the image sensor can be adjusted; and 5, a near infrared light source.

In order to adapt to blood vessels of different crowds with obesity, dark skin and the like, the projection light machine adopts a mode structure controlled by a key; the light machine can be controlled by the keys to project green, yellow, blue and yellow enhanced images. The green mode can completely extract blood vessels and project a green image. The yellow mode increases the background, and makes the blood vessel image more obvious with the surrounding environment of the hand. The blue mode can highlight the depth information of blood vessels more, makes things convenient for medical personnel to carry out venipuncture. Yellow enhancement mode, a more subtle capillary image can be projected. In addition, the blood vessels with different thicknesses can be processed by adjusting the threshold value through the key, so that the projection image is clearer. And the free switching between the near infrared light source of 850nm and the near infrared light source of 940nm can be realized through a key structure. Meanwhile, the projection position and direction can be controlled to be adjusted through the keys.

Claims

1. A vein re-projection system method based on FPGA is characterized by comprising the following steps:

firstly, a camera driving module acquires image data;

2. The system method of FPGA-based vein reprojection of claim 1, wherein: the method comprises the following steps that firstly, blood vessel structure detection is carried out after image data are collected, and images are processed by utilizing two-dimensional matched filtering; after filtering processing, the most useful target information can be separated from the original image; meanwhile, blood vessel-like templates representing different directions are designed to serve as filters to match the vein images of the back of the hand; observing the gray distribution diagram of the cross section of the blood vessels, finding that the intensity distribution of the blood vessels has the characteristic of inverted bell shape and can be approximated by a Gaussian function; the mathematical expression of the gaussian curve is:

wherein K (x, y) is a Gaussian kernel function, and the coefficient defines the intensity distribution range of the Gaussian function along the x-axis coordinate center, namely the width of the blood vessel; l is the length of the vessel truncated along the y-axis.

3. The system method of FPGA-based vein reprojection of claim 1, wherein: and step two, generating a link structure buffer array corresponding to the image resolution by adopting a buffer, buffering the image data read in series by using an N-1 line buffer for an image window with the size of NxN, and converting the pixel values of the N lines into parallel output.

4. The system method of FPGA-based vein reprojection of claim 1, wherein: step three, four rotated matched filters are used, the angular resolution is 45 degrees, and the specific calculation method is as follows: let p = [ x, y]Is a discrete point in the kernel, θ _i Is the direction in which the ith nucleus matches the vessel angle;

the matrix rotation is calculated assuming the Gaussian kernel is centered around the origin [0,0 ]:

one domain N is defined as:

wherein u and v represent the rotation weight values in the ith kernel, and are new coordinates after rotation;

the convolution templates finally used were:

wherein,

l is the length of the vessel that is truncated along the y-axis.

5. The system method of FPGA-based vein reprojection of claim 1, wherein: and step four, the data comparison module adopts a parallel pipeline comparison mode.

6. The system method of FPGA-based vein reprojection of claim 1, wherein: and step five, the threshold segmentation adopts a maximum inter-class variance method.

7. A vein re-projection terminal based on FPGA comprises an image acquisition module, an image processing module and an image display module; the image acquisition module comprises 4 LED light sources, a dichroic mirror and a CMOS camera; the image processing module is composed of an FPGA core circuit board; the image display module is composed of a DLP projector.

8. The FPGA-based vein reprojection terminal of claim 7, wherein: the mounting frame structure of the vein re-projection terminal is divided into three layers, the first layer is a light source structure, and 4 LED light sources are arranged in a square shape with the camera as the center so as to achieve the effect of uniform illumination; the second layer is a dichroic mirror, and consists of a projection optical machine and an FPGA circuit board; the last layer consists of a camera and a fan; and, adopt non-coaxial design structure on the light path, namely: the projector is not on one axis with the image sensor.

9. The FPGA-based vein reprojection terminal of claim 7, wherein: setting the environment of a darkroom to filter out image interference caused by visible light; the optical filter with double laminated layers is adopted, so that the problem of visible light interference caused by insufficient compactness of the optical filter is effectively solved; moreover, the DLP is equidistant from the image sensor optical path, and the image sensor position is adjustable.

10. The FPGA-based vein reprojection terminal of claim 7, wherein: the projector is provided with a mode structure controlled by keys, and can project green, yellow, blue or yellow enhanced images under the control of the keys; moreover, the blood vessels with different thicknesses can be processed by adjusting the threshold value through a key, so that the projected image is clearer; the free switching between the 850nm near-infrared light source and the 940nm near-infrared light source can be realized through a key structure; meanwhile, the projection position and direction can be controlled to be adjusted through the keys.