CN114332058A - Serum quality identification method, device, equipment and medium based on neural network - Google Patents

Abstract

The invention discloses a serum quality identification method, device, terminal device and computer-readable storage medium based on a neural network. A test tube image is obtained by image acquisition of a test tube containing serum samples; the test tube image is input into a preset neural network. In the network model, for the neural network model to output the serum quality identification result for the serum sample, wherein the neural network model is obtained by performing convolutional neural network model training on test tube images. The present invention can improve the efficiency of serum quality identification, and effectively reduce the influence of labels on test tubes, thereby ensuring the accuracy of model identification of serum quality.

Description

Serum quality identification method, device, equipment and medium based on neural network

technical field

The present invention relates to the technical field of serum quality analysis, and in particular, to a method, device, terminal device and computer-readable storage medium for identifying serum quality based on a neural network.

Background technique

With the progress of society, the inspection level and automation of in vitro diagnostic equipment have gradually improved. In vitro diagnostic equipment such as biochemical analyzers and immunological analyzers must test the quality of the serum samples before performing biochemical or immunological tests on the serum. Only when the test results are qualified can the samples be used for further related tests. Because the blood collected through the test tube usually removes normal samples, it may also include abnormal samples such as hemolytic state, chylous state or jaundice state. These abnormal samples will be screened out in the previous process of testing the quality of the samples. .

In the past, the quality detection and identification of serum samples to screen out abnormal samples were all observed and judged by experienced technicians with the naked eye. Therefore, not only did different technicians have different judgment standards, but also the overall efficiency of screening abnormal samples was also improved. relatively low. In addition, in the prior art, serum quality can also be determined by extracting the color value of the liquid surface of the serum to be detected after dividing the respective color ranges for serum samples in normal, hemolytic, chyle, icteric and other states in advance. judge. However, this method is easily affected by the label affixed to the test tube containing serum, resulting in errors in color judgment.

The above content is only used to assist the understanding of the technical solutions of the present invention, and does not mean that the above content is the prior art.

SUMMARY OF THE INVENTION

The main purpose of the present invention is to provide a serum quality identification method, device, terminal device and computer readable storage medium based on a deep neural network, aiming to solve the existing method for quality identification and judgment of serum samples contained in test tubes, and the efficiency Technical issues that are low or prone to errors due to labels on tubes.

The embodiment of the present invention proposes a method for identifying serum quality based on a neural network, and the method for identifying serum quality based on a neural network includes:

Image acquisition is performed on the test tube containing the serum sample to obtain the test tube image;

Inputting the test tube image into a preset neural network model for the neural network model to output the serum quality identification result for the serum sample, wherein the neural network model performs convolutional neural network model training through the test tube image get.

Optionally, the method performs convolutional neural network model training through multiple exposure test tube images;

The described neural network-based serum quality identification method also includes:

acquiring a multi-exposure test tube image, and extracting a matrix area belonging to the non-label area of the test tube from the multi-exposure test tube image;

Input the matrix region into the preset first convolution module to train the first convolutional neural network model, and obtain the output of the first convolutional neural network model after the first convolution module performs the first convolutional neural network model training for the matrix region. feature map;

The feature maps are stacked and then input into a preset second convolution module to train a second convolutional neural network model to obtain a neural network model for serum quality identification for serum samples.

Optionally, the first convolution module and the second convolution module include: a convolution layer and a pooling layer, the first convolution module and the second convolution module except the last two volumes In addition to the accumulation layer, a pooling layer is connected after every two convolutional layers;

The convolutional layers in the first convolutional module include multiple first convolutional layers with stride 1 and multiple second convolutional layers with stride 2. In every two first convolutional layers, the output The first convolutional layer that is not connected to the pooling layer at the end is connected to one of the second convolutional layers;

The number of convolution kernels of the two first convolution layers at the end of the first convolution module is smaller than the number of convolution kernels of the other first convolution layers and the second convolution layer.

Optionally, the end of the second convolution module is connected to a fully connected layer and a logistic regression layer, and the feature maps are stacked and then input to a preset second convolution module to train a second convolutional neural network model. steps, including:

The feature maps are stacked and input into a preset second convolution module, and the second convolution module processes the feature maps based on a plurality of the convolution layers and the pooling layer and outputs the output. The new feature map of ;

Inputting the new feature map into the fully connected layer for feature classification to obtain a quality category of serum, wherein the quality category includes: normal, hemolysis, lipemia and jaundice;

The new feature map is input into the logistic regression layer to calculate the probability value of each of the quality classes, so as to determine the quality class corresponding to the hemolysis, the lipemia and the jaundice when the quality class is.

Optionally, the described neural network-based serum quality identification method also includes:

After assigning weights to the feature maps, and multiplying the feature maps by the assigned weights, the step of stacking the feature maps and inputting them into a preset second convolution module is performed.

Optionally, the step of extracting the matrix region belonging to the unlabeled region of the test tube from the multi-exposure test tube image includes:

Performing serum sample analysis on the multi-exposure test tube image, to calculate the highest position of serum liquid level and the lowest position of serum liquid level of the serum sample contained in the test tube in the multi-exposure test tube image;

According to the position of the highest level of the serum liquid level and the position of the lowest level of the serum liquid level, a matrix area of a preset size is extracted from the non-labeled area of the test tube.

Optionally, the method performs image acquisition on the test tube by using a preset industrial camera;

The step of extracting a matrix area with a preset size from the non-labeled area of the test tube according to the highest position of the serum level and the lowest position of the serum level includes:

Determine the serum image area from the non-label area of the test tube according to the highest position of the serum level and the lowest position of the serum level;

The matrix area is cut out from the serum image area according to the preset size, wherein the preset size is smaller than the size of the serum image area.

In addition, in order to achieve the above object, the present invention also provides a neural network-based serum quality identification device, the neural network-based serum quality identification device includes:

The test tube image acquisition module is used for image acquisition of the test tube containing the serum sample to obtain the test tube image;

A serum quality identification module, configured to input the test tube image into a preset neural network model, so that the neural network model outputs a serum quality identification result for the serum sample, wherein the neural network model passes the test tube image It is obtained by training the convolutional neural network model.

Wherein, the neural network-based serum quality identification device of the present invention implements the steps of the neural network-based serum quality identification method of the present invention as described above when each of the above-mentioned functional modules is run.

In addition, in order to achieve the above object, the present invention also provides a terminal device, the terminal device includes: a memory, a processor, and a neural network-based serum quality recognition system stored on the memory and running on the processor. A program, when the neural network-based serum quality identification program is executed by the processor, implements the steps of the neural network-based serum quality identification method of the present invention as described above.

In addition, in order to achieve the above object, the present invention also provides a computer-readable storage medium on which a neural network-based serum quality identification program is stored, and the neural network-based serum quality identification program is processed When the device is executed, the steps of the above-mentioned neural network-based serum quality identification method of the present invention are realized.

A method, device, terminal device, and computer-readable storage medium for serum quality identification based on a neural network proposed in the embodiment of the present invention, the test tube image is obtained by image acquisition of a test tube containing serum samples; the test tube image is input into a preset In the neural network model, the neural network model outputs the serum quality identification result for the serum sample, wherein the neural network model is obtained by performing convolutional neural network model training on test tube images.

Compared with the existing method of identifying and judging the quality of the serum in the test tube, the present invention preliminarily collects the test tube image of the test tube containing the serum, and uses the test tube image to train the convolutional neural network model to obtain the neural network model. In the actual serum quality detection process, it is only necessary to collect the image of the test tube containing the serum sample to obtain the test tube image, and then input the test tube image into the neural network model, and then the neural network model can be output based on the test tube image. Serum quality identification result of the serum sample obtained after calculation. In this way, the present invention can identify and judge the quality of the serum sample based on the collection of suitable test tube images. Compared with the way of manually judging the quality of serum with the naked eye, the efficiency of serum quality identification is greatly improved, and further. Using appropriate multi-exposure test tube images to train the neural network model effectively reduces the influence of the label on the test tube, thereby ensuring the accuracy of the model to identify serum quality.

Description of drawings

1 is a schematic structural diagram of a hardware operating environment of a terminal device involved in an embodiment of the present invention;

FIG. 2 is a schematic flowchart of a method for identifying serum quality based on a neural network according to an embodiment of the present invention, which involves the acquisition of test tube images;

3 is a schematic diagram of the system architecture involved in an embodiment of the neural network-based serum quality identification method of the present invention;

4 is a schematic diagram of an application scenario involved in an embodiment of the neural network-based serum quality identification method of the present invention;

5 is a schematic diagram of another application scenario involved in an embodiment of the neural network-based serum quality identification method of the present invention;

6 is a schematic diagram of another application scenario involved in an embodiment of the neural network-based serum quality identification method of the present invention;

7 is a schematic diagram of an application process involved in an embodiment of the neural network-based serum quality identification method of the present invention;

8 is a schematic diagram of a scene in which an embodiment of the neural network-based serum quality identification method of the present invention involves calculating the height of a test tube in a real-time image;

9 is a schematic diagram of another scenario in which an embodiment of the neural network-based serum quality identification method of the present invention involves calculating the height of a test tube in a real-time image;

10 is a schematic diagram of another scene in which an embodiment of the neural network-based serum quality identification method of the present invention involves calculating the height of a test tube in a real-time image;

11 is a binarized image involved in an embodiment of the neural network-based serum quality identification method of the present invention;

12 is an embodiment of the neural network-based serum quality identification method of the present invention involving a formula for calculating the area of a non-labeled area;

13 is a schematic diagram of an application scenario of a method for recognizing serum quality based on a neural network according to an embodiment of the present invention involving a preset angle of rotation of a test tube;

FIG. 14 is a schematic diagram of the requirements for labeling and pasting test tubes involved in an embodiment of the neural network-based serum quality identification method of the present invention;

15 is a schematic diagram of a scene involving calculating a preset angle of test tube rotation according to an embodiment of the neural network-based serum quality identification method of the present invention;

16 is a schematic flow chart of serum sample analysis according to an embodiment of the neural network-based serum quality identification method of the present invention;

FIG. 17 is a schematic diagram of an application scenario of an embodiment of the neural network-based serum quality identification method of the present invention involving the interception of an image ROI region;

18 is a schematic diagram of an application scenario involving determining the liquid level of a serum sample according to an embodiment of the neural network-based serum quality identification method of the present invention;

19 is a schematic diagram of a test tube with or without gel according to an embodiment of the neural network-based serum quality identification method of the present invention;

Figure 20 is the mean value of the HSV three channels involved in an embodiment of the neural network-based serum quality identification method of the present invention;

21 is a schematic diagram of an application scenario of determining the serum level through a single-pixel line segment involved in an embodiment of the neural network-based serum quality identification method of the present invention;

22 is a schematic flowchart of an embodiment of the neural network-based serum quality identification method of the present invention;

23 is a schematic structural diagram of a convolutional neural network model involved in an embodiment of the neural network-based serum quality identification method of the present invention;

24 is a schematic structural diagram of a first convolution module in a convolutional neural network model involved in an embodiment of the neural network-based serum quality identification method of the present invention;

25 is a schematic structural diagram of a second convolution module in a convolutional neural network model involved in an embodiment of the neural network-based serum quality identification method of the present invention;

Fig. 26 is a schematic diagram of the modules of the neural network-based serum quality identification device of the present invention.

The realization, functional characteristics and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments.

Detailed ways

It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

The main solution of the embodiment of the present invention is: image acquisition of a test tube containing serum samples to obtain a test tube image; inputting the test tube image into a preset neural network model, so that the neural network model can output the test tube image for the serum sample The serum quality identification result of , wherein, the neural network model is obtained by performing convolutional neural network model training on test tube images.

Since the previous quality detection and identification of serum samples to screen out abnormal samples were all conducted by experienced technicians through observation and judgment with the naked eye, not only did different technicians have different judgment standards, but also the overall efficiency of screening abnormal samples Also lower. In addition, in the prior art, serum quality can also be determined by extracting the color value of the liquid surface of the serum to be detected after dividing the respective color ranges for serum samples in normal, hemolytic, chyle, icteric and other states in advance. judge. However, this method is easily affected by the label affixed to the test tube containing serum, resulting in errors in color judgment.

The present invention provides a solution, in which the test tube image of the test tube containing serum is collected in advance, and the neural network model is obtained by using the test tube image to train the convolutional neural network model, so that in the actual serum quality detection process, only the The test tube containing the serum sample is imaged to obtain the test tube image, and then the test tube image is input into the neural network model, and the serum quality identification of the serum sample obtained after training and calculation based on the test tube image can be output based on the neural network model. result. In this way, the present invention can identify and judge the quality of the serum sample based on the collection of suitable test tube images. Compared with the way of manually judging the quality of serum with the naked eye, the efficiency of serum quality identification is greatly improved, and further. Appropriate multi-exposure test tube images are used to train the neural network model, and the color information of the images under different exposures is used to train the model, which effectively reduces the influence of the label background on the test tube, thus ensuring the accuracy of the model to identify serum quality.

As shown in FIG. 1 , FIG. 1 is a schematic diagram of a terminal structure of a hardware operating environment involved in an embodiment of the present invention.

The terminal device in the embodiment of the present invention may be various terminal devices configured to perform test tube image acquisition, such as a terminal server, a PC, or even a mobile terminal device such as a smartphone and a tablet computer, or an immovable terminal device .

As shown in FIG. 1 , the terminal device may include: a processor 1001 , such as a CPU, a network interface 1004 , a user interface 1003 , a memory 1005 , and a communication bus 1002 . Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. Optionally, the network interface 1004 may include a standard wired interface and a wireless interface (eg, a WI-FI interface). The memory 1005 may be high-speed RAM memory, or may be non-volatile memory, such as disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001 .

Those skilled in the art can understand that the terminal structure shown in FIG. 1 does not constitute a limitation on the terminal device, and may include more or less components than the one shown, or combine some components, or arrange different components.

As shown in FIG. 1 , the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module and a neural network-based serum quality identification program.

In the terminal device shown in FIG. 1 , the network interface 1004 is mainly used to connect to the background server and perform data communication with the background server; the user interface 1003 is mainly used to connect the client (client) and perform data communication with the client; and the processing The device 1001 can be used to invoke the neural network-based serum quality identification program stored in the memory 1005, and perform the following operations:

Image acquisition is performed on the test tube containing the serum sample to obtain the test tube image;

Inputting the test tube image into a preset neural network model for the neural network model to output the serum quality identification result for the serum sample, wherein the neural network model performs convolutional neural network model training through the test tube image get.

Further, the processor 1001 can be used to call the neural network-based serum quality identification program stored in the memory 1005, and perform the following operations:

acquiring a multi-exposure test tube image, and extracting a matrix area belonging to the non-label area of the test tube from the multi-exposure test tube image;

Input the matrix region into the preset first convolution module to train the first convolutional neural network model, and obtain the output of the first convolutional neural network model after the first convolution module performs the first convolutional neural network model training for the matrix region. feature map;

The feature maps are stacked and then input into a preset second convolution module to train a second convolutional neural network model to obtain a neural network model for serum quality identification for serum samples.

Further, the first convolution module and the second convolution module include: a convolution layer and a pooling layer, the first convolution module and the second convolution module except the last two convolution modules In addition to the layers, a pooling layer is connected after every two convolutional layers;

The convolutional layers in the first convolutional module include multiple first convolutional layers with stride 1 and multiple second convolutional layers with stride 2. In every two first convolutional layers, the output The first convolutional layer that is not connected to the pooling layer at the end is connected to one of the second convolutional layers;

The number of convolution kernels of the two first convolution layers at the end of the first convolution module is smaller than the number of convolution kernels of the other first convolution layers and the second convolution layer.

Further, the end of the second convolution module is connected to the fully connected layer and the logistic regression layer, and the processor 1001 can be used to call the neural network-based serum quality identification program stored in the memory 1005, and perform the following operations:

The feature maps are stacked and input into a preset second convolution module, and the second convolution module processes the feature maps based on a plurality of the convolution layers and the pooling layer and outputs the output. The new feature map of ;

Inputting the new feature map into the fully connected layer for feature classification to obtain a quality category of serum, wherein the quality category includes: normal, hemolysis, lipemia and jaundice;

The new feature map is input into the logistic regression layer to calculate the probability value of each of the quality classes, so as to determine the quality class corresponding to the hemolysis, the lipemia and the jaundice when the quality class is.

Further, the processor 1001 can be used to call the neural network-based serum quality identification program stored in the memory 1005, and perform the following operations:

After assigning weights to the feature maps, and multiplying the feature maps by the assigned weights, the step of stacking the feature maps and inputting them into a preset second convolution module is performed.

Further, the processor 1001 can be used to call the neural network-based serum quality identification program stored in the memory 1005, and perform the following operations:

Performing serum sample analysis on the multi-exposure test tube image, to calculate the highest position of serum liquid level and the lowest position of serum liquid level of the serum sample contained in the test tube in the multi-exposure test tube image;

According to the position of the highest level of the serum liquid level and the position of the lowest level of the serum liquid level, a matrix area of a preset size is extracted from the non-labeled area of the test tube.

Further, the processor 1001 can be used to call the neural network-based serum quality identification program stored in the memory 1005, and perform the following operations:

Determine the serum image area from the non-label area of the test tube according to the highest position of the serum level and the lowest position of the serum level;

The matrix area is cut out from the serum image area according to the preset size, wherein the preset size is smaller than the size of the serum image area.

Based on the above hardware structure, various embodiments of the neural network-based serum quality identification method of the present invention are proposed.

It should be noted that the neural network-based serum quality identification method of the present invention can be applied to the system architecture of image acquisition for test tubes as shown in FIG. A unit and a data processing unit, wherein the rotating part can be used to rotate the test tube from 0° to 360°, the photographing system is used for image acquisition of the test tube, and the control unit is used to control the rotating device and the photographing system in real time, and the data The processing unit is used for processing and storing the captured test tube images. Specifically, when the control unit specifically applies the image acquisition method for automatically rotating test tubes of the present invention to control the rotating component to perform the first rotation operation or the second rotating operation for the test tube, when it is detected that there is a test tube on the current tube rack, the control unit in the rotating component The gripping part grabs the cap end of the test tube, and then the image is captured by the shooting system and sent to the data processing unit for a series of image processing, such as: test tube positioning, non-label area calculation, threshold judgment, rotation angle calculation, etc. , after that, the control unit further performs corresponding operations according to the output results of the data processing unit, such as: controlling the rotating part to rotate the test tube to a predetermined angle to finally capture the largest image at the label gap on the test tube, or, for the test tube that does not meet the requirements. Images or test tubes for alerts and reminders.

Referring to FIG. 22 , a specific embodiment of the neural network-based serum quality identification method of the present invention is described with a terminal device instead of the system architecture shown in FIG. 3 for image acquisition for a test tube. In one embodiment of the neural network-based serum quality identification method of the present invention, the neural network-based serum quality identification method of the present invention includes:

Step S10, performing image acquisition on the test tube containing the serum sample to obtain a test tube image;

In this embodiment, when the terminal device actually performs real-time serum quality identification for the serum sample contained in the test tube whose serum quality is to be identified, the terminal device first performs image acquisition on the test tube to obtain an appropriate test tube image.

It should be noted that, in this embodiment, when the terminal device uses the industrial camera to collect the test tube image, the different exposure times of the industrial camera have a great influence on the color of the serum. Tube images of exposure time.

Specifically, for example, after the terminal device rotates the test tube containing the serum sample of the serum quality to be identified in the test tube, so that the midpoint of the non-label area of the test tube is facing the industrial camera, the exposure time T1 (the exposure time T1) is set. The color of serum/plasma photographed by an industrial camera is equal to the actual value under time), and the test tube image P1 with the largest area of the non-labeled area is obtained.

In addition, in the process of training the convolutional neural network model, the terminal device can also collect images of the test tube containing the serum sample according to the preset exposure time rule to obtain multiple test tube images.

Specifically, after rotating the test tube to satisfy the situation that the midpoint of the non-label area is facing the industrial camera, in addition to collecting the test tube image P1 with the exposure time T1, the terminal device further sets the exposure time T2 shorter than T1, and collects the image P2; And set the exposure time T3 longer than T1, and acquire the image P3.

Step S20, inputting the test tube image into a preset neural network model, so that the neural network model outputs the serum quality identification result for the serum sample, wherein the neural network model performs convolutional neural network by using the test tube image. The network model is trained.

In this embodiment, after acquiring an appropriate test tube image, the terminal device immediately inputs the test tube image into a neural network model that has been trained in advance by using the multi-exposure test tube images to train the convolutional neural network model, so that the neural network model is trained by the neural network. After the network model performs training calculation based on the appropriate test tube image, it outputs the serum quality identification result for the serum sample contained in the test tube corresponding to the test tube image.

Further, in a feasible embodiment, the neural network-based serum quality identification method of the present invention conducts convolutional neural network model training by using multiple exposure test tube images, that is, acquiring multiple exposure test tube images, and according to the multiple exposure test tube images. The exposed test tube images were subjected to convolutional neural network model training to obtain a neural network model for serum quality identification for serum samples.

In this embodiment, before the terminal device actually performs real-time serum quality identification for the serum samples contained in the test tubes to be identified, the terminal device also obtains the multi-exposure test-tube images, so as to use the multi-exposure test-tube images for scrolling The neural network model is trained to obtain the above-mentioned neural network model for real-time serum quality identification of serum samples in test tubes.

The present invention's method for identifying serum quality based on neural network can also include:

Step 30, acquiring a multi-exposure test tube image, and extracting a matrix area belonging to the non-label area of the test tube from the multi-exposure test tube image;

In this embodiment, in the process of using the collected multi-exposure test tube images to train the convolutional neural network model to obtain the above-mentioned neural network model, the terminal device first performs serum sample analysis on the multi-exposure test tube images to determine the corresponding After the serum index of , the matrix area belonging to the non-label area is extracted from the test tube image based on the serum index.

That is, the terminal device performs serum sample analysis on the multi-exposure test tube images in advance to calculate the highest position of the serum level and the lowest position of the serum level of the serum sample contained in the test tube; The lowest position of the serum level, in the multi-exposure test tube image, a matrix area of preset size is extracted from the unlabeled area of the test tube.

Further, in this embodiment, specifically, the serum index includes: the highest position of the serum liquid level and the lowest position of the serum liquid level of the serum sample. The above-mentioned steps of extracting a matrix area of a preset size from the unlabeled area of the test tube may include:

Determine the serum image area from the non-label area of the test tube according to the highest position of the serum level and the lowest position of the serum level;

The matrix area is cut out from the serum image area according to the preset size, wherein the preset size is smaller than the size of the serum image area.

It should be noted that, in this embodiment, the preset size is smaller than the size of the serum image area, for example, taking the pixel unit of the image as the basic unit, the specific size of the preset size may be 32*32. It should be understood that, based on different design requirements of practical applications, in different feasible embodiments, the specific size of the preset size can of course be set to be different from the specific size listed in this embodiment. The present invention is based on the serum quality of the neural network. The identification method is not limited to the specific size of the preset size, as long as the preset size is smaller than the size of the serum image area in the test tube image.

That is, the terminal device is specifically based on the highest position of the serum liquid level and the lowest position of the serum liquid level of the serum sample contained in the test tube obtained by analyzing the serum sample, in the multi-exposure test tube images P1, P2, and P3 collected above. , extract the matrix areas F1, F2 and F3 with a preset size of 32*32 at any position in the non-labeled area of the test tube, respectively.

Step S40, inputting the matrix region into a preset first convolution module to train the first convolutional neural network model, and obtaining the first convolutional module to train the first convolutional neural network model for the matrix region The feature map of the post output;

Step S50, the feature maps are stacked and then input into a preset second convolution module to train a second convolutional neural network model, so as to obtain a neural network model for serum quality identification for serum samples.

It should be noted that, in this embodiment, the first convolution module and the second convolution module include: a convolution layer and a pooling layer, the first convolution module and the second convolution module In addition to the two convolutional layers at the end of the module, a pooling layer is connected after every two convolutional layers;

The convolutional layers in the first convolutional module include multiple first convolutional layers with stride 1 and multiple second convolutional layers with stride 2. In every two first convolutional layers, the output The first convolutional layer that is not connected to the pooling layer at the end is connected to one of the second convolutional layers;

The number of convolution kernels of the two first convolution layers at the end of the first convolution module is smaller than the number of convolution kernels of the other first convolution layers and the second convolution layer.

Specifically, please refer to the convolutional neural network model shown in FIG. 23 , the first convolution module in the model shown in FIG. 24 , and the second convolution module in the model shown in FIG. 25 . The terminal device first inputs the extracted images of the three matrix regions F1, F2 and F3 into the first convolution module, which specifically includes 8 convolution layers and 2 pooling layers. The stride of the convolutional layers conv1 to conv6 is 1, and the output feature map is the same size as the input feature map, while the stride of the convolutional layers conv7 to conv8 is 2, and the output feature map is the input feature map. half. In the first convolution module, the convolution kernels of the convolutional layers conv5 and conv6 are 20, and the number of convolution kernels of the other convolutional layers is 30 (it should be understood that the size of the convolution kernel can be 5×5 or 3 ×3, the specific number of convolution kernels can be adjusted according to the actual situation of the network). In the first convolution module, every two convolutional layers is followed by a 2×2 maximum pooling layer. After the feature map passes through the pooling layer, its feature map is half of the input feature map. In addition, the convolutional layers Conv1 and conV3 are also connected to the convolutional layers Conv7 and Conv8, respectively, and the resulting feature map and the feature map after the pooling operation are combined and input to the next convolutional layer operation. In addition, the second convolution module consists of 4 convolutional layers and 1 max-pooling layer, and the convolution kernel size of each convolutional layer is 60. The size of the feature map output by the second convolution module is 4×4×60. In the second convolution module, each convolutional layer is followed by a BN layer and a relu layer, and the relu layer is used to increase the nonlinear expression ability of the network.

Further, in the above step S50, "after stacking the feature maps and inputting the preset second convolution module to train the second convolutional neural network model" may include:

The feature maps are stacked and input into a preset second convolution module, and the second convolution module processes the feature maps based on a plurality of the convolution layers and the pooling layer and outputs the output. The new feature map of ;

Inputting the new feature map into the fully connected layer for feature classification to obtain a quality category of serum, wherein the quality category includes: normal, hemolysis, lipemia and jaundice;

The new feature map is input into the logistic regression layer to calculate the probability value of each of the quality classes, so as to determine the quality class corresponding to the hemolysis, the lipemia and the jaundice when the quality class is.

Specifically, please refer to the convolutional neural network model shown in Figure 23. The above-mentioned three matrix regions F1, F2 and F3 images pass through the first convolution module to output a feature map with a size of 8×8×60, and the feature map will be further stacked along the channel direction, and then stacked to get The feature map with a size of 8×8×60 is further input into the second convolution module to train the second convolutional neural network model to obtain a new feature map.

In addition, in this embodiment, the last part of the above-mentioned convolutional neural network model is the fully connected layer and the logistic regression layer: the softmax layer, and the new feature map output after the second convolution module will pass through the fully connected layer respectively. Perform classification to obtain multiple quality categories of serum samples, such as: normal, hemolytic, lipemic, icteric, and calculate the respective probability values of the respective quality categories through the softmax layer to determine the respective categories when the quality categories are hemolysis, lipemia and icterus the corresponding quality level.

It should be noted that, in this embodiment, the fully connected layer is used to classify the features. In order to prevent the over-fitting of the image, a dropout operation is introduced here to randomly delete some neurons in the convolutional neural network. In addition, local normalization and data augmentation can be performed to increase robustness. The softmax layer is the final processing step of the entire network model, suitable for solving multi-classification problems, and its output is the probability value of each classification category. In this network, serum can be divided into four categories: normal, hemolysis, lipidemia, and jaundice, and can be further subdivided, such as hemolysis grade 1, hemolysis grade 2, hemolysis grade N; jaundice grade 1, jaundice grade 2 , jaundice grade N; lipid blood grade 1, lipid blood grade 2, lipid blood grade 3, etc.

The terminal device uses thousands or tens of thousands of the above-mentioned multi-exposure test tube images to train the convolutional neural network model, so as to obtain the above-mentioned neural network model trained for real-time serum quality identification of serum samples.

Further, in a feasible embodiment, the neural network-based serum quality identification method of the present invention may also include:

After assigning weights to the feature maps, and multiplying the feature maps by the assigned weights, the step of stacking the feature maps and inputting them into a preset second convolution module is performed.

It should be noted that, in this embodiment, since its image P1 is closer to the color seen by the real naked eye than P2 and P3, the weight can be further allocated before stacking. For example, the weight of P1 is 2, The weights of P2 and P3 are both 1. Therefore, before stacking the feature maps above, the feature map and the weight are multiplied, and then the memory stack is input into the second convolution module for the second convolutional neural network model. training process.

In this embodiment, compared with the existing method of identifying and judging the quality of the serum in the test tube, the present invention collects the multi-exposure test tube image of the test tube containing serum in advance, so as to use the test tube image to carry out the convolutional neural network model. The neural network model is obtained by training, so that in the actual serum quality detection process, only the image of the test tube containing the serum sample is collected to obtain the test tube image, and then the test tube image is input into the neural network model. The model outputs the serum quality identification result of the serum sample obtained after training and calculation based on the test tube image. In this way, the present invention can identify and judge the quality of the serum sample based on the collection of suitable test tube images. Compared with the way of manually judging the quality of serum with the naked eye, the efficiency of serum quality identification is greatly improved, and further. Using appropriate multi-exposure test tube images to train the neural network model can effectively avoid the influence of labels on the test tube, thus ensuring the accuracy of the model to identify serum quality.

Further, please refer to FIG. 16 , in an embodiment of the test tube image-based serum sample analysis method of the present invention, the terminal device performs serum sample analysis on the serum sample contained in the test tube, that is, the above-mentioned "multi-exposure test tube". Image for serum sample analysis" steps, which can include:

In step a, the multi-exposure test tube images are separated in the RGB color space, and from the separated channel images and the multi-exposure test tube images, a region of interest ROI is determined respectively;

In this embodiment, the terminal device further separates the test tube image into corresponding three-channel images in the RGB color space after collecting the image of the test tube containing the serum sample to obtain the above-mentioned test tube image that meets the requirements, and further extracts the three-channel image from the three-channel image. In the channel image and the original test tube image, the corresponding region of interest ROI is determined respectively.

Further, in a feasible embodiment, the separated images of each channel include: a B channel image and an R channel image, and the above step a may include:

Step a1, according to each coordinate of the preset image area, intercept a first matrix area within each of the coordinates from the B channel image, and determine the first matrix area as a region of interest ROI-B, wherein, The preset image area belongs to the non-label area in the test tube image;

Step a2, intercepting the second matrix region within each of the coordinates from the R channel image according to each of the coordinates, and determining the second matrix region as a region of interest ROI-R;

Step a3, intercepting a third matrix region within each of the coordinates from the test tube image according to each of the coordinates, and determining the third matrix region as a region of interest ROI-P.

Specifically, for example, referring to the application scenario shown in Figure 17, the terminal device separates the collected test tube images into P1_R, P1_G, and P1_B three-channel images in the RGB color space, and intercepts part of each image respectively, That is, intercept the matrix area within four points of coordinates (0, H-10), (0, H+10) (M-10, H-10), (M-10, H+10), and set the The intercepted matrix region is set as the corresponding region of interest ROI. For example, the part of the region within the above four coordinate points is intercepted from the B channel image, the R channel image, and the original test tube image P1, respectively, and named ROI_B, ROI_R and ROI_P1 as the corresponding regions of interest ROI.

Step b, analyzing the sample index corresponding to the serum sample according to each ROI of the region of interest, wherein the sample index corresponding to the serum sample includes: the highest position of the serum liquid level and the lowest position of the serum liquid level. Location.

In this embodiment, after the terminal device determines the ROI corresponding to the region of interest from the three-channel image and the original test tube image, respectively, it further uses the sample index corresponding to the ROI of the region of interest for analysis to calculate The serum level position of the serum sample contained in the test tube, and the serum amount is further calculated based on the serum level position.

It should be noted that, in this embodiment, after the terminal device analyzes and calculates the highest position and lowest position of the serum level in the test tube, it can directly calculate it based on the ordinary volume calculation method combined with the known circumference of the test tube. Get the serum volume.

Further, in a feasible embodiment, the sample index corresponding to the serum sample includes: the highest position of the serum liquid level, the highest position of the blood clot liquid level, and the lowest position of the serum liquid level; the above step b, may also include:

Step b1, calculate the highest position of the serum liquid level according to the region of interest ROI-B;

Step b2, calculating the highest position of the blood clot liquid level according to the region of interest ROI-R;

In this embodiment, the terminal device analyzes and calculates the position of the highest level of the serum level of the serum sample in the test tube by using the region of interest ROI_B corresponding to the B channel image determined in the above process, and by using the R determined in the above process The region of interest ROI_R corresponding to the channel image is used to analyze and calculate the position of the highest level of the blood clot in the serum sample in the test tube.

Specifically, for example, referring to the application scenario shown in Figure 18, the terminal device performs operations such as averaging or median on ROI_B in the row direction to turn the ROI_B matrix block into a single-pixel line, so as to find the turning point of the line segment, which is the serum The position of the highest liquid level. Similarly, the terminal device performs operations such as averaging or median on the ROI_R in the row direction to convert the ROI_R matrix block into a single-pixel line, so as to find the turning point of the line segment, which is the highest position of the blood clot level.

Step b3: Detecting whether there is gel in the test tube according to the region of interest ROI-P to obtain a detection result, and determining the position of the lowest level of the serum liquid level according to the detection result.

In this embodiment, the terminal device determines whether there is gel in the test tube containing the serum sample through the region of interest ROI-P corresponding to the original test tube image determined in the above process, so as to determine whether there is gel or not based on the presence of gel or the absence of gel. The detection result is used to determine the lowest position of the serum level of the serum sample in the test tube.

Further, in a feasible embodiment, the above step b3 may further include:

After transforming the region of interest ROI-P from the RGB color space to a preset color space, take the average value of the region of interest ROI-P along the horizontal direction, so as to determine whether there is a presence in the test tube through the average value The gel obtained the corresponding detection result.

In this embodiment, after the terminal device has determined the highest position of the serum liquid level and the highest position of the blood clot liquid level of the serum sample contained in the test tube, the terminal device further determines whether there is gel in the test tube to determine the serum level of the serum sample. The lowest position of the serum level.

Specifically, please refer to the test tube with gel shown in the left image of FIG. 19 and the test tube without gel shown in the right image. The terminal device transforms the ROI_P image of the region of interest corresponding to the original test tube image P1 from the RGB color space to the HSV color space (of course, it can also be converted to other color spaces based on different design needs of practical applications, such as RGB, YUV, HIS, etc.), Then the average or median value of the matrix of the region of interest ROI_P is taken along the horizontal direction to obtain the average value of the three HSV channels of the matrix as shown in FIG. 20 , wherein whether there is gel in the blood sample can be clearly observed from the second channel.

That is, in the application scenario shown in Figure 21, L1 is the highest position of the liquid level determined by the single-pixel line segment of ROI_B, L2 is the highest position of the blood clot liquid level determined by the single-pixel line segment of ROI_R, and the difference between L1 and L2 If the original test tube image P1 is a test tube without gel as shown on the left side of Figure 21, there should be no obvious layering corresponding to the middle line segment. On the contrary, if the test tube image P1 is as shown in Figure 19 For the test tube containing the gel shown on the right, it can be clearly seen that there will be two distinct segments in the middle line segment, wherein the lower pixel part on the right is the separating agent area, and the higher pixel on the left is the serum area.

Finally, the terminal device searches for the turning point of the middle line segment by traversing the points of the middle line segment between L1 and L2, wherein the turning point is the point that maximizes the total pixel difference between the left and right ends of the middle line segment, and judges the two ends Whether the pixel difference is greater than the threshold X determines the position of the lowest level of the serum level. That is, if the difference is greater than the threshold X, the test tube contains gel, and the lowest position of the serum level (if it is L3) is the turning point; on the contrary, if the difference is less than the threshold X, the test tube does not contain gel Therefore, the lowest level position L3 of the serum level is the highest level position L2 of the blood clot level.

It should be noted that, in this embodiment, the above-mentioned threshold X selection process is: taking multiple serum samples, collecting and obtaining the pixel average value A1 in the serum area of each sample and the pixel average value A2 in the sample gel area, then the threshold value X=| A1-A2|.

In this embodiment, when the terminal device analyzes the serum sample contained in the test tube and needs to calculate the serum level and the serum volume, it first analyzes the test tube containing the serum sample through a preset industrial camera or other image capturing device. image acquisition operation to obtain the optimal test tube image that meets the requirements; after that, the test tube image is further separated into corresponding three-channel images in the RGB color space, and further determined from the three-channel image and the original test tube image. Finally, use the sample index corresponding to the region of interest ROI for analysis to calculate the serum level position of the serum sample contained in the test tube, and further calculate the serum level based on the serum level position. amount, etc.

Compared with the existing method of analyzing the serum samples contained in the test tubes, the present invention performs the corresponding serum sample analysis by first separating the collected test tube images and then extracting the ROI corresponding to the region of interest, which can avoid the test tube labels to the images. Influence of color, thus ensuring the accuracy of analyzing and calculating the blood sample level and serum volume, further avoiding the waste of human resources caused by manual visual observation, and effectively improving the efficiency of serum sample analysis based on the collection of test tube images. .

Further, please refer to FIG. 2 , in an embodiment of the neural network-based serum quality identification method of the present invention, the test tube image obtained by performing image collection on the test tube containing the serum sample may include:

Step i, for the test tube of the image to be collected, count the number of pixel values of the non-label area of the test tube in the real-time test tube image to determine the area of the non-label area;

In this embodiment, in the process of collecting images for the test tubes, the terminal device firstly counts the non-labeled areas on the test tubes that are not covered by labels or barcodes for the test tubes that need to be image collected, and then collects the test tubes by the industrial camera. The number of pixel values in the real-time test tube image, so as to determine the non-label area area of the non-label area in the real-time test tube image.

Specifically, for example, referring to the application process shown in Figure 7, when the terminal device performs image acquisition for any test tube, it first uses a preset industrial camera to photograph the test tube to obtain a real-time test tube image of the test tube in its initial state, and then, Then, data analysis is performed on the captured real-time test tube image through the data processing unit in the system architecture. That is, as shown in FIG. 8, the terminal device takes the upper left corner of the real-time test tube image as the coordinate origin, the vertical direction of the test tube in the image as the x-axis, and the height direction of the test tube in the image as the y-axis. A coordinate system is established in the image, and then the image is processed to obtain the height of the test tube. Finally, use the calculation process shown in Figure 12 to convert the image from RGB space to HSV and other color spaces, so as to intercept 50 pixels in the middle area of the S channel test tube based on the determined test tube height as the judgment matrix, that is, take H/2-25 A matrix block with 50 pixel values between H/2+25, H means the height of the test tube, divide each pixel value of the image in the matrix, and count the number of pixel values in the non-label area in the matrix block to get The area of the non-labeled area in the current real-time tube image as the non-labeled area on the test tube.

It should be noted that, in this embodiment, it should be understood that the value of 50 pixels intercepted by the terminal device is not a unique value for the judgment matrix. Based on different design requirements of practical applications, in different feasible real-time methods, the terminal device is of course The number of intercepted pixel values can be adjusted according to the size of the real-time test tube image actually captured.

In addition, please refer to FIG. 4, since in the real-time test tube image, the background and the label area are relatively dark, while the non-label area (or also called the label gap area) is relatively bright. Therefore, the terminal device can specifically set the pixel threshold by setting the pixel threshold. A, the area with pixels larger than the threshold A in the image is judged as a non-label area, and the area smaller than the threshold A is judged as the background and label area.

It should be noted that, in this embodiment, after the terminal device establishes the coordinate system in the real-time test tube image, the position of the uppermost end of the test tube is fixed immediately, so it is only necessary to locate the lowest position of the test tube in the image to obtain Test tube height. It should be understood that, based on different design requirements of practical applications, in different feasible implementation manners, the terminal device can locate and find the base mark of the test tube through various methods, such as: first, directly binarize the B channel image Or by detecting the edge of the image (specifically, detecting the edge of the test tube through edge detection operators such as Sobel, Roberts, Prewitt, Canny, etc.), and then performing morphological processing to separate the test tube area and the background area, and finally, by moving along the test tube Find the coordinates (H, M) of the bottom end of the test tube from the pixel value of the statistical image in the height direction.

Specifically, the terminal device can use the B channel image to locate the coordinates of the bottom of the test tube. That is, the terminal device separates the real-time test tube image into three channels, R, G, and B, and performs binarization and morphological processing on the B channel image. Or the background area is white. Afterwards, as shown in Figure 9, the terminal device superimposes or averages the processed images along the line direction, so that the image changes from a matrix to a single-pixel line, and searches upwards from the bottom of the image to The first position that is not 0 is found to be the height H of the test tube. In addition, in the scenario shown in Figure 10, the terminal device can also intercept a small part of the image above the H point (for example, select 30 pixel values) to average or superimpose the column direction, and then extract from The first point that is not 0 from left to right is taken as x1, and the first point that is not 0 from right to left is taken as x2, and the mean value is the central area M of the test tube in the image at this time.

Further, in another feasible embodiment, the terminal device may specifically use an edge detection algorithm to locate the base marker of the test tube in the image.

Specifically, the terminal device selects a filter of an appropriate size according to the size of the real-time test tube image in advance, and then uses the Canny operator to perform edge detection on the image to obtain the contour image of the test tube in the image; then, use morphological processing (such as closing operation) , the two-pass scanning method is used to process the edge image and binarize the image to obtain a binary image in which the edge part is white and the rest part is black as shown in Figure 11. Finally, the pixel value of the image is counted along the height direction of the test tube through the above way to find the coordinates (H, M) of the bottom end of the test tube.

Step ii, detecting whether the area of the non-label area satisfies the preset rotation condition, and when it is detected, collecting an image after performing the first rotation operation for the test tube;

It should be noted that, in this embodiment, the preset rotation condition is that the area of the non-label area of the test tube is 0 at this time, and the first rotation operation is to rotate the test tube by 180°.

In this embodiment, the terminal device detects whether the area is 0 after the non-label area of the test tube in the current real-time test tube image is obtained by statistics, and when it detects that the area is 0, immediately rotates the test tube by 180° , and then perform image acquisition for the test tube.

Specifically, for example, referring to the application process shown in FIG. 7 , when the terminal device statistically obtains that the area of the non-labeled area of the test tube in the current real-time test tube image is 0, it first performs a rotation operation on the test tube to rotate the test tube by 180°, and then When the process of the above step S10 is repeatedly performed and it is calculated that the area of the new non-label area is greater than or equal to the preset area threshold, image acquisition is performed for the test tube to obtain a test tube image that meets the requirements for sample detection.

Further, in a feasible embodiment, the test tube image-based serum sample analysis method of the present invention may further include:

Step A, outputting a preset first alarm prompt for the test tube after performing the first rotation operation on the test tube;

It should be noted that, in this embodiment, the preset first alarm prompt is a prompt prompting the user that all angles of the test tube are covered by labels.

In this embodiment, after the terminal device performs the first rotation operation to rotate the test tube by 180°, if the terminal device further collects statistics on the new non-label area of the test tube in the new real-time test tube image, the area is still 0 , the terminal device outputs a preset first alarm prompt for the test tube to prompt the user that all angles of the test tube are covered by labels.

Further, in a feasible embodiment, the above step A may include:

Step A1, after performing the first rotation operation on the test tube, determine the new non-label area area of the test tube;

Step A2, detecting whether the area of the new non-label area satisfies the preset rotation condition, and outputting a preset first alarm prompt for the test tube when it is detected.

Referring to the application process shown in FIG. 7 , in this embodiment, after the terminal device performs the rotation operation on the test tube to rotate the test tube by 180°, the terminal device repeats the process of the above step S10 to count the new non-label area area, and then, If the terminal device further detects that the area of the new non-label area is still 0, the terminal device immediately outputs a preset first alarm prompt for the test tube to the user to remind the user that all angles of the test tube are covered by labels.

Step iii, when detecting No, detect the size relationship between the area of the non-labeled area and the preset area threshold to obtain a detection result;

In this embodiment, when the terminal device detects whether the area of the non-label area of the test tube in the current real-time test tube image is 0, and thus detects that the area is not 0, the terminal device further detects the difference between the area of the non-label area and the preset area threshold. The size relationship between the two, so as to obtain the detection result that the area of the non-label area is greater than the area threshold, or obtain the detection result that the area of the non-label area is smaller than the area threshold.

Specifically, for example, referring to the application process shown in FIG. 7 , when the terminal device statistically obtains that the area of the non-label area of the test tube in the current real-time test tube image is not 0, the terminal device further compares the area of the non-label area with the preset area threshold. A comparison is made to determine whether the non-label area area is greater than the area threshold or less than the area threshold.

It should be noted that, in this example, the terminal device pre-sets the area threshold based on the process in which the unlabeled area of the test tube is capturing images of the industrial camera. Further, in a feasible embodiment, the test tube image-based serum sample analysis method of the present invention may further include:

Step 1, using a preset industrial camera to collect a test tube image in which the non-labeled area of the test tube is facing the industrial camera;

Step 2, converting the test tube image from RGB space to HSV color space and extracting the matrix block in the middle area of S channel;

Step 3: Count the number of pixel values of the non-label area in the matrix block to determine the area threshold of the area of the non-label area of the test tube.

In this embodiment, the terminal device preliminarily obtains the test tube image of the non-labeled area of the test tube facing the industrial camera based on the industrial camera, and then converts the test tube image from the RGB space to the HSV color space, so as to extract the middle of the S channel A matrix block of pixel values in the region. Finally, the terminal device counts the number of pixel values of the non-label area in the image in the matrix block to determine when the test tube is subjected to sample detection, the non-label area of the test tube is the smallest. area threshold.

It should be noted that, in this embodiment, before judging whether the non-label area meets the sample detection requirements in the collected test tube image, the minimum area threshold of the area of the non-label area needs to be given first for the test tube. Therefore, if the area of the unlabeled area of the test tube counted by the terminal device in the real-time test tube image is greater than or equal to the area threshold, it can be determined that the test tube image obtained by the current image acquisition for the test tube is the unlabeled area that meets the requirements of the device for sample detection.

Please refer to the application scenario shown in Figure 4. After the terminal device converts the RGB space of the real-time test tube image captured by the industrial camera into the HSV color space, the non-labeled area and label of the test tube in the image can be clearly observed in the S channel. The difference between the regions, that is, the background and white label regions in the image are dark black in the S channel, while the color of the non-label regions is brighter. Therefore, the terminal device determines which positions of the test tubes are labeled according to the gray level of the pixel values in the image.

In addition, please refer to the application scenario shown in Figure 5. When the serum sample contained in the test tube contains blood clots (such as serum after centrifugation), the pixel value of the blood clot in the above S channel is darker, so the terminal equipment It is possible that when calculating the area of the non-labeled area of the test tube, the image area where the blood clot is located will be divided into the labeling area of the test tube, resulting in an error in identification. However, usually in serum samples containing gel, the blood clot is not higher than 1/2 of the overall liquid level. Based on this, in order to obtain the minimum area threshold for the accurate non-label area, the terminal device detects the height of the test tube by detecting the height of the test tube. , and then intercept the middle fixed area of the test tube to detect the area of the non-labeled area of the test tube.

Specifically, please refer to the application scenario shown in Figure 6. The terminal device uses an industrial camera to take an image of the test tube facing the camera at the non-labeled area of the test tube (specifically, it can be obtained by manually adjusting the angle of the test tube by the user), and take the S channel. A matrix block with a height between H/2-25 and H/2+25 in the middle area, after that, count the number of pixel values C in the non-label area of the test tube in this matrix block area, and use this number of pixel values as the area threshold V =C-α, α is the allowable number of deviations.

In step iiii, according to the detection result, an image is directly collected for the test tube or an image is collected after a second rotation operation is performed for the test tube.

In this embodiment, the terminal device detects the size relationship between the area of the non-label area and a preset area threshold, thereby obtaining a detection result that the area of the non-label area is greater than the area threshold, or obtains the area of the non-label area After the detection result is smaller than the area threshold, the terminal device further based on the different detection results, the corresponding direct image acquisition for the test tube or the second rotation operation for the test tube is performed first to rotate the test tube by a preset angle, and then the test tube is rotated by a preset angle. Perform image acquisition.

Further, in a feasible embodiment, the above-mentioned step iiii may include:

Step i1, when the detection result is that the area of the non-label area is greater than or equal to the area threshold, directly collect an image of the test tube through the industrial camera;

In this embodiment, please refer to the application process shown in FIG. 7 , assuming that the real-time test tube image currently collected by the terminal device is specifically the rightmost image in FIG. 13 , the terminal device is detecting the current real-time test tube image in the test tube image. Whether the area of the non-label area is 0, so that when it is detected that the area is not 0, the size relationship between the area of the non-label area and the preset area threshold is further detected, and the area of the non-label area greater than the area threshold is obtained. When the test result is detected, the terminal device determines that the currently collected image has met the requirements of the device for sample detection, so that image acquisition can be performed directly without rotating the test tube (specifically, the current real-time test tube image can be directly used as the best test tube image). Tube images are stored for subsequent sample testing by the device).

Step i2, when the detection result is that the area of the non-label area is smaller than the area threshold, the second rotation operation is performed on the test tube to rotate the midpoint of the non-label area to the area of the industrial camera. After being directly opposite, an image of the test tube is captured by the industrial camera.

In this embodiment, please refer to the application process shown in FIG. 7 , when the terminal device detects whether the area of the non-labeled area of the test tube in the current real-time test tube image is 0, and thus detects that the area is not 0, it further detects the non-labeled area of the test tube. When the size relationship between the area of the label area and the preset area threshold is obtained, and the detection result that the area of the non-label area is smaller than the area threshold is obtained, the terminal device further performs a second rotation operation on the test tube to rotate the test tube by the preset value. angle, so that the area of the new non-labeled area of the test tube meets the requirements of the device for sample detection, and then image acquisition is performed on the test tube to obtain the best test tube image for the device to perform subsequent sample detection.

Further, in a feasible embodiment, in the above step i2, "the second rotation operation is performed on the test tube to rotate the midpoint of the non-label area to the direct opposite of the industrial camera". steps, which can include:

Step i21, dividing the first area in the non-label area currently collected by the industrial camera into a left area and a right area in half;

Step i2, count the number of pixel values of the left area and the right area respectively to determine the area of the left area and the area of the right area, and detect the difference between the area of the left area and the area of the right area. the size relationship between

Step i3, if it is detected that the area of the left area is larger than the area of the right area, rotate the test tube to the right by a preset angle to rotate the midpoint of the non-label area to the front of the industrial camera ;or,

Step i4, if it is detected that the area of the right side area is larger than the area of the left side area, rotate the test tube to the left by a preset angle to rotate the midpoint to the opposite side of the industrial camera.

In this embodiment, please refer to the application scenario shown in Figure 7, if the terminal device detects that the area of the non-labeled area of the test tube is not 0 is smaller than the preset area threshold, the terminal device determines that it has found the non-labeled area on the test tube. The labeling area, so that the midpoint of the non-labeling area is rotated to the direct opposite of the industrial camera by rotating a preset angle for the test tube, that is, the terminal device counts the central area of the matrix block of the non-labeling area of the test tube in the current real-time test tube image At M, the matrix block is divided to the left and right to obtain the left area and the right area. After that, the terminal device counts the number of pixel values in the unlabeled area of the test tube in the left area to obtain the area of the left area, and counts the right area. The number of pixel values in the unlabeled area of the area test tube gives the area on the right side.

Then, the terminal device detects the size relationship between the area on the left side and the area on the right side. If the area on the left side is larger than the area on the right side (if the real-time test tube image currently collected by the terminal device is as shown in the image in the middle of Figure 13 ) , then the area of the left area is larger than the area of the right area), then the terminal device determines that the midpoint of the non-labeled area on the test tube is on the left side of the test tube, so that the second rotation operation is performed on the test tube to rotate the test tube to the right by a preset angle N, so that the midpoint is facing the industrial camera.

Or, if the terminal device detects that the area of the left area is smaller than the area of the right area (if the real-time test tube image currently collected by the terminal device is as shown in the rightmost image in Figure 13, the area of the left area is smaller than the area of the right area), The terminal device determines that the midpoint of the non-labeled area on the test tube is on the right side of the test tube, and performs a second rotation operation for the test tube to rotate the test tube to the left by a preset angle N, so that the midpoint faces the industrial camera.

It should be noted that, in this embodiment, the preset angle is calculated based on the diameter of the test tube and the angle at the label gap of the test tube in the real-time test tube image currently collected by the industrial camera, wherein the The angle at the label gap is calculated based on the label area of the label on the test tube and the perimeter of the test tube.

Specifically, please refer to the application scenario shown in FIG. 15 , when the terminal device controls the test tube to rotate to the left or right by a preset angle N, based on the pre-collected diameter d and perimeter C of the test tube, and the label affixed on the test tube Or the area size M of the barcode can be calculated in real time according to the following formula 1 to obtain the preset angle N:

Formula 1:

为试管在实时试管图像中非标签区域的角度，该角度

通过如下公式2计算得到：in,

is the angle of the non-labeled area of the tube in the live tube image, the angle

It is calculated by the following formula 2:

Formula 2:

Further, in a feasible embodiment, the test tube image-based serum sample analysis method of the present invention may further include:

Step B, outputting a preset second alarm prompt for the test tube after the second rotation operation is performed on the test tube;

It should be noted that, in this embodiment, the preset second alarm prompt is a prompt prompting the user that the label or barcode on the test tube does not meet the sample detection requirements.

In this embodiment, after the terminal device performs the second rotation operation to rotate the test tube to a preset angle, if the terminal device further counts the new non-label area of the test tube in the new real-time test tube image, although the area is not If it is 0 but still smaller than the area threshold, the terminal device outputs a preset second alarm prompt for the test tube to remind the user that the label or barcode on the test tube does not meet the sample detection requirements.

Further, in a feasible embodiment, the above step B may include:

Step B1, after performing the second rotation operation on the test tube, determine the new non-label area area of the test tube;

Step B2: Detect the size relationship between the area of the new non-label area and the area threshold, and when it is detected that the area of the new non-label area is smaller than the area threshold, output a preset number for the test tube. 2. Warning prompt.

Please refer to the application process shown in FIG. 7 , in this embodiment, after the terminal device performs a rotation operation on the test tube to rotate the test tube by a preset angle N, it repeats the process of the above step S10 to count the new non-label area area, Then, if the terminal device further detects the size relationship between the area of the new non-labeled area and the area threshold, if the terminal device still detects that the area of the non-labeled area is smaller than the area threshold, the terminal device determines that the current The test tube is a test tube that does not meet the requirements in the non-label area (as shown in Figure 14, the left side is a test tube with a label or barcode that meets the requirements, and the right side is a test tube with a label or barcode that does not meet the requirements). Therefore, the terminal device immediately A preset second warning prompt is output to the user for the test tube to remind the user that the label or barcode on the test tube does not meet the sample detection requirements.

In this embodiment, in this embodiment, in the process of image acquisition for the test tube, the terminal device firstly counts the non-labeled areas on the test tube that are not covered by labels or barcodes for the test tubes that need to be imaged, and then The number of pixel values in the real-time test tube image collected by the industrial camera for the test tube, so as to determine the non-label area area of the non-label area in the real-time test tube image; After labeling the area of the area, check whether the area is 0, and when it is detected that the area is 0, immediately rotate the test tube by 180°, and then perform image acquisition on the test tube; or, the terminal device is detecting the current real-time test tube image. When the area of the non-label area of the test tube is not 0, further detect the size relationship between the area of the non-label area and the preset area threshold, so as to obtain the detection result that the area of the non-label area is greater than the area threshold, or, obtain The detection result that the area of the non-labeled area is smaller than the area threshold value; the terminal device is further based on the different detection results, and the corresponding direct image acquisition of the test tube or the second rotation operation for the test tube is performed to rotate the test tube by a preset angle. Image acquisition was performed for this tube.

Compared with the existing method of image acquisition for the test tube, the present invention firstly determines the non-label area area of the test tube in the real-time image, and then performs the first rotation operation for the test tube according to the non-label area area. The size relationship between the area of the non-labeled area and a preset area threshold is used to collect an image after performing the second rotation operation on the test tube. In this way, the present invention can collect test tube images that meet the requirements of equipment for sample detection only based on shooting 1-3 times, which greatly shortens the time for collecting images and avoids the occupation of storage resources due to shooting a large amount of image data. , and can also accurately collect the best image of the non-label area of the test tube facing the camera based on the rotation operation, which effectively improves the overall collection efficiency of the test tube image.

In addition, please refer to FIG. 26 , the embodiment of the present invention also proposes a neural network-based serum quality identification device, and the neural network-based serum quality identification device of the present invention includes:

The test tube image acquisition module 10 is used for image acquisition of the test tube containing the serum sample to obtain the test tube image;

The serum quality identification module 20 is used to input the test tube image into a preset neural network model, so that the neural network model outputs the serum quality identification result for the serum sample, wherein the neural network model passes through the test tube The image is obtained by training a convolutional neural network model.

Preferably, the neural network-based serum quality identification device of the present invention further includes:

The model training module is used to train the convolutional neural network model through multi-exposure test tube images;

Model training modules, including:

an extraction unit, configured to extract a matrix area belonging to the unlabeled area of the test tube from the multi-exposure test tube image;

A first convolution unit, configured to input the matrix region into a preset first convolution module for training a first convolutional neural network model, and obtain the first convolution module for the matrix region to perform the first volume The feature map output by the product neural network model after training;

The second convolution unit is configured to stack the feature maps and then input them into a preset second convolution module to train a second convolutional neural network model to obtain a neural network model for serum quality identification for serum samples.

Preferably, the first convolution module and the second convolution module include: a convolution layer and a pooling layer, the first convolution module and the second convolution module except the last two convolution modules In addition to the layers, a pooling layer is connected after every two convolutional layers;

The convolutional layers in the first convolutional module include multiple first convolutional layers with stride 1 and multiple second convolutional layers with stride 2. In every two first convolutional layers, the output The first convolutional layer that is not connected to the pooling layer at the end is connected to one of the second convolutional layers;

The number of convolution kernels of the two first convolution layers at the end of the first convolution module is smaller than the number of convolution kernels of the other first convolution layers and the second convolution layer.

Preferably, the end of the second convolution module is connected to the fully connected layer and the logistic regression layer, and the second convolution unit is also used for:

The feature maps are stacked and input into a preset second convolution module, and the second convolution module processes the feature maps based on a plurality of the convolution layers and the pooling layer and outputs the output. The new feature map of ;

Inputting the new feature map into the fully connected layer for feature classification to obtain a quality category of serum, wherein the quality category includes: normal, hemolysis, lipemia and jaundice;

The new feature map is input into the logistic regression layer to calculate the probability value of each of the quality classes, so as to determine the quality class corresponding to the hemolysis, the lipemia and the jaundice when the quality class is.

Preferably, the model training module of the neural network-based serum quality identification device of the present invention is further configured to assign weights to the feature maps, and after multiplying the feature maps by the assigned weights, execute the The step of stacking the feature maps and inputting the preset second convolution module.

Preferably, the extraction unit is further configured to perform serum sample analysis on the multi-exposure test tube images, so as to calculate the position of the highest level of serum liquid level and the position of the lowest level of serum liquid level of the serum sample contained in the test tube; The highest position of the serum level and the lowest position of the serum level, in the multi-exposure test tube image, a matrix area of a preset size is extracted from the non-label area of the test tube.

Preferably, according to the position of the highest level of the serum level and the position of the lowest level of the serum, the extraction unit is further configured to, according to the position of the highest level of the serum level and the lowest position of the serum level, extract from Determine the serum image area in the unlabeled area of the test tube;

The matrix area is cut out from the serum image area according to the preset size, wherein the preset size is smaller than the size of the serum image area.

When each functional module of the neural network-based serum quality identification device proposed in this embodiment is running, the steps of implementing the above-mentioned neural network-based serum quality identification method will not be repeated here.

In addition, an embodiment of the present invention also proposes a computer-readable storage medium on which a neural network-based serum quality identification program is stored, and when the neural network-based serum quality identification program is executed by a processor, the above-mentioned The steps of a neural network-based serum mass identification method.

For the specific implementation manner of the computer-readable storage medium of the present invention, reference may be made to the above embodiments of the method for identifying serum quality based on a neural network, which will not be repeated here.

It should be noted that, herein, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or system comprising a series of elements includes not only those elements, It also includes other elements not expressly listed or inherent to such a process, method, article or system. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article or system that includes the element.

The above-mentioned serial numbers of the embodiments of the present invention are only for description, and do not represent the advantages or disadvantages of the embodiments.

From the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on such understanding, the technical solutions of the present invention can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products are stored in a storage medium (such as ROM/RAM) as described above. , magnetic disk, optical disc), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

The above are only preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied in other related technical fields , are similarly included in the scope of patent protection of the present invention.

Claims (10)

1. A serum quality identification method based on a neural network is characterized by comprising the following steps:

acquiring an image of a test tube containing a serum sample to obtain a test tube image;

inputting the test tube image into a preset neural network model so that the neural network model can output a serum quality identification result aiming at the serum sample, wherein the neural network model is obtained by performing convolutional neural network model training on the test tube image.

2. The neural network-based serum quality recognition method according to claim 1, wherein the method is characterized in that convolutional neural network model training is performed through multi-exposure test tube images;

the serum quality identification method based on the neural network further comprises the following steps:

acquiring multi-exposure test tube images, and extracting a matrix area belonging to the non-label area of the test tube from the multi-exposure test tube images;

inputting the matrix area into a preset first convolution module to perform first convolution neural network model training, and acquiring a characteristic diagram output by the first convolution module after performing first convolution neural network model training on the matrix area;

and stacking the characteristic graphs, inputting the stacked characteristic graphs into a preset second convolution module, and performing second convolution neural network model training to obtain a neural network model for performing serum quality identification on the serum sample.

3. The neural network-based serum quality identification method of claim 2, wherein the first convolution module and the second convolution module comprise: the first convolution module and the second convolution module are connected with one pooling layer after every two convolution layers except the last two convolution layers;

the convolution layers in the first convolution module comprise a plurality of first convolution layers with the step length of 1 and a plurality of second convolution layers with the step length of 2, and in every two first convolution layers, the first convolution layer of which the output end is not connected with the pooling layer is connected with one second convolution layer;

the number of convolution kernels of two first convolution layers at the end of the first convolution module is smaller than the number of convolution kernels of other first convolution layers and the second convolution layer.

4. The serum quality identification method based on the neural network as claimed in any one of claims 2 or 3, wherein a full connection layer and a logistic regression layer are connected to the end of the second convolution module, and the step of inputting the stacked feature maps into a preset second convolution module for second convolution neural network model training comprises:

stacking the feature maps, inputting the stacked feature maps into a preset second convolution module, and acquiring a new feature map output by the second convolution module after the feature maps are processed on the basis of the plurality of convolution layers and the pooling layer;

inputting the new feature map into the full-connection layer for feature classification to obtain a quality category of serum, wherein the quality category comprises: normal, hemolysis, lipemia and jaundice;

inputting the new feature map into the logistic regression layer to calculate a probability value of each quality class for determining the quality class as the corresponding quality grade when the hemolysis, the lipemia and the jaundice.

5. The neural network-based serum quality recognition method according to any one of claims 2 or 3, wherein the neural network-based serum quality recognition method further comprises:

and distributing weights to the feature maps, multiplying the feature maps by the distributed weights, and then executing the step of inputting the stacked feature maps into a preset second convolution module.

6. The neural network-based serum quality identification method according to claim 2, wherein the step of extracting a matrix area belonging to the tube non-label area from the multi-exposed tube image comprises:

performing serum sample analysis on the multi-exposure test tube image to calculate the highest position and the lowest position of the serum liquid level of the serum sample contained in the test tube in the multi-exposure test tube image;

and extracting a matrix area with a preset size from the non-label area of the test tube according to the highest position and the lowest position of the serum liquid level.

7. The neural network-based serum quality recognition method according to claim 6, wherein the step of extracting a matrix region of a preset size from the test tube non-label region according to the highest position of the serum level and the lowest position of the serum level comprises:

determining a serum image area from the non-label area of the test tube according to the highest position and the lowest position of the serum liquid level;

and intercepting the matrix region from the serum image region according to the preset size, wherein the preset size is smaller than the size of the serum image region.

8. A neural network-based serum quality identification apparatus, comprising:

the device comprises a test tube image acquisition module, a blood serum sample collection module and a blood serum sample collection module, wherein the test tube image acquisition module is used for acquiring an image of a test tube containing a blood serum sample;

and the serum quality identification module is used for inputting the test tube image into a preset neural network model so that the neural network model can output a serum quality identification result aiming at the serum sample, wherein the neural network model is obtained by performing convolutional neural network model training on the test tube image.

9. A terminal device, characterized in that the terminal device comprises: a memory, a processor and a neural network-based serum quality identification program stored on the memory and executable on the processor, the neural network-based serum quality identification program when executed by the processor implementing the steps of the neural network-based serum quality identification method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a neural network-based serum quality identification program, which when executed by a processor, implements the steps of the neural network-based serum quality identification method according to any one of claims 1 to 7.

Images