CN115985509A - Medical imaging data retrieval system, method, device and storage medium - Google Patents
Medical imaging data retrieval system, method, device and storage medium Download PDFInfo
- Publication number
- CN115985509A CN115985509A CN202211606957.1A CN202211606957A CN115985509A CN 115985509 A CN115985509 A CN 115985509A CN 202211606957 A CN202211606957 A CN 202211606957A CN 115985509 A CN115985509 A CN 115985509A
- Authority
- CN
- China
- Prior art keywords
- data
- retrieval
- image
- text
- iconography
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000002059 diagnostic imaging Methods 0.000 title claims abstract description 39
- 238000003384 imaging method Methods 0.000 claims abstract description 46
- 238000012549 training Methods 0.000 claims abstract description 11
- 239000013598 vector Substances 0.000 claims description 90
- 238000013135 deep learning Methods 0.000 claims description 18
- 238000007781 pre-processing Methods 0.000 claims description 17
- 230000006870 function Effects 0.000 claims description 16
- 238000000605 extraction Methods 0.000 claims description 11
- 238000012217 deletion Methods 0.000 claims description 6
- 230000037430 deletion Effects 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 6
- 238000012800 visualization Methods 0.000 claims 2
- 230000008569 process Effects 0.000 abstract description 11
- 230000000694 effects Effects 0.000 abstract description 9
- 238000003745 diagnosis Methods 0.000 abstract description 5
- 238000011160 research Methods 0.000 abstract description 5
- 230000007246 mechanism Effects 0.000 abstract description 4
- 238000005516 engineering process Methods 0.000 description 15
- 238000010586 diagram Methods 0.000 description 5
- 238000007689 inspection Methods 0.000 description 5
- 238000002591 computed tomography Methods 0.000 description 4
- 238000005259 measurement Methods 0.000 description 4
- 238000002601 radiography Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000013523 data management Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000001575 pathological effect Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application discloses a medical imaging data retrieval system, method, device and storage medium, the system includes: the system comprises a data synchronization module, a data retrieval module and an algorithm model. According to the method and the system, the medical image data can be preprocessed and extracted with features, then the image retrieval process is obtained similarly to the features, the model training, predicting, evaluating and deploying processes are achieved, the imaging data retrieval system is customized and can be deployed in different mechanisms, the imaging data can be obtained conveniently, and activities of the data in aspects of auxiliary diagnosis, scientific research activities and the like are exerted.
Description
Technical Field
The present application relates to the field of data retrieval, and in particular, to a system, a method, an apparatus, and a storage medium for medical imaging data retrieval.
Background
With the development of scientific technology and the increasing medical level, many imaging technologies for patient benefit have appeared, from the first X-ray to the present various digital imaging technologies, including Computed Radiography (CR), computed Tomography (CT), magnetic Resonance (MR), digital Radiography (DR), pathological smear scan imaging, camera imaging, etc., which are different from imaging technologies and principles, but have respective application scenarios. With the continuous improvement of the information level, an advanced image data management method in the medical industry is to introduce a Picture Archiving and Communication System (PACS) to collect, centralize, store and manage unstructured data in a digital manner. PACS is designed with DICOM3.0 international standard. The conventional PACS technology is not popularized and applied in a large scale, and a retrieval mode supported by a system is single, most of the retrieval modes are used for searching matching results through text keywords, and image content characteristics of images are not combined, so that the retrieval mode of images which are used for uploading image data, automatically analyzing and extracting and similarly acquiring is not supported.
The related image data retrieval technology has the problems that the matching result is searched through text keywords, and the image content characteristics of the image are not combined, namely most PACSs do not support image retrieval modes of uploading image data, automatic analysis and extraction and similar acquisition, the required information cannot be quickly inquired from massive image data, but a set of customized and deployed PACSs is expensive, the acquisition of the image data by scientific researchers is not facilitated, and the value of the data in the aspects of auxiliary diagnosis, scientific research activities and the like cannot be exerted.
Therefore, the above technical problems in the related art need to be solved.
Disclosure of Invention
The present application is directed to solving one of the technical problems in the related art. To this end, embodiments of the present application provide a medical imaging data retrieval system, method, apparatus and storage medium, which can perform medical imaging data retrieval using PACS technology.
According to an aspect of an embodiment of the present application, there is provided a medical imaging data retrieval system, including: the system comprises a data synchronization module, a data retrieval module and an algorithm model;
the data synchronization module is used for synchronizing the iconography image and the description text information from the data center, analyzing the description text information and extracting the structured text data, then extracting a characteristic vector from the image through a deep learning algorithm model, storing the structured text data into a full text retrieval database, storing the image characteristic vector into a characteristic vector database, and finally storing the incidence relation between the iconography data to provide retrieval basic data for the medical iconography data retrieval based on the combination of the image content and the text;
the data retrieval module is used for analyzing the iconography images and the multi-condition description text information uploaded from the data center, then converting the iconography images and the multi-condition description text information into multi-condition retrieval and executing different retrieval path text retrieval paths according to the routing rule, and the retrieval path is a result of searching and matching the description text information of the iconography images;
the algorithm model is used for training, predicting, evaluating and deploying the characteristic vector extraction algorithm model of the imaging image content.
In one embodiment, the system further comprises a data security module for logging and access right control;
the log record comprises a system log and a user service operation log, and the access authority control comprises data access authority and interface calling authority of a control user.
In one embodiment, the system comprises a video data deletion synchronization function, and after a user deletes video data of the system, the data center station automatically deletes the deleted data in the system synchronously.
In one embodiment, the data center calls a photographical data deleting interface, executes deleting specified feature vectors from the image feature vector database, deletes image structural text data stored in the full-text retrieval database and association relation information between the image feature vectors and the structural text data stored in the relational database, and updates data statistical information, including updating the total number of the inspection parts corresponding to the deleted images.
In one embodiment, the search path comprises a single text search path, a single image content search path, image content and a text search path;
the single image content retrieval path comprises the steps of extracting a characteristic vector of a imagery image, then calculating the similarity with the characteristic vector of the image of the characteristic vector database, returning the only main key of the characteristic vector, and matching the corresponding image result through the only main key; the combined image content and text retrieval path comprises a final retrieval result generated by combining retrieval results of the single text retrieval path and the single image content retrieval path.
In one embodiment, the system further comprises a data preprocessing module, wherein the data preprocessing module provides an API to the data center station to request, and the data center station executes different processing modes according to different data types after analyzing the request parameters.
According to an aspect of the embodiments of the present application, there is provided a medical imaging data retrieval method applied to the medical imaging data retrieval system described in the foregoing embodiments, the method including:
synchronizing the imagery data and the description text information from the data center, analyzing the description text information, extracting structured text data, extracting a feature vector from the image through a deep learning algorithm model, storing the structured text data into a full text retrieval database, storing the image feature vector into a feature vector database, and finally storing the association relationship between the imagery data to provide retrieval basic data for medical imagery data retrieval based on the combination of image content and text;
the method comprises the steps of analyzing the iconography image and the multi-condition description text information uploaded from the data center, converting the iconography image and the multi-condition description text information into multi-condition retrieval, and executing different retrieval path text retrieval paths according to routing rules.
According to an aspect of an embodiment of the present application, there is provided a medical imaging data retrieval apparatus, including:
the system comprises a first module, a second module and a third module, wherein the first module is used for synchronizing an iconography image and description text information from a data center, analyzing the description text information, extracting structured text data, extracting a characteristic vector from the image through a deep learning algorithm model, storing the structured text data into a full text retrieval database, storing the image characteristic vector into a characteristic vector database, and finally storing the incidence relation between the iconography data to provide retrieval basic data for medical iconography data retrieval based on the combination of image content and text;
the second module is used for analyzing the iconography images and the multi-condition description text information uploaded from the data center, then converting the iconography images and the multi-condition description text information into a multi-condition retrieval and executing different retrieval path text retrieval paths according to the routing rule, and the matching result of the description text information of the iconography images is searched.
According to an aspect of an embodiment of the present application, there is provided a medical imaging data retrieval apparatus, including:
at least one processor;
at least one memory for storing at least one program;
at least one of the programs, when executed by at least one of the processors, implements a medical imaging data retrieval system as described in the previous embodiments.
According to an aspect of the embodiments of the present application, a storage medium is provided, where the storage medium stores a program executable by a processor, and the program executable by the processor is executed by the processor to implement a medical imaging data retrieval system according to the foregoing embodiments.
The medical imaging data retrieval system, the method, the device and the storage medium provided by the embodiment of the application have the beneficial effects that: the system comprises: the system comprises a data synchronization module, a data retrieval module and an algorithm model; the data synchronization module is used for synchronizing the imaging and the description text information from the data center, analyzing the description text information, extracting the structured text data, extracting the feature vector of the image through a deep learning algorithm model, storing the structured text data into a full text retrieval database, storing the image feature vector into a feature vector database, and finally storing the incidence relation between the imaging data to provide retrieval basic data for the medical imaging data retrieval based on the combination of the image content and the text; the data retrieval module is used for analyzing the iconography images and the multi-condition description text information uploaded from the data center, then converting the images into the multi-condition retrieval and executing different retrieval path text retrieval paths according to the routing rules, and the data retrieval module is used for searching and matching the description text information of the iconography images. According to the method and the system, the medical image data can be preprocessed and extracted with features, then the image retrieval process is obtained similarly to the features, the model training, predicting, evaluating and deploying processes are achieved, the imaging data retrieval system is customized and can be deployed in different mechanisms, the imaging data can be obtained conveniently, and activities of the data in aspects of auxiliary diagnosis, scientific research activities and the like are exerted.
Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is a schematic diagram illustrating an architecture of a medical imaging data retrieval system according to an embodiment of the present application;
fig. 2 is a flowchart of the imaging data synchronization according to the embodiment of the present application;
FIG. 3 is a flowchart of a descriptive text based retrieval provided by an embodiment of the present application;
FIG. 4 is a flowchart of an image content based retrieval provided by an embodiment of the present application;
FIG. 5 is a flowchart of a retrieval based on image content combined with text provided by an embodiment of the present application;
fig. 6 is a schematic diagram of a medical imaging data retrieval device according to an embodiment of the present application;
fig. 7 is a schematic diagram of another medical imaging data retrieval device according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort shall fall within the protection scope of the present application.
The terms "first," "second," "third," and "fourth," etc. in the description and claims of this application and the accompanying drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
With the development of scientific technology and the improvement of medical level, many imaging technologies for patient benefit have appeared, and from the first X-ray to the present, various digital imaging technologies include Computed Radiography (CR), computed Tomography (CT), magnetic Resonance (MR), digital Radiography (DR), pathological smear scan imaging, camera imaging, etc., which are different from imaging technologies and principles, but have respective application scenarios. With the continuous improvement of the information level, an advanced image data management method in the medical industry is to introduce a Picture Archiving and Communication System (PACS) to collect, centralize, store and manage unstructured data in a digital manner. PACS is designed with DICOM3.0 international standard. The conventional PACS technology is not popularized and applied in a large scale, and a retrieval mode supported by a system is single, most of the retrieval modes are used for searching matching results through text keywords, and image content characteristics of images are not combined, so that the retrieval mode of images which are used for uploading image data, automatically analyzing and extracting and similarly acquiring is not supported.
The related image data retrieval technology has the problems that the matching result is searched through text keywords and the image content characteristics of the image are not combined, namely most of PACSs do not support the image retrieval modes of uploading image data, automatic analysis and extraction and similar acquisition, the required information cannot be quickly inquired from massive image data, but a set of PACSs is customized and deployed, the price of the PACS is high, scientific researchers are not facilitated to acquire the image data, and the value of the data in the aspects of auxiliary diagnosis, scientific research activities and the like cannot be exerted.
In order to solve the above problems, the present application provides a medical imaging data retrieval system. The application provides a medical imaging data retrieval system based on combination of image content and text, which is used as a supplement to an imaging data retrieval function of a data center station. The system is based on a Browser/Server (B/S) system framework, adopts a very popular flash frame at present, is a lightweight customizable frame, is written by using Python language, is more flexible, portable, safe and easy to operate compared with other frames of the same type, and can be well developed by combining with an MVC mode. The resource layer is connected with the relational database and abstracted and packaged on the basis of accessing the database, so that the separation of data persistence operation and business logic is realized, and the complexity and the workload of developing the business logic are reduced.
Fig. 1 is a schematic diagram illustrating an architecture of a medical imaging data retrieval system according to an embodiment of the present disclosure. As shown in fig. 1, the system of the present application includes: the system comprises a data synchronization module, a data retrieval module and an algorithm model; the data synchronization module is used for synchronizing the iconography image and the description text information from the data center, analyzing the description text information and extracting the structured text data, then extracting a characteristic vector from the image through a deep learning algorithm model, storing the structured text data into a full text retrieval database, storing the image characteristic vector into a characteristic vector database, and finally storing the incidence relation between the iconography data to provide retrieval basic data for the medical iconography data retrieval based on the combination of the image content and the text; the data retrieval module is used for analyzing the iconography images and the multi-condition description text information uploaded from the data center, then converting the iconography images and the multi-condition description text information into multi-condition retrieval and executing different retrieval path text retrieval paths according to the routing rule, and the retrieval path is a result of searching and matching the description text information of the iconography images; the algorithm model is used for training, predicting, evaluating and deploying the characteristic vector extraction algorithm model of the imaging image content.
As shown in fig. 1, the whole platform of the present application is a running environment layer, a data resource layer, a data service layer, a platform application layer, a display layer, a front end, and an application system layer. The system supports independent deployment and operation, and provides an Application Programming Interface (API) of a Restful architecture for Application system call with data interaction. The operation environment layer provides a basic environment of the whole platform and comprises a local machine room, a server, a cloud service, a network, a load balancer and a firewall. The data resource layer comprises a relational database, a characteristic vector database, an object storage system and a local file storage system. The data service layer comprises data transaction, data reading and writing, data statistics, image feature extraction, data retrieval, model training, model evaluation and model deployment. The system application layer comprises data element query, detection data set retrieval and detection data set configuration. The display layer provides template engine rendering and API and provides an application system with data interaction, the application system comprises a Get request and a Post request, the front end and application system layer is integrated with the front end and the rear end of the data center, the front end renders data returned by the retrieval request, and the rear end performs data interaction service with the system. Technologies such as log recording, access authority control, network firewall and the like are used for data security.
The system in this embodiment mainly includes the following functions:
(1) And data synchronization, namely synchronizing the image and the description text information of the iconography from the data center, firstly analyzing the description text information, then extracting the structured text data, then extracting the feature vector of the image through a deep learning algorithm model, storing the structured text data into a full text retrieval database, storing the image feature vector into a feature vector database, and finally storing the association relationship between the iconography data to provide retrieval basic data for the medical iconography data retrieval based on the combination of the image content and the text. And supporting the synchronous function of the deletion of the imaging data.
(2) And data retrieval, namely analyzing the video image and the multi-condition description text information uploaded from the data center, converting the information into multi-condition retrieval, and executing different retrieval paths according to routing rules, wherein three retrieval paths exist, namely single text, single image content, combination of the image content and the text, and the like. The text retrieval path is a result of searching and matching the description text information of the imaging image; the single image content retrieval path is to extract the image characteristic vector of the imagery, then calculate the similarity with the image characteristic vector of the characteristic vector database and return TopN (N > = 1) unique main keys of the characteristic vector, and match the corresponding image result through the unique main keys; the combined image content and text retrieval path is a retrieval result of a single text and a single imaging image retrieval path, and the retrieval results are combined to generate a final retrieval result.
(3) The algorithm model realizes the functions of training, predicting, evaluating, deploying and the like of the characteristic vector extraction algorithm model of the image content of the iconography.
(4) Data security, logging and access authority control. The log record comprises a system log and a user service operation log, and the access authority control comprises data access authority and interface calling authority of a control user.
It should be noted that the system of the present application includes an imaging data deletion synchronization function, and after the user deletes the imaging data of the system, the data center automatically deletes the deleted data in the system in synchronization. The data center station calls a photographical data deleting interface, deletes specified characteristic vectors from the image characteristic vector database, deletes image structured text data stored in the full-text retrieval database and incidence relation information between the image characteristic vectors and the structured text data stored in the relational database, and updates data statistical information, including updating the total number of the inspection parts corresponding to the deleted images.
In this embodiment, the search path includes a single text search path, a single image content search path, an image content and a text search path; the single image content retrieval path comprises the steps of extracting a characteristic vector of a imagery image, then calculating the similarity with the characteristic vector of the image of the characteristic vector database, returning the only main key of the characteristic vector, and matching the corresponding image result through the only main key; the image content and text combined retrieval path comprises a final retrieval result generated by combining retrieval results of the single text retrieval path and the single image content retrieval path.
In addition, the system of the application further comprises a data preprocessing module, wherein the data preprocessing module provides an API (application programming interface) to the data center station to request, and the data center station executes different processing modes according to different data types after analyzing the request parameters. The data preprocessing specifically comprises the following steps:
a. the data preprocessing function is requested by the system to provide an API for a data center station, the request parameters comprise an imaging image and descriptive text information, and the imaging descriptive text information comprises an imaging image unique id, an image OSS path, an inspection part, an inspection direction, a layer thickness, a layer spacing, an inspection date and the like.
b. And executing different processing modes according to different data types after analyzing the request parameters.
c. And analyzing the descriptive text of the imaging image.
c1. The descriptive text information is extracted and converted into structured text data.
c2. And the structured text data serving as an input value is transmitted into the feature extraction and storage module to be processed in the next step.
d. The imaging image is analyzed.
d1. Downloaded locally from the stored imagery images at the OSS.
d2. Image preprocessing, which mainly converts a plurality of imaging image formats (such as jpg, jpeg, png, bmp, dcm and nii) into a jpg format in a unified manner, converts the image size into the input size of a deep learning algorithm model, namely, one dimension is added under the original three dimensions to enable the image to conform to the size required by the algorithm model, and outputs an array with four dimensions.
d3. And the preprocessed image array is used as an input value and is transmitted into the feature extraction and storage module for further processing.
As shown in fig. 2, the method for data synchronization of the present application includes:
a. and receiving the imaging image array and the structured text data input by the data preprocessing module.
b. Structured text data of a visual image is processed.
b1. Storing the data into a full-text retrieval database.
c. And the received data preprocessing module carries out the next processing after inputting the imaging image array.
c1. And inputting the imaging image array into a pre-trained VGG16 deep learning algorithm model, and then extracting an image feature vector.
c2. And regularizing image feature vectors.
c3. The image feature vector is inserted into a collection of image feature vector databases.
c4. And after the successful insertion, the image feature vector database returns the only main key of the image in the set, and the corresponding image result is matched through the only main key.
c5. And data statistics and updating, wherein the data statistics and updating are mainly to count and update the total number of the body parts corresponding to the synchronously-stored images.
d. The image and the structured text data are successfully stored in the database, and then the incidence relation information of the image characteristic vector and the structured text data is recorded in the relational database, so that reliable index data are provided for the subsequent image data retrieval.
e. Completing the data synchronization of the imaging.
As shown in fig. 3, the process of retrieving based on the description text includes: and the data center performs retrieval based on the description text, analyzes the description text and converts the description text into retrieval conditions, searches the database in a full-text manner under multiple conditions, calculates scores and obtains similarity, and finally generates and displays a retrieval result.
As shown in fig. 4, the process of retrieving based on image content includes: and (3) image preprocessing is carried out, a VGG16 model is called to extract a feature vector, the feature vector is regularized, the cosine distance measurement similarity is calculated to obtain the similarity, and finally a retrieval result is generated and displayed.
As shown in fig. 5, the process of retrieving based on the description text and the image content includes: and analyzing the description text and the image content, converting the description text and the image content into retrieval conditions, respectively performing retrieval based on the description text and retrieval based on the image content, finally merging retrieval results of the retrieval based on the description text and the retrieval based on the image content, generating and displaying the retrieval results.
Specifically, the description text-based retrieval and the image content-based retrieval include:
(1) Descriptive text based retrieval
The user inputs a multi-condition retrieval parameter based on the iconography description text from the data center retrieval front end page and requests to the system. And analyzing the multi-condition retrieval parameters and converting the parameters into the retrieval semantics of the full-text retrieval database. The full-text search database multi-conditionally searches for matching image results. And calculating a similarity score, and retrieving images and description texts which return a specified number N (N > = 1) of similarities and are ranked from high to low. And generating a retrieval result and returning the retrieval result to the data center for displaying.
(2) Retrieval based on image content
The user retrieves a file of a legal image (such as jpg, jpeg, png and bmp formats) or image data (such as dcm and nii formats) uploaded from a front-end page from the data center, checks that the part parameters are optional, and then requests the image data retrieval interface of the system. Image preprocessing, which mainly converts a plurality of imaging image formats into a jpg format in a unified manner, then converts the image size into the input size of a deep learning algorithm model, namely, one dimension is added under the original three dimensions to enable the image to conform to the size required by the algorithm model, and an array of four dimensions is output. And extracting image feature vectors through the fine-tuned VGG16 deep learning neural network after preprocessing. And regularizing image feature vectors. And calculating the similarity between the extracted image feature vector and the image feature vector in the feature database through cosine distance measurement. The retrieval returns a unique primary key of the image feature vectors ordered by similarity from high to low of a specified number N (N > = 1) in the feature vector database set. And inquiring the image association relation information stored in the relational database through the unique main key in the feature vector database set, wherein the image corresponding to the matching result is the retrieval result. And generating a retrieval result and returning the retrieval result to the data center for displaying.
(3) Retrieval based on combination of image content and text
The user inputs multiple condition retrieval parameters based on the iconography description text from the data center retrieval front-end page, uploads a legal image (such as jpg, jpeg, png and bmp format) or an iconography data (such as dcm and ni format) file, and then requests the iconography data retrieval interface of the system. And analyzing the multi-condition retrieval parameters and executing different retrieval paths according to the route. A multiple conditional search path for a text-to-execute full-text search database is described. And converting the data into the retrieval semantics of a full-text retrieval database, returning a matched image result after the full-text retrieval database is retrieved by multiple conditions, and calculating a similarity score. And retrieving images and description texts which return a specified number N (N > = 1) of similarity degrees and are sorted from high to low. The image performs a single image content retrieval path. Image preprocessing, which mainly converts a plurality of imaging image formats into a jpg format in a unified manner, then converts the image size into the input size of a deep learning algorithm model, namely, one dimension is added under the original three dimensions to enable the image to conform to the size required by the algorithm model, and an array of four dimensions is output. And extracting image feature vectors through the fine-tuned VGG16 deep learning neural network. And regularizing image feature vectors. And calculating the similarity between the extracted image feature vector and the image feature vector in the feature database through cosine distance measurement. The retrieval returns a unique primary key of the image feature vectors ordered by similarity from high to low of a specified number N (N > = 1) in the feature vector database set. And inquiring the image association relation information stored in the relational database through the unique main key in the feature vector database set, wherein the image corresponding to the matching result is the retrieval result. And combining the retrieval results of the two retrieval paths to generate a final retrieval result, and returning the generated retrieval result to the data center for display.
The method for training, predicting and deploying the algorithm model in the embodiment of the application comprises the following steps:
(1) Algorithm model training
a. The VGG16 algorithm model was pre-trained using ImageNet-based data sets, and then imaging data of different examination sites of the patient were collected and used.
b. Image preprocessing, which mainly converts a plurality of imaging image formats (such as jpeg, png, bmp, dcm, ni) into a jpg format in a unified manner, converts the image size into the input size of a deep learning algorithm model, namely, one dimension is added under the original three dimensions to enable the image to conform to the size required by the algorithm model, and outputs an array of four dimensions.
c. And (5) fine adjustment of the VGG16 algorithm model.
d. And generating an optimal model after multiple iterations.
e. Saving the model to a model library.
(2) Algorithmic model deployment
a. A specified model is selected from a library of models.
b. Deployment is achieved using the Tensorflow serving + Docker technique. The deployed model supports the API of Restful architecture and gRPC request, and provides online support for the image feature extraction task of the algorithm model.
(3) Algorithmic model prediction
a. The user uploads a valid imaging image.
b. And extracting the image feature vector through a VGG16 algorithm model.
c. And calculating similarity with image feature vectors in a feature vector database by using cosine distance measurement, and predicting the image feature vectors with the specified number N (N > = 1).
d. And generating a retrieval result and displaying the result.
(4) Algorithm model incremental training
a. And synchronously warehousing the new imaging images.
b. And extracting the characteristic vector of the imaging image.
c. The fine tuning continues using the trained and fine tuned VGG16 model.
d. And generating an optimal model after multiple iterations.
And finally, storing the model in a model library to provide a multi-version model for subsequent model deployment.
The invention supports the image data uploading, the data preprocessing and the feature extraction, the image retrieval process similar to the feature acquisition, the model training, predicting, evaluating and deploying processes are realized, the B/S architecture image data retrieval system is customized and deployed, the deployment in different mechanisms is supported, the implementation and management of a platform in multiple mechanisms at different places are supported, the acquisition of the image data is facilitated, and the activities of the data in the aspects of auxiliary diagnosis, scientific research activities and the like are exerted.
The present application further provides a medical imaging data retrieval method applied to the medical imaging data retrieval system of the foregoing embodiment, the method including: synchronizing the imagery data and the description text information from the data center, analyzing the description text information, extracting structured text data, extracting a feature vector from the image through a deep learning algorithm model, storing the structured text data into a full text retrieval database, storing the image feature vector into a feature vector database, and finally storing the association relationship between the imagery data to provide retrieval basic data for medical imagery data retrieval based on the combination of image content and text; the method comprises the steps of analyzing the iconography images and the multi-condition description text information uploaded from the data center, then converting the iconography images and the multi-condition description text information into multi-condition retrieval, and executing different retrieval paths according to routing rules.
The present application also proposes a medical imaging data retrieval device, as shown in fig. 6, the device includes:
the system comprises a first module, a second module and a third module, wherein the first module is used for synchronizing an image of the iconography and description text information from a data center, analyzing the description text information, extracting structured text data, extracting a feature vector from the image through a deep learning algorithm model, storing the structured text data into a full text retrieval database, storing the image feature vector into a feature vector database, and finally storing the incidence relation between the iconography data to provide retrieval basic data for the retrieval of the medical iconography data based on the combination of image content and text;
the second module is used for analyzing the iconography images and the multi-condition description text information uploaded from the data center, then converting the iconography images and the multi-condition description text information into a multi-condition retrieval and executing different retrieval path text retrieval paths according to the routing rule, and the matching result of the description text information of the iconography images is searched.
The present application further provides a medical imaging data retrieval device, as shown in fig. 7, the device includes:
at least one processor;
at least one memory for storing at least one program;
at least one of the programs, when executed by at least one of the processors, implements a medical imaging data retrieval system as described in previous embodiments.
The present application also provides a storage medium storing a program executable by a processor, wherein the program executable by the processor realizes a medical imaging data retrieval system according to the foregoing embodiment when executed by the processor.
Similarly, the contents in the foregoing method embodiments are all applicable to this storage medium embodiment, the functions specifically implemented by this storage medium embodiment are the same as those in the foregoing method embodiments, and the advantageous effects achieved by this storage medium embodiment are also the same as those achieved by the foregoing method embodiments.
In alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flowcharts of the present application are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are performed independently.
Furthermore, although the present application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and/or features may be integrated in a single physical device and/or software module, or one or more functions and/or features may be implemented in a separate physical device or software module. It will also be appreciated that a detailed discussion regarding the actual implementation of each module is not necessary for an understanding of the present application. Rather, the actual implementation of the various functional modules in the apparatus disclosed herein will be understood within the ordinary skill of an engineer, given the nature, function, and internal relationship of the modules. Accordingly, those of ordinary skill in the art will be able to implement the present application as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative of and not intended to limit the scope of the application, which is defined by the appended claims and their full scope of equivalents.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
In the foregoing description of the specification, reference to the description of "one embodiment/example," "another embodiment/example," or "certain embodiments/examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that: numerous changes, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the application, the scope of which is defined by the claims and their equivalents.
The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.
Claims (10)
1. A medical imaging data retrieval system, the system comprising: the system comprises a data synchronization module, a data retrieval module and an algorithm model;
the data synchronization module is used for synchronizing the imaging and the description text information from the data center, analyzing the description text information, extracting the structured text data, extracting the feature vector of the image through a deep learning algorithm model, storing the structured text data into a full text retrieval database, storing the image feature vector into a feature vector database, and finally storing the incidence relation between the imaging data to provide retrieval basic data for the medical imaging data retrieval based on the combination of the image content and the text;
the data retrieval module is used for analyzing the iconography images and the multi-condition description text information uploaded from the data console, then converting the iconography images and the multi-condition description text information into multi-condition retrieval and executing different retrieval path text retrieval paths according to the routing rule, wherein the retrieval path is a result of searching and matching the description text information of the iconography images;
the algorithm model is used for training, predicting, evaluating and deploying the characteristic vector extraction algorithm model of the imaging image content.
2. A medical imaging data retrieval system as claimed in claim 1, wherein the system further comprises a data security module for logging and access rights control;
the log record comprises a system log and a user service operation log, and the access authority control comprises data access authority and interface calling authority of a control user.
3. The system of claim 1, wherein the system comprises a visualization data deleting synchronization function, and after the user deletes the visualization data of the system, the data center automatically deletes the deleted data in the system in synchronization.
4. The medical imaging data retrieval system according to claim 3, wherein the data center calls an imaging data deletion interface, and executes deletion of the specified feature vector from the image feature vector database, deletion of the image structured text data stored in the full-text retrieval database and the association relationship information between the image feature vector and the structured text data stored in the relational database, and updating of data statistics, including updating of the total number of the examination parts corresponding to the deleted image.
5. The medical imaging data retrieval system of claim 1, wherein the retrieval path comprises a single text retrieval path, a single image content retrieval path, image content and text retrieval path;
the single image content retrieval path comprises the steps of extracting a characteristic vector of a imagery image, then calculating the similarity with the characteristic vector of the image of the characteristic vector database, returning the only main key of the characteristic vector, and matching the corresponding image result through the only main key; the image content and text combined retrieval path comprises a final retrieval result generated by combining retrieval results of the single text retrieval path and the single image content retrieval path.
6. The medical imaging data retrieval system of claim 1, further comprising a data preprocessing module, wherein the data preprocessing module provides an API to the data center station request, and the data center station performs different processing modes according to different data types after analyzing the request parameters.
7. A medical imaging data retrieval method applied to the medical imaging data retrieval system according to claim 1, the method comprising:
synchronizing the imagery data and the description text information from the data center, analyzing the description text information, extracting structured text data, extracting a feature vector from the image through a deep learning algorithm model, storing the structured text data into a full text retrieval database, storing the image feature vector into a feature vector database, and finally storing the association relationship between the imagery data to provide retrieval basic data for medical imagery data retrieval based on the combination of image content and text;
the method comprises the steps of analyzing the iconography image and the multi-condition description text information uploaded from the data center, converting the iconography image and the multi-condition description text information into multi-condition retrieval, and executing different retrieval path text retrieval paths according to routing rules.
8. A medical imaging data retrieval apparatus, the apparatus comprising:
the system comprises a first module, a second module and a third module, wherein the first module is used for synchronizing an image of the iconography and description text information from a data center, analyzing the description text information, extracting structured text data, extracting a feature vector from the image through a deep learning algorithm model, storing the structured text data into a full text retrieval database, storing the image feature vector into a feature vector database, and finally storing the incidence relation between the iconography data to provide retrieval basic data for the retrieval of the medical iconography data based on the combination of image content and text;
and the second module is used for analyzing the iconography images and the multi-condition description text information uploaded from the data console, then converting the iconography images and the multi-condition description text information into multi-condition retrieval and executing different retrieval path text retrieval paths according to the routing rule, and is used for searching and matching the description text information of the iconography images.
9. A medical imaging data retrieval apparatus, the apparatus comprising:
at least one processor;
at least one memory for storing at least one program;
a medical imaging data retrieval system as claimed in any one of claims 1-6, when at least one of said programs is executed by at least one of said processors.
10. Storage medium, characterized in that the storage medium stores a program executable by a processor, which program, when executed by the processor, implements a medical imaging data retrieval system as claimed in any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211606957.1A CN115985509A (en) | 2022-12-14 | 2022-12-14 | Medical imaging data retrieval system, method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211606957.1A CN115985509A (en) | 2022-12-14 | 2022-12-14 | Medical imaging data retrieval system, method, device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115985509A true CN115985509A (en) | 2023-04-18 |
Family
ID=85958780
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211606957.1A Pending CN115985509A (en) | 2022-12-14 | 2022-12-14 | Medical imaging data retrieval system, method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115985509A (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101441658A (en) * | 2008-10-22 | 2009-05-27 | 中国科学院上海技术物理研究所 | Search method and system facing to radiation image in PACS database based on content |
| CN102156715A (en) * | 2011-03-23 | 2011-08-17 | 中国科学院上海技术物理研究所 | Retrieval system based on multi-lesion region characteristic and oriented to medical image database |
| CN103164539A (en) * | 2013-04-15 | 2013-06-19 | 中国传媒大学 | Interactive type image retrieval method of combining user evaluation and labels |
| CN105005578A (en) * | 2015-05-21 | 2015-10-28 | 中国电子科技集团公司第十研究所 | Multimedia target information visual analysis system |
| KR101873926B1 (en) * | 2017-11-22 | 2018-07-04 | 김광호 | Method for providing medical counseling service between insurance organization and specialist based on bigdata |
| CN111625667A (en) * | 2020-05-18 | 2020-09-04 | 北京工商大学 | Three-dimensional model cross-domain retrieval method and system based on complex background image |
| CN111919260A (en) * | 2018-03-19 | 2020-11-10 | 威里利生命科学有限责任公司 | Surgical video retrieval based on preoperative images |
| KR102179583B1 (en) * | 2020-03-11 | 2020-11-17 | 주식회사 딥노이드 | Assistance diagnosis system for disease based on deep learning and assistance diagnosis method thereof |
| CN112242185A (en) * | 2020-09-09 | 2021-01-19 | 山东大学 | Medical image report automatic generation method and system based on deep learning |
| US20210345955A1 (en) * | 2020-05-05 | 2021-11-11 | Dave Engineering Llc | Portable real-time medical diagnostic device |
| CN114896438A (en) * | 2022-05-10 | 2022-08-12 | 西安电子科技大学 | Image-text retrieval method based on hierarchical alignment and generalized pooling graph attention machine mechanism |
-
2022
- 2022-12-14 CN CN202211606957.1A patent/CN115985509A/en active Pending
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101441658A (en) * | 2008-10-22 | 2009-05-27 | 中国科学院上海技术物理研究所 | Search method and system facing to radiation image in PACS database based on content |
| CN102156715A (en) * | 2011-03-23 | 2011-08-17 | 中国科学院上海技术物理研究所 | Retrieval system based on multi-lesion region characteristic and oriented to medical image database |
| CN103164539A (en) * | 2013-04-15 | 2013-06-19 | 中国传媒大学 | Interactive type image retrieval method of combining user evaluation and labels |
| CN105005578A (en) * | 2015-05-21 | 2015-10-28 | 中国电子科技集团公司第十研究所 | Multimedia target information visual analysis system |
| KR101873926B1 (en) * | 2017-11-22 | 2018-07-04 | 김광호 | Method for providing medical counseling service between insurance organization and specialist based on bigdata |
| CN111919260A (en) * | 2018-03-19 | 2020-11-10 | 威里利生命科学有限责任公司 | Surgical video retrieval based on preoperative images |
| KR102179583B1 (en) * | 2020-03-11 | 2020-11-17 | 주식회사 딥노이드 | Assistance diagnosis system for disease based on deep learning and assistance diagnosis method thereof |
| US20210345955A1 (en) * | 2020-05-05 | 2021-11-11 | Dave Engineering Llc | Portable real-time medical diagnostic device |
| CN111625667A (en) * | 2020-05-18 | 2020-09-04 | 北京工商大学 | Three-dimensional model cross-domain retrieval method and system based on complex background image |
| CN112242185A (en) * | 2020-09-09 | 2021-01-19 | 山东大学 | Medical image report automatic generation method and system based on deep learning |
| CN114896438A (en) * | 2022-05-10 | 2022-08-12 | 西安电子科技大学 | Image-text retrieval method based on hierarchical alignment and generalized pooling graph attention machine mechanism |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zhou et al. | Semantic understanding of scenes through the ade20k dataset | |
| JP6005837B2 (en) | Image analysis apparatus, image analysis system, and image analysis method | |
| JP2022541199A (en) | A system and method for inserting data into a structured database based on image representations of data tables. | |
| EP4207772A1 (en) | Video processing method and apparatus | |
| US20100106522A1 (en) | System and method for organizing and displaying of longitudinal multimodal medical records | |
| CN111522987A (en) | Image auditing method and device and computer readable storage medium | |
| WO2023108980A1 (en) | Information push method and device based on text adversarial sample | |
| KR101925603B1 (en) | Method for faciliating to read pathology image and apparatus using the same | |
| CN109308324A (en) | A kind of image search method and system based on hand drawing style recommendation | |
| CN114913942A (en) | Intelligent matching method and device for patient recruitment projects | |
| CN112307303A (en) | Efficient and accurate network page duplicate removal system based on cloud computing | |
| CN110489613B (en) | Collaborative visual data recommendation method and device | |
| CN114003758B (en) | Training method and device of image retrieval model and retrieval method and device | |
| CN111199801B (en) | Construction method and application of model for identifying disease types of medical records | |
| Sethi et al. | Large-scale multimedia content analysis using scientific workflows | |
| Jin et al. | PAI-WSIT: an AI service platform with support for storing and sharing whole-slide images with metadata and annotations | |
| Girdhar et al. | Benchmarking nas for article separation in historical newspapers | |
| CN115985509A (en) | Medical imaging data retrieval system, method, device and storage medium | |
| CN113343012B (en) | News matching method, device, equipment and storage medium | |
| CN112086174A (en) | Three-dimensional knowledge diagnosis model construction method and system | |
| Ho et al. | Uit at vbs 2022: An unified and interactive video retrieval system with temporal search | |
| CN112052365A (en) | Cross-border scene portrait construction method and device | |
| Rodriguez et al. | Extracting, Assimilating, and Sharing the Results of Image Analysis on the FSA/OWI Photography Collection | |
| Müller et al. | Studying public medical images from the open access literature and social networks for model training and knowledge extraction | |
| Devera et al. | Team 3: Object Detection and Topic Modeling (Objects&Topics) CS 5604 F2022 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |