CN112765131B - Heterogeneous medical health data storage and retrieval method and system - Google Patents
Heterogeneous medical health data storage and retrieval method and system Download PDFInfo
- Publication number
- CN112765131B CN112765131B CN202110087530.4A CN202110087530A CN112765131B CN 112765131 B CN112765131 B CN 112765131B CN 202110087530 A CN202110087530 A CN 202110087530A CN 112765131 B CN112765131 B CN 112765131B
- Authority
- CN
- China
- Prior art keywords
- data
- index
- retrieval
- query
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/211—Schema design and management
- G06F16/212—Schema design and management with details for data modelling support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention relates to a heterogeneous medical health data storage and retrieval method and system, and belongs to the technical field of data storage and retrieval. The method comprises a heterogeneous medical health data storage method and a heterogeneous medical health data retrieval method. The invention provides a heterogeneous medical health data storage and retrieval method and system, which can be used for classifying and efficiently storing and managing heterogeneous data, can improve the retrieval efficiency and the management efficiency in a big data environment, and provides a theoretical method and system reference for realizing medical data sharing among medical institutions.
Description
Technical Field
The invention belongs to the technical field of data storage and retrieval, and relates to a heterogeneous medical health data storage and retrieval method and system.
Background
Common types of health data are structured, text, images, and the like. The structured data mainly comprises basic information, physical sign data and the like of a patient; the text data is mainly medical text or file data of a patient, and is mainly expressed in an XML format file; the image data is mainly DICOM medical image data of CT, MR and other equipment, and is mainly expressed in a DCM file. In order to fully mine and utilize the data of the medical institution, fusion and sharing of the medical data are required to be realized. Under the current situation, data is often scattered in each large medical institution, which is not beneficial to data sharing, and meanwhile, the management efficiency is low; in the aspect of realizing data sharing among organizations, an ETL tool is generally used to connect databases of large organizations, and data is integrated when needed to be used and analyzed. The method is established on the own retrieval mechanism of the large database systems of each organization, and the retrieval efficiency of the large database systems of each organization is often different, so that the retrieval efficiency of the whole system is low.
Therefore, in view of the above problems, a method for storing and managing heterogeneous data in a classified and efficient manner is needed.
Disclosure of Invention
In view of the above, the present invention provides a method and a system for storing and retrieving heterogeneous medical health data.
In order to achieve the purpose, the invention provides the following technical scheme:
a heterogeneous medical health data storage and retrieval method comprises a heterogeneous medical health data storage method and a heterogeneous medical health data retrieval method;
the heterogeneous medical health data storage method comprises the following steps:
s11: establishing MySQL and HBase database tables;
s12: adopting a pre-partitioning method to partition regions for HBase, setting a Rowkey range of the regions, and forbidding automatic Split of the HBase;
s13: different types of Rowkeys are set to have different composition structures, the data Rowkey consists of Hash + Rowkey, and the index Rowkey consists of StartKey + IndexKey + IndexValue + DataRowkey;
s14: the extraction, conversion and loading ETL module extracts data to a database according to different medical protocol data exchange standards, wherein structured data is inserted into MySQL, image data and medical text data are inserted into HDFS, and metadata is inserted into HBase;
s15: HBase receives a data insertion request of a Client end, a coprocessor captures a Put request, and a rewritten prePut method is executed;
the heterogeneous medical health data retrieval method comprises the following steps:
s21: creating a Protubuf file, and customizing an RPC request method and a response method;
s22: the retrieval module judges the query requirement of the Client terminal, and if the query data is structured data, the S23 is carried out; if the query data is unstructured data, entering S24; if the type is heterogeneous, using a multithreading mode to asynchronously acquire data, and respectively executing S23 and S24-S210;
s23: querying MySQL according to conditions, and entering S211 after the query is completed;
s24: the retrieval module queries the index configuration file according to the query field of the Client, matches the optimal index and generates a query condition expression;
s25: the retrieval module takes the index type IndexKey and the query condition expression as RPC request method parameters, then requests to each RegionServer in parallel, and the RegionServer performs query operation on each Region by using the agent component;
s26: the query operation at each Region is: firstly, acquiring StartKey of a current Region, splicing the StartKey and an index type to be used as an index prefix, splicing a query range boundary value of a first field of a query conditional expression and the index prefix to be used as StartRow and StopRow of Scan, and then performing Scan query on the current Region;
s27: AND if the Scan result set is not empty AND the subsequent conditional operator is AND, screening the subsequent field value according to the query conditional expression, extracting DataRowkey, AND entering S29. If the subsequent condition operator is OR, extracting DataRowkey in the Scan data set, and entering S29;
s28: if the Scan result set is null AND the subsequent condition operator is AND, then a null result is returned AND S210 is entered. If the subsequent conditional operator is OR, performing full Scan operation in the index Region in the current Region, screening the value of the subsequent field, and extracting DataRowkey;
s29; iterating the DataRowkey data set, redirecting back to the current Region, and performing Get operation according to the DataRowkey to obtain data in the data area;
s210: the retrieval module receives a non-empty result responded by one RPC request, then the step directly enters S211, and then the retrieval module continuously waits for response results of other RPC requests;
s211: and packaging the result according to the query requirement and returning the result to the client through the interface.
Optionally, the MySQL table is divided into different tables according to different patient information, and the HBase divides the index data and the common data into the same Region according to different data types.
Optionally, the prePut method specifically includes: and acquiring data Rowkey and an insertion field of the inserted data, inquiring whether the index configuration file establishes an index on the field, if so, generating the index Rowkey according to the index Rowkey structure in the step S3, inserting the index data into the index area, and if not, inserting the data into the HBase data area without establishing the index.
Optionally, the data insertion manner in step S4 is specifically:
structured data insertion: inserting data into each MySQL table according to the identity ID of each patient;
the image data insertion comprises the following specific steps:
s1: analyzing the DCM file by using a DCM4CHE framework, extracting metadata information of a patient, and inserting the metadata information into the m _ data of the column family;
s2: merging the DCM small files into a sequence File file according to the series UID;
s3: merging the sequence File files into a MapFile file according to the StudyUID, and setting the index interval to be 1, namely adding an index to each sequence File in the MapFile;
s4: and storing the MapFile into the HDFS, updating a location column corresponding to a p _ data column family of the HBase, and identifying the position of the image file in the HDFS.
The text data insertion comprises the following specific steps:
s1: analyzing the XML-format medical text file by using DOM4J, extracting metadata information of a patient, and inserting the metadata information into the column family t _ data;
s2: merging the medical text files into a MapFile according to the identity ID of the patient, and setting a fixed-length index interval;
s3: and storing the MapFile into the HDFS, updating a text column corresponding to the t _ data column family of the HBase, and identifying the position of the medical text file in the HDFS.
Optionally, the system comprises an ETL module, a data storage module, a data management module, and a retrieval module;
the extraction conversion loading ETL module is in signal connection with the data storage module;
the data storage module is in signal connection with the retrieval module;
the retrieval module is in signal connection with the data management module;
the extraction conversion loading ETL module is used for extracting and converting medical health data according to different medical protocol exchange standards and loading the medical health data into a database;
the data storage module is used for managing structured and unstructured data and providing a visual Web interface for data and system operation and management;
the retrieval module is used for constructing a database index, maintaining an index configuration file, and simultaneously performing data interaction with a Client terminal to complete a retrieval task.
The invention has the beneficial effects that: according to the medical health data storage and retrieval method and system, firstly, different types of databases can be used for storing by combining different medical health data structures and expression forms, structured data are stored by adopting MySQL data with high performance and reliability, hadoop is selected for massive text and image data to be stored, and a retrieval module is used for adapting and communicating data requirements of Client terminals, so that the management efficiency of the medical health data is improved; then, a scheme of combining sequence File and MapFile by Hadoop is provided for a large number of small files, so that the problem of low efficiency of small files stored under Hadoop is solved; finally, a two-stage index scheme applicable to HBase is provided, on the basis that indexes and data are placed in the same Region under the same Region server, the efficiency of data acquisition is improved by utilizing parallel asynchronous query of a coprocessor on each Region server, and meanwhile, remote RPC requests can be reduced by utilizing a proxy component of the Region server, so that IO (input/output) resources are saved. In conclusion, the method improves the management and retrieval efficiency of the medical health data under the conditions of isomerism and mass, and has strong application value in the aspects of use and research of the medical health data.
Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention. The objectives and other advantages of the invention may be realized and attained by the means of the instrumentalities and combinations particularly pointed out hereinafter.
Drawings
For the purposes of promoting a better understanding of the objects, aspects and advantages of the invention, reference will now be made to the following detailed description taken in conjunction with the accompanying drawings in which:
FIG. 1 is a flow chart of medical health data storage;
FIG. 2 is a flow chart of medical health data retrieval;
FIG. 3 is a block diagram of data flow of HBase data retrieval;
fig. 4 is a block diagram of the overall structure of the medical health data storage and retrieval system.
Detailed Description
The embodiments of the present invention are described below with reference to specific embodiments, and other advantages and effects of the present invention will be easily understood by those skilled in the art from the disclosure of the present specification. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention. It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present invention in a schematic way, and the features in the following embodiments and examples may be combined with each other without conflict.
Wherein the showings are for the purpose of illustrating the invention only and not for the purpose of limiting the same, and in which there is shown by way of illustration only and not in the drawings in which there is no intention to limit the invention thereto; for a better explanation of the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of an actual product; it will be understood by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted.
The same or similar reference numerals in the drawings of the embodiments of the present invention correspond to the same or similar components; in the description of the present invention, it should be understood that if there is an orientation or positional relationship indicated by the terms "upper", "lower", "left", "right", "front", "rear", etc., based on the orientation or positional relationship shown in the drawings, it is only for convenience of description and simplification of description, but it is not intended to indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore the terms describing the positional relationship in the drawings are only used for illustrative purposes and are not to be construed as limiting the present invention, and the specific meaning of the terms described above will be understood by those skilled in the art according to the specific circumstances.
Referring to fig. 1 to 4, a method and a system for storing and retrieving heterogeneous medical health data are shown.
Fig. 1 is a medical health data storage process, which includes data storage and index creation, and specifically includes the following steps:
s11: and establishing MySQL and HBase database tables, wherein the MySQL tables are divided into different tables according to the patient information, and meanwhile, indexes are established for the high-frequency query fields in advance. The HBase divides the index data and the common data into the same Region according to the difference of the data types, and as shown in table 1, is a logical structure of the HBase database. The column under the column family m _ data stores metadata information of the patient, the column families p _ data and t _ data store metadata information of the image data related to the text data, respectively, and the column family index stores information related to the index. In the index area, only the column family index stores valid data, and in the data area, only the column families m _ data, t _ data and p _ data store valid data;
TABLE 1 HBase database logical Structure
S12: adopting a pre-partitioning method to partition regions for HBase, setting a Rowkey range of the regions, and forbidding automatic Split of the HBase;
s13: rowkey of the index area and the data area should have different structures, so as to achieve the effect of maximizing the utilization of the index. The data area Rowkey is composed of Hash and Rowkey, wherein the Hash is a Hash value generated by hashing in the Region of the whole HBase table, and the Rowkey is the original Rowkey when data is inserted; the index Rowkey is composed of StartKey + IndexKey + IndexValue + DataRowkey, wherein StartKey is the starting partition point of Region and is within the Rowkey range of Region; indexKey represents an index type and represents the names of single columns or joint indexes established aiming at different columns; indexValue represents the value of the index; the DataRowkey is data Rowkey, namely Hash + Rowkey;
s14: the ETL module extracts data to a database according to different medical protocol data exchange standards, wherein structured data are stored in MySQL, image data and medical text data are inserted into HDFS, and metadata are inserted into HBase;
s15: HBase receives a data insertion request of a Client end, a coprocessor captures a Put request, and a rewritten prePut method is executed: and acquiring data Rowkey and an insertion field of the inserted data, inquiring whether the index configuration file establishes an index on the field, if so, generating the index Rowkey according to the index Rowkey structure in S13, inserting the index data into the index area, and if not, inserting the data into the HBase data area without establishing the index.
Further, S14 is described separately for different structural data, where the specific steps involved are as follows:
structured data insertion: data is inserted into the various tables according to the identity ID of each patient.
The unstructured data comprises image data and medical text data, and the image data and the medical text data are usually massive and small in file size, so that the small files need to be combined in order to be stored in Hadoop and reduce performance loss in MapReduce calculation.
The image data insertion comprises the following specific steps:
s141: analyzing the DCM file by using a DCM4CHE framework, extracting metadata information of a patient, and storing the metadata information into the m _ data of the column family;
s142: the same series UID of the DCM file represents one check of the patient, and is the minimum merging unit, so the DCM small files are merged into a sequence File file according to the series UID of the patient, and the files are merged in a non-compression mode for the view definition;
s143: merging the sequence File files into a MapFile file according to the StudyUID, and setting an index interval to be 1, namely generating an index file for each sequence File so as to facilitate subsequent retrieval and improve the efficiency;
s144: and storing the file into the HDFS, updating a location column corresponding to a p _ data column family of the HBase, and identifying the position of the image file in the HDFS.
The text data insertion comprises the following specific steps:
s141: analyzing the XML-format medical text file by using DOM4J, extracting metadata information of a patient, and inserting the metadata information into the column family t _ data;
s142: the medical text files are combined into a MapFile according to the identity ID of the patient, and index intervals with fixed lengths are set, so that the MapFile index occupies less memory, and the query efficiency is guaranteed;
s143: and storing the file into the HDFS, updating a text column corresponding to the t _ data column family of the HBase, and identifying the position of the medical text file in the HDFS.
Further, the retrieval method of medical health data based on coprocessor EndPoint, as shown in fig. 2, includes the following steps:
s21: creating a Protubuf file, and customizing an RPC request method and a response method;
s22: the retrieval module judges the query requirement of the Client terminal, and if the query data is structured data, the S23 is carried out; if the query data is unstructured data, entering S24; if the data is of the heterogeneous type, asynchronously acquiring the data by using a multithreading mode, and respectively executing S23 and S24-S210;
s23: querying MySQL according to conditions, and entering S211 after the query is completed;
s24: the retrieval module queries the index configuration file according to the query field of the Client, matches the optimal index and generates a query condition expression;
s25: the retrieval module takes the index type IndexKey and the query condition expression as RPC request method parameters, then requests to each RegionServer in parallel, and the RegionServer performs query operation on each Region by using the agent component;
s26: the query operation at each Region is: firstly, acquiring StartKey of a current Region, splicing the StartKey and an index type to be used as an index prefix, splicing a query range boundary value of a first field of a query conditional expression and the index prefix to be used as StartRow and StopRow of Scan, and then performing Scan query on the current Region;
s27: AND if the Scan result set is not empty AND the subsequent conditional operator is AND, screening the subsequent field value according to the query conditional expression, extracting DataRowkey, AND entering S29. If the subsequent condition operator is OR, extracting DataRowkey in the Scan data set, and entering S29;
s28: if the Scan result set is null AND the subsequent condition operator is AND, then a null result is returned AND S210 is entered. If the subsequent conditional operator is OR, performing full Scan operation in the index Region in the current Region, screening the value of the subsequent field, and extracting DataRowkey;
s29; iterating the DataRowkey data set, redirecting back to the current Region, and performing Get operation according to the DataRowkey to obtain data in the data area;
s210: the retrieval module receives a non-empty result responded by one RPC request, then the step directly enters S211, and then the retrieval module continuously waits for response results of other RPC requests;
s211: and packaging the result according to the query requirement and returning the result to the client through the interface. Fig. 3 shows the data flow during data retrieval.
It is another object of the present invention to provide a heterogeneous medical health data storage and retrieval system, as shown in fig. 4. The method comprises the steps of storing data of different structure types by adopting different schemes, storing structured data by adopting a MySQL database in the aspect of storage, storing image data and medical text data in unstructured data by adopting HDFS (Hadoop distributed file system), storing metadata information of the text data and the image data by adopting HBase, and constructing an index for the database, particularly constructing and inquiring a secondary index of the HBase to improve the retrieval efficiency under a big data environment. The system mainly comprises the following modules:
an ETL module: the system is used for extracting and converting medical health data according to different medical protocol exchange standards and loading the medical health data into a database;
a data management module: the system is used for managing structured and unstructured data and providing a visual Web interface for data and system operation and management;
the retrieval module: the method is used for constructing the database index, maintaining the index configuration file, and simultaneously performing data interaction between the retrieval module and the Client terminal to complete the retrieval task.
Finally, the above embodiments are only intended to illustrate the technical solutions of the present invention and not to limit the present invention, and although the present invention has been described in detail with reference to the preferred embodiments, it will be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions, and all of them should be covered by the claims of the present invention.
Claims (5)
1. A heterogeneous medical health data storage and retrieval method is characterized by comprising the following steps: the method comprises a heterogeneous medical health data storage method and a heterogeneous medical health data retrieval method;
the heterogeneous medical health data storage method comprises the following steps:
s11: establishing MySQL and HBase database tables;
s12: dividing regions for the HBase by adopting a pre-partitioning method, setting the Rowkey range of the regions, and forbidding the automatic Split of the HBase;
s13: setting different types of Rowkeys to have different composition structures, wherein the composition of the data Rowkey is Hash + Rowkey, the Hash is a Hash value generated by hashing in the Region of the whole HBase table, and the Rowkey is the original Rowkey when the data is inserted; the index Rowkey consists of StartKey + IndexKey + IndexValue + DataRowkey, wherein the StartKey is a starting partition point of Region, indexValue represents the value of the index, and the DataRowkey is the data Rowkey, namely Hash + Rowkey;
s14: the extraction, conversion and loading ETL module extracts data to a database according to different medical protocol data exchange standards, wherein structured data is inserted into MySQL, image data and medical text data are inserted into HDFS, and metadata is inserted into HBase;
s15: HBase receives a data insertion request of a Client end, a coprocessor captures a Put request, and a rewritten prePut method is executed;
the heterogeneous medical health data retrieval method comprises the following steps:
s21: creating a Protubuf file, and customizing an RPC request method and a response method;
s22: the retrieval module judges the query requirement of the Client terminal, and if the query data is structured data, the S23 is carried out; if the query data is unstructured data, entering S24; if the type is heterogeneous, using a multithreading mode to asynchronously acquire data, and respectively executing S23 and S24-S210;
s23: querying MySQL according to conditions, and entering S211 after the query is completed;
s24: the retrieval module queries the index configuration file according to the query field of the Client, matches the optimal index and generates a query condition expression;
s25: the retrieval module takes the index type IndexKey and the query condition expression as RPC request method parameters, then requests to each RegionServer in parallel, and the RegionServer performs query operation on each Region by using the agent component;
s26: the query operation at each Region is: firstly, acquiring StartKey of a current Region, splicing the StartKey and an index type to be used as an index prefix, splicing a query range boundary value of a first field of a query conditional expression and the index prefix to be used as StartRow and StopRow of Scan, and then performing Scan query on the current Region;
s27: if the Scan result set is not empty AND the subsequent conditional operator is AND, screening the subsequent field value according to the query conditional expression, extracting DataRowkey, AND entering S29; if the subsequent condition operator is OR, extracting DataRowkey in the Scan data set, and entering S29;
s28: if the Scan result set is empty AND the subsequent conditional operator is AND, returning an empty result AND entering S210; if the subsequent conditional operator is OR, performing full Scan operation in the index Region in the current Region, screening the value of the subsequent field, and extracting DataRowkey;
s29; iterating the DataRowkey data set, redirecting back to the current Region, and performing Get operation according to the DataRowkey to obtain data in the data area;
s210: the retrieval module receives a non-empty result responded by one RPC request, then the step directly enters S211, and then the retrieval module continuously waits for response results of other RPC requests;
s211: and packaging the result according to the query requirement and returning the result to the client through the interface.
2. The heterogeneous medical health data storage and retrieval method of claim 1, wherein: the MySQL table is divided into different tables according to different patient information, and the HBase divides index data and common data into the same Region according to different data types.
3. The heterogeneous medical health data storage and retrieval method of claim 1, wherein: the prePut method specifically comprises the following steps: and acquiring data Rowkey and an insertion field of the inserted data, inquiring whether the index configuration file establishes an index on the field, if so, generating the index Rowkey according to the index Rowkey structure in the step S3, inserting the index data into the index area, and if not, inserting the data into the HBase data area without establishing the index.
4. The method of claim 1, wherein the heterogeneous medical health data storage and retrieval method comprises: the data insertion method in step S4 specifically includes:
structured data insertion: inserting data into each MySQL table according to the identity ID of each patient;
the image data insertion comprises the following specific steps:
s1: analyzing the DCM file by using a DCM4CHE framework, extracting metadata information of a patient, and inserting the metadata information into the m _ data of the column family;
s2: merging the DCM small files into a sequence File file according to the series UID;
s3: merging the sequence File files into a MapFile file according to the StudyUID, and setting the index interval to be 1, namely adding an index to each sequence File in the MapFile;
s4: storing the MapFile into the HDFS, updating a location column corresponding to a p _ data column family of the HBase, and identifying the position of the image file in the HDFS;
the text data insertion comprises the following specific steps:
s1: analyzing the XML-format medical text file by using DOM4J, extracting metadata information of a patient, and inserting the metadata information into the column family t _ data;
s2: merging the medical text files into a MapFile according to the identity ID of the patient, and setting a fixed-length index interval;
s3: and storing the MapFile into the HDFS, updating a text column corresponding to the t _ data column family of the HBase, and identifying the position of the medical text file in the HDFS.
5. The heterogeneous medical health data storage and retrieval system based on the method of any one of claims 1 to 4, wherein: the system comprises an ETL module, a data storage module, a data management module and a retrieval module;
the extraction conversion loading ETL module is in signal connection with the data storage module;
the data storage module is in signal connection with the retrieval module;
the retrieval module is in signal connection with the data management module;
the extraction, conversion and loading ETL module is used for extracting and converting medical health data according to different medical protocol exchange standards and loading the medical health data into a database;
the data storage module is used for managing structured and unstructured data and providing a visual Web interface for data and system operation and management;
the retrieval module is used for constructing a database index, maintaining an index configuration file, and simultaneously performing data interaction with a Client terminal to complete a retrieval task.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110087530.4A CN112765131B (en) | 2021-01-22 | 2021-01-22 | Heterogeneous medical health data storage and retrieval method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110087530.4A CN112765131B (en) | 2021-01-22 | 2021-01-22 | Heterogeneous medical health data storage and retrieval method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112765131A CN112765131A (en) | 2021-05-07 |
| CN112765131B true CN112765131B (en) | 2023-03-24 |
Family
ID=75705671
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110087530.4A Active CN112765131B (en) | 2021-01-22 | 2021-01-22 | Heterogeneous medical health data storage and retrieval method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112765131B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116303447A (en) * | 2023-02-28 | 2023-06-23 | 北京北明数科信息技术有限公司 | Health information data management method and system |
| CN117594198B (en) * | 2024-01-18 | 2024-04-05 | 万里云医疗信息科技（北京）有限公司 | Medical image record retrieval method and device based on heterogeneous data |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104145472A (en) * | 2012-02-29 | 2014-11-12 | 瑞典爱立信有限公司 | Method and apparatus for storage of data records |
| WO2016180123A1 (en) * | 2015-09-25 | 2016-11-17 | 中兴通讯股份有限公司 | Hbase second-level index creation method and device |
| CN106528674A (en) * | 2016-10-31 | 2017-03-22 | 厦门服云信息科技有限公司 | Method and device for high-performance query based on Hbase row keys |
| CN106777258A (en) * | 2016-12-28 | 2017-05-31 | 银江股份有限公司 | The coding and compression method of Hbase line units in a kind of medical big data storage |
| CN106815338A (en) * | 2016-12-25 | 2017-06-09 | 北京中海投资管理有限公司 | A kind of real-time storage of big data, treatment and inquiry system |
| CN109165222A (en) * | 2018-08-20 | 2019-01-08 | 福州大学 | A kind of HBase secondary index creation method and system based on coprocessor |
| KR102014267B1 (en) * | 2018-11-15 | 2019-08-26 | 국방과학연구소 | Method and Apparatus of building Inverse Index DB for high speed searching of moving picture object |
| CN110941618A (en) * | 2019-11-27 | 2020-03-31 | 河钢数字技术股份有限公司 | Mass heterogeneous data storage method and system |
| CN112131295A (en) * | 2020-09-27 | 2020-12-25 | 平安医疗健康管理股份有限公司 | Data processing method and device based on Elasticissearch |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105740405B (en) * | 2016-01-29 | 2020-06-26 | 华为技术有限公司 | Method and device for storing data |
-
2021
- 2021-01-22 CN CN202110087530.4A patent/CN112765131B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104145472A (en) * | 2012-02-29 | 2014-11-12 | 瑞典爱立信有限公司 | Method and apparatus for storage of data records |
| WO2016180123A1 (en) * | 2015-09-25 | 2016-11-17 | 中兴通讯股份有限公司 | Hbase second-level index creation method and device |
| CN106528674A (en) * | 2016-10-31 | 2017-03-22 | 厦门服云信息科技有限公司 | Method and device for high-performance query based on Hbase row keys |
| CN106815338A (en) * | 2016-12-25 | 2017-06-09 | 北京中海投资管理有限公司 | A kind of real-time storage of big data, treatment and inquiry system |
| CN106777258A (en) * | 2016-12-28 | 2017-05-31 | 银江股份有限公司 | The coding and compression method of Hbase line units in a kind of medical big data storage |
| CN109165222A (en) * | 2018-08-20 | 2019-01-08 | 福州大学 | A kind of HBase secondary index creation method and system based on coprocessor |
| KR102014267B1 (en) * | 2018-11-15 | 2019-08-26 | 국방과학연구소 | Method and Apparatus of building Inverse Index DB for high speed searching of moving picture object |
| CN110941618A (en) * | 2019-11-27 | 2020-03-31 | 河钢数字技术股份有限公司 | Mass heterogeneous data storage method and system |
| CN112131295A (en) * | 2020-09-27 | 2020-12-25 | 平安医疗健康管理股份有限公司 | Data processing method and device based on Elasticissearch |
Non-Patent Citations (3)
| Title |
|---|
| Feasibility analysis of big log data real time search based on Hbase and ElasticSearch;Jun Bai;《2013 Ninth International Conference on Natural Computation (ICNC)》;20140519;1166-1170 * |
| 区域健康医疗大数据平台解决方案策划与初步设计;李伟等;《医疗卫生装备》;20180715;第39卷(第07期);34-40 * |
| 慢性病健康监护系统中间件的设计与实现;李锦;《中国优秀硕士学位论文全文数据库 (医药卫生科技辑)》;20220315(第03期);E080-169 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112765131A (en) | 2021-05-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109800222B (en) | HBase secondary index self-adaptive optimization method and system | |
| Li et al. | A storage solution for massive IoT data based on NoSQL | |
| CN110032604B (en) | Data storage device, translation device and database access method | |
| CN112765131B (en) | Heterogeneous medical health data storage and retrieval method and system | |
| US8880463B2 (en) | Standardized framework for reporting archived legacy system data | |
| CN110427368A (en) | Data processing method, device, electronic equipment and storage medium | |
| CN111881223B (en) | Data management method, device, system and storage medium | |
| CN109086573B (en) | Multi-source biological big data fusion system | |
| CN106294695A (en) | A kind of implementation method towards the biggest data search engine | |
| CN104731945A (en) | Full-text searching method and device based on HBase | |
| US11468031B1 (en) | Methods and apparatus for efficiently scaling real-time indexing | |
| US9569441B2 (en) | Archival of objects and dynamic search | |
| CN110570928A (en) | HBase and ozone based medical image file access method | |
| CN112416960A (en) | Data processing method, device and equipment under multiple scenes and storage medium | |
| CN115576905A (en) | Archive file management method and device, electronic equipment and storage medium | |
| CN112148731A (en) | Data paging query method, device and storage medium | |
| CN110413807B (en) | Image query method and system based on content semantic metadata | |
| CN115587082A (en) | Multi-modal data storage management method and system | |
| CN111984745B (en) | Database field dynamic expansion method, device, equipment and storage medium | |
| EP3282372B1 (en) | Method and apparatus for storing data | |
| Ulrich et al. | Using Graph Tools on Metadata Repositories. | |
| US10185742B2 (en) | Flexible text searching for data objects of object notation | |
| CN110795448B (en) | Metadata management method and device and readable storage medium | |
| CN110740046B (en) | Method and device for analyzing service contract | |
| CN107247796B (en) | Client head portrait loading method and device and terminal equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230714 Address after: Room 801, 85 Kefeng Road, Huangpu District, Guangzhou City, Guangdong Province Patentee after: Yami Technology (Guangzhou) Co.,Ltd. Address before: 400065 Chongqing Nan'an District huangjuezhen pass Chongwen Road No. 2 Patentee before: CHONGQING University OF POSTS AND TELECOMMUNICATIONS Effective date of registration: 20230714 Address after: Room 601-5, Building 19, No. 785, Hutai Road, Jing'an District, Shanghai, 200040 Patentee after: Shanghai Canming Pharmaceutical Technology Co.,Ltd. Address before: Room 801, 85 Kefeng Road, Huangpu District, Guangzhou City, Guangdong Province Patentee before: Yami Technology (Guangzhou) Co.,Ltd. |