CN112463765A - Medical data management method and system based on big data framework - Google Patents
Medical data management method and system based on big data framework Download PDFInfo
- Publication number
- CN112463765A CN112463765A CN202011413510.3A CN202011413510A CN112463765A CN 112463765 A CN112463765 A CN 112463765A CN 202011413510 A CN202011413510 A CN 202011413510A CN 112463765 A CN112463765 A CN 112463765A
- Authority
- CN
- China
- Prior art keywords
- data
- platform
- big data
- big
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 85
- 238000013523 data management Methods 0.000 title claims abstract description 41
- 238000004458 analytical method Methods 0.000 claims abstract description 37
- 230000008569 process Effects 0.000 claims abstract description 19
- 238000007726 management method Methods 0.000 claims abstract description 16
- 238000012544 monitoring process Methods 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 13
- 238000012545 processing Methods 0.000 claims description 12
- 230000000007 visual effect Effects 0.000 claims description 8
- 238000002955 isolation Methods 0.000 claims description 7
- 238000007405 data analysis Methods 0.000 claims description 5
- 238000007418 data mining Methods 0.000 claims description 5
- 230000015572 biosynthetic process Effects 0.000 claims description 4
- 239000008280 blood Substances 0.000 claims description 4
- 210000004369 blood Anatomy 0.000 claims description 4
- 238000013499 data model Methods 0.000 claims description 4
- 238000013524 data verification Methods 0.000 claims description 4
- 238000005553 drilling Methods 0.000 claims description 4
- 238000007781 pre-processing Methods 0.000 claims description 4
- 238000003786 synthesis reaction Methods 0.000 claims description 4
- 230000010354 integration Effects 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 8
- 238000003759 clinical diagnosis Methods 0.000 description 4
- 238000013480 data collection Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000004141 dimensional analysis Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000013479 data entry Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000005111 flow chemistry technique Methods 0.000 description 1
- 230000004907 flux Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- General Business, Economics & Management (AREA)
- Public Health (AREA)
- Quality & Reliability (AREA)
- Primary Health Care (AREA)
- Medical Informatics (AREA)
- General Health & Medical Sciences (AREA)
- Epidemiology (AREA)
- Business, Economics & Management (AREA)
- Biomedical Technology (AREA)
- Medical Treatment And Welfare Office Work (AREA)
Abstract
The application relates to a medical data management method, a medical data management system, a computer device and a storage medium based on a big data framework, wherein the method comprises the following steps: a data lake in data is set up, all collected data, analysis process data, analysis models, analysis result data and operation scheduling are managed, and the analysis process of all data on the whole platform is supported to be smoothly carried out; building a big data acquisition platform; building a big data computing platform; building a big data service platform; and building a big data exchange platform and a big data application platform. The invention optimizes the quality of the whole hospital data by establishing a uniform data integration standard and establishes a data decision basis to improve the competitiveness of the hospital. The large data computing platform, the service platform and the application platform provide a full-flow and refined management analysis result of each service for a manager, and the hospital management decision is promoted to be scientific and refined.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a medical data management method and system based on a big data framework, computer equipment and a storage medium.
Background
With the development of computer technology, computer technology has been widely used in various fields, and since computer technology can efficiently perform operation management on various information, efficient data management can be effectively provided.
However, many medical data centers, such as CDRs, that are available today are more and more used for data queries only, the value of the data is not released, and the saved data is more biased towards clinical data, and the operational data involves less. Therefore, the traditional medical data center does not integrate the whole hospital data to improve the accessibility, the availability and the credibility of hospital data services, and fails to fully release the data value and has less data involvement, so that the problem source cannot be quickly traced when a problem occurs, and the decision support cannot be quickly and timely provided for the hospital.
Disclosure of Invention
In view of the above, there is a need to provide a big data architecture-based medical data management method, system, computer device and storage medium that can improve accessibility, availability and credibility of hospital data services.
A method of medical data management based on a big data architecture, the method comprising:
a data lake in data is set up, all collected data, analysis process data, analysis models, analysis result data and operation scheduling are managed, and the analysis process of all data on the whole platform is supported to be smoothly carried out;
building a big data acquisition platform, wherein the big data acquisition platform expands different read-write plug-ins aiming at data interfaces created by different data types based on an open-source heterogeneous data synchronization engine, and stores acquired data to a big data center in a classified manner after preprocessing;
building a big data computing platform, wherein the big data computing platform stores the acquired data in a hybrid storage mode, and provides data analysis and mining for the stored data by using a Spark computing engine;
the method comprises the steps that a big data service platform is built, the big data service platform is used for carrying out unified management and monitoring on information of a big data center, and the information of the big data center comprises the total number of a database, the total number of data, the total capacity, a basic table and the integral information of a synthetic table;
building a big data exchange platform, wherein the big data exchange platform uniformly manages information exchange between a big data center of a hospital and an in-hospital business system and other information platforms outside the hospital;
and building a big data application platform, wherein the big data application platform is used for directly inquiring the information of the big data center and displaying the analysis result in a plurality of different charts.
In one embodiment, the big data collection platform is further configured to:
the method supports the collection of data sources of different types of databases including Mysql, Postgresql, Clickhouse, Oracle and Sqlserver;
supporting writing collected data into hive, Mysql, Postgresql, Clickhouse, Oracle and Sqlserver;
the read-write speed is supported, and the read-write task is prevented from occupying too many server resources and influencing service application;
the method comprises the steps that a data source acquired through a visual interface is appointed, a data table needing to be extracted is selected in a check mode to preview the data table, and a data range needing to be extracted is locked through data fields under the appointed data table in a check mode;
the method supports self-defining setting of the primary key for the data source table without the primary key, and the big data center removes the duplicate according to the primary key;
the method supports the fine setting of the acquisition conditions on the basis of the established data range by utilizing the SQL sentence;
and the mode of setting the workflow through a visual interface is supported, and the whole flow of data acquisition and processing is directly set.
In one embodiment, the big data service platform is further configured to:
the method supports that a new synthesis table is rapidly generated by compiling the SQL statement on the basis of the basic table, and the blood relationship between the synthesis table and the basic table can be checked;
unified monitoring on scheduling tasks initiated by a data center is supported, and the overall situation is displayed in a chart form;
the method supports the timely early warning of illegal operation in the data acquisition process;
the data verification and data monitoring of the collected data are supported, and the accuracy of the data center information is guaranteed.
In one embodiment, the big data application platform is further configured to:
previewing data from a data source by compiling an SQL query statement, and dragging and pulling to generate a data model;
supporting the simultaneous drilling of a plurality of data and the direct tracing of generating SQL statistical statements of the report on the interface;
the access of various data sources and various charts is supported;
the user-defined data isolation is supported, the data isolation is controlled by taking the dimension as the minimum granularity, and statistical data can be filtered according to different users.
In one embodiment, the step of constructing the data lake of the platform dataLake further comprises:
the API of the dataLake data lake provides interface service for the UI layer, and workflow configuration of system configuration, resource center, data processing and data management is stored and updated into a database;
the API layer obtains the meta-information of the butted data source, stores the meta-information in the es full-text retrieval engine, facilitates subsequent rapid positioning of data assets, message pushing and cluster monitoring, and the Master and the Worker nodes can register temporary nodes on the ZooKeeper when deployment and creation are carried out, and store server information in the heartbeat package to the temporary nodes.
In one embodiment, the Master cluster supports a high availability configuration, and a plurality of daemon threads are maintained in the Master cluster and used for monitoring the states of other masters and Workers.
In one embodiment, the Worker cluster is used for acquiring and executing the task to be executed from the distributed queue, and updating the execution result into the database.
A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of any of the above methods when executing the computer program.
A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of any of the methods described above.
The medical data management method, the medical data management system, the computer equipment and the storage medium based on the big data framework optimize the quality of the whole hospital data by establishing a uniform data integration standard and establish a data decision basis to improve the competitiveness of the hospital. The big data computing platform, the service platform and the application platform provide a full-flow and refined management analysis result of each service for a manager, so that the scientification and refinement of hospital management decisions are promoted, and clinical diagnosis and treatment decision support can be provided, so that the scientific research level of the hospital is improved. In addition, interconnection and intercommunication between the hospital and the regional platform and between external institutions and data sharing and cooperation under safety control can be realized through the big data exchange platform.
Drawings
FIG. 1 is a flow diagram illustrating a method for big data architecture based medical data management in one embodiment;
FIG. 2 is a flow chart illustrating a method for medical data management based on big data architecture in another embodiment;
FIG. 3 is a flow chart illustrating a method for medical data management based on big data architecture in a further embodiment;
FIG. 4 is a flow chart of a medical data management method based on big data architecture in yet another embodiment;
FIG. 5 is an overall architecture diagram of a big data architecture based medical data management system in one embodiment;
FIG. 6 is a platform architecture diagram of a big data acquisition platform in one embodiment;
FIG. 7 is a platform architecture diagram of a big data computing platform, under an embodiment;
FIG. 8 is a block diagram of a big data architecture based medical data management system in one embodiment;
FIG. 9 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
Currently, many medical data centers, such as CDRs, which are available today are more and more used only for data query, the value of the data is not released, and the saved data is more biased towards clinical data and the operational data is less involved. Therefore, the traditional medical data center does not integrate the whole hospital data to improve the accessibility, the availability and the credibility of hospital data services, and fails to fully release the data value and has less data involvement, so that the problem source cannot be quickly traced when a problem occurs, and the decision support cannot be quickly and timely provided for the hospital.
Based on the method, the medical data center based on the big data framework is provided. Specifically, the data center integrates the whole yard data, provides whole-process processing and monitoring from the extraction of the whole data to the final data application and exchange, and can quickly trace the root of a problem when the problem occurs; the method has the advantages of getting through the information sharing and cooperation inside and outside the hospital, improving the utilization rate of data, providing data support for clinical diagnosis and treatment, and promoting the scientization and refinement of hospital management decision.
In one embodiment, as shown in fig. 1, there is provided a medical data management method based on big data architecture, the method comprising:
102, establishing a dataLake data base in data, managing all collected data, analysis process data, analysis models, analysis result data and job scheduling, and supporting the smooth operation of the analysis process of all data on the whole platform;
104, building a big data acquisition platform, expanding different read-write plug-ins aiming at data interfaces created by different data types and storing acquired data to a big data center in a classified manner after preprocessing the acquired data, wherein the big data acquisition platform is based on an open-source heterogeneous data synchronization engine;
106, building a big data computing platform, storing the acquired data in a hybrid storage mode by the big data computing platform, and providing data analysis and mining for the stored data by utilizing a Spark Streaming computing tool;
step 108, a big data service platform is set up, the big data service platform carries out unified management and monitoring on the information of the big data center, and the information of the big data center comprises the total number of the database, the total number of the data, the total capacity, the basic table and the integral information of the synthetic table;
step 110, a big data exchange platform is set up, and the big data exchange platform manages information exchange between a big data center of a hospital and other information platforms in the hospital in a unified manner;
and 112, building a big data application platform, wherein the big data application platform is used for directly inquiring the information of the big data center and displaying the analysis result in a plurality of different charts.
In the present embodiment, a medical data management method based on a big data architecture is provided, which can be applied in a big data center as shown in fig. 5. Specifically, the large data center is designed by adopting a large-scale distributed computing platform and a distributed large data service cluster, the scale can be expanded to PB-level data processing, a large data system with high capacity, high flux, expandability and easy maintenance is realized, the rapid retrieval of mass data is supported, and the deep analysis of the data is realized by the following steps:
firstly, a data lake of a data middle platform is set up: managing all the collected data, the analysis process data, the analysis model, the analysis result data and the operation scheduling, and supporting the smooth operation of the analysis process of all the data on the whole platform. Specifically, the flow of collecting data + processing data + managing data + outputting data may be solidified on dataLake by adopting a data stage design method. The data center is convenient to process data, manage the data and improve the processing efficiency of output data.
Then, a unified big data acquisition platform is built, and for different data types, for example: the data synchronization method comprises structured data, text data, image data, video data and the like, a heterogeneous data synchronization engine based on an open source, data interfaces established aiming at different data types, different reading and writing plug-ins are expanded, and collected data are stored in a data center in a classified manner after being preprocessed. In addition, for information which cannot be directly extracted through a service system, the integrity of the data center information is ensured by adopting a data entry mode.
Next, by building a big data computing platform, the platform architecture of the big data computing platform is shown in FIG. 6. Specifically, the acquired data is stored in a hybrid storage manner. And for the stored data, a computing tool mainly based on Spark Streaming is adopted, and various data analysis and mining are provided according to different data applications. In addition, the data calculation tool can also acquire data from the data storage library, and after calculation results are obtained, the data are fed back to the data storage library on one hand, and on the other hand, various analysis results are displayed on the big data application platform through the data display tool.
And then, realizing unified management and monitoring of the medical big data center information by building a big data service platform, wherein the unified management and monitoring comprises the total database number, the total data number, the total capacity, the basic table, the integral information of the synthetic table and the like contained in the data center.
And finally, constructing a big data exchange platform for uniformly managing a big data center in a hospital and various platforms inside and outside the hospital, for example: information exchange between internet hospital platforms, regional information platforms, and other institutions, etc. And a big data application platform is set up, so that the information of the big data center can be directly inquired, and the analysis result is displayed by abundant and various graphs.
In the embodiment, the quality of the whole hospital data is optimized and a data decision basis is established to improve the competitiveness of the hospital by establishing unified data integration specifications and standards. The big data computing platform, the service platform and the application platform provide a full-flow and refined management analysis result of each service for a manager, so that the scientification and refinement of hospital management decisions are promoted, and clinical diagnosis and treatment decision support can be provided, so that the scientific research level of the hospital is improved. In addition, interconnection and intercommunication between the hospital and the regional platform and between external institutions and data sharing and cooperation under safety control can be realized through the big data exchange platform.
In one embodiment, as shown in fig. 2, a big data architecture-based medical data management method is provided, in which a big data acquisition platform is further configured to:
step 202, supporting the collection of data sources of different types of databases including Mysql, Postgresql, Clickhouse, Oracle and Sqlserver;
step 204, supporting the appointed collected data source through a visual interface, previewing the data table through checking the data table needing to be extracted, and locking the range of the data to be extracted through checking the data field under the appointed data table;
step 206, supporting the user-defined setting of a primary key for a data source table without the primary key, and removing the duplicate according to the primary key by the big data center;
step 208, supporting the refinement and setting of acquisition conditions on the basis of the established data range by utilizing SQL statements;
step 210, supporting the mode of setting the workflow through the visual interface, and directly setting the whole flow of data acquisition and processing.
In this embodiment, a specific big data acquisition platform is built, and specifically, a platform frame diagram of the big data acquisition platform shown in fig. 7 may be referred to, where the big data acquisition platform has the following characteristics:
1. the method supports the collection of data sources of various mainstream databases in the market at present, including Mysql, Postgresql, Clickhouse, Oracle, Sqlserver and the like.
2. The data extraction method supports the data source appointed to be collected through a visual interface, the data table required to be extracted is selected in a checking mode, the data table is previewed, and the data range to be extracted is locked through the data field under the appointed data table in a checking mode.
3. The method supports the user-defined setting of the primary key for the data source table without the primary key, and the data center can remove the duplicate according to the primary key.
4. And further refining and setting acquisition conditions on the basis of the established data range by utilizing SQL sentences.
5. And the mode of setting the workflow through a visual interface is supported, and the whole flow of data acquisition and processing is directly set.
In the embodiment, the data source collection supporting various databases of different types is realized through the big data collection platform, the collected data can be preprocessed through functions such as preview screening, and the effectiveness of data collection is improved.
In one embodiment, as shown in fig. 3, there is provided a big data architecture-based medical data management method, in which the big data service platform is further configured to:
step 302, supporting to quickly generate a new synthetic table by writing SQL statements on the basis of the basic table, and checking the blood relationship between the synthetic table and the basic table;
step 304, supporting unified monitoring of scheduling tasks initiated by the data center, and displaying the overall situation in a chart form;
step 306, providing timely early warning when illegal operation occurs in the data acquisition process;
and 308, supporting data verification and data monitoring on the acquired data and ensuring the accuracy of the data center information.
In this embodiment, a specific big data service platform is built, and the big data service platform has the following characteristics:
1. the method supports the mode of writing SQL sentences on the basis of the basic table to quickly generate a new synthetic table and can check the blood relationship between the synthetic table and the basic table.
2. Unified monitoring on scheduling tasks initiated by the data center is supported, and the overall situation is displayed in the form of charts such as instrument panels.
3. And the early warning is provided in various modes such as in-station sending, short message or mailbox when illegal operation occurs in the data acquisition process.
4. The data verification and data monitoring of the collected data are supported, and the accuracy of the data center information is guaranteed.
In the embodiment, the big data service platform is used for realizing the unified management and monitoring of the big data center information, including the total database count, the total data count, the total capacity, the overall information of the basic table and the synthetic table, and the like, which are contained in the data center, thereby effectively improving the efficiency of data management and ensuring the accuracy of data.
In one embodiment, as shown in fig. 4, there is provided a big data architecture-based medical data management method, in which the big data application platform is further configured to:
step 402, previewing data from a data source by compiling an SQL query statement, and dragging and pulling to generate a data model;
step 404, supporting the simultaneous drilling of multiple data and the direct tracing of SQL statistical statements generating reports on the interface, wherein the vertical drilling of different levels can be performed through the reports;
step 406, supporting access of various data sources and various charts;
step 408, supporting custom data isolation, controlling data isolation with dimension as minimum granularity, and filtering statistical data according to different users.
In this embodiment, a specific big data application platform is built, and the big data application platform has the following characteristics:
1. the method supports data previewing from a data source by writing SQL query sentences, dragging and dragging to generate a data model, and sets dimensions and indexes to support that a multi-dimensional analysis model is set from extracted data attributes directly in a dragging mode, wherein the multi-dimensional analysis model comprises the dimensions and analysis indexes.
2. The report can be drilled up and down in different levels, multiple pieces of data can be drilled simultaneously, and the SQL statistical statement of the report can be directly traced on an interface, so that the result can be conveniently checked.
3. Support multiple data sources, multiple charts: access to a variety of data sources needs to be supported, for example: jdbc, elastic search, etc.
4. The data isolation is controlled by using the dimension as the minimum granularity, and statistical data can be filtered according to different users, such as: the data of chart statistics displayed to users of different departments only comprises the data of the department where the user is located.
In the embodiment, the information of the big data center is queried through the big data application platform, and the analysis result is shown in a rich and various chart.
In one embodiment, a medical data management method based on a big data framework is provided, and the method for building a dailake data lake in data further comprises the following steps:
the API of the dataLake data lake provides interface service for the UI layer, and workflow configuration of system configuration, resource center, data processing and data management is stored and updated into the database; the API layer obtains the meta-information of the butted data source, stores the meta-information in the es full-text retrieval engine, facilitates subsequent rapid positioning of data assets, message pushing and cluster monitoring, and the Master and the Worker nodes can register temporary nodes on the ZooKeeper when deployment and creation are carried out, and store server information in the heartbeat package to the temporary nodes.
Specifically, in this embodiment, a medical data management method based on a big data framework is provided, in which the dataLake data lake is designed according to the following concept:
firstly, interface service is provided for a UI layer through a DataLake-API, system configuration, a resource center, data processing, data management workflow configuration and the like are stored and updated to DB (mysql 8+), meanwhile, the API layer acquires meta information of a butted data source and stores the meta information to an es full-text retrieval engine, and therefore subsequent fast positioning of data assets, message pushing and cluster monitoring are facilitated. Specifically, ZooKeeper monitoring is that an API layer is directly connected with the ZooKeeper to acquire data stored in Master and Worker temporary nodes on the ZooKeeper, the Master and Worker nodes register the temporary nodes on zk when deployment and creation are carried out, and server information such as a memory, a CPU and the like are stored in the temporary nodes in a heartbeat package.
The Master supports the HA, namely only one of the HA is effective at the same time, maintains several kinds of daemon threads such as heartbeat, captures tasks to be executed from the DB and puts the tasks into a distributed queue, and can be realized by using a ZooKeeper to monitor states of other masters and Worker.
The main responsibility of the Worker cluster is to acquire and execute the task to be executed from the distributed queue, update the execution result to the DB, and facilitate the UI layer to call the API to acquire the task state or trigger the alarm service. In order to ensure that the Worker cannot repeatedly execute the same task, the distributed lock is used by the product, and the cluster data consistency is ensured.
In the above embodiments, the extraction from the whole data to the final data application and exchange is designed, and the full flow processing and monitoring is provided, which has the following advantages:
1. the hospital-wide data quality is optimized by establishing a uniform data integration standard and standard, a data decision basis is established, and the hospital competitiveness is improved.
2. The method provides the full-flow and refined management analysis results of all services for the manager, and promotes the scientification and refinement of hospital management decisions.
3. The system provides clinical diagnosis and treatment decision support and helps to improve the scientific research level of hospitals.
4. And a unified data reporting standard and an exchange standard are formulated, and data sharing and cooperation under interconnection and intercommunication, safety control between a hospital and a regional platform and between a hospital and an external mechanism are realized.
It should be understood that although the various steps in the flow charts of fig. 1-7 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 1-7 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternating with other steps or at least some of the sub-steps or stages of other steps.
In one embodiment, as shown in FIG. 8, a big data architecture based medical data management system 800 is provided, the system comprising:
a dataLake data lake 801, which contains all the collected data, the analysis process data, the analysis models and the analysis result data, and is used for supporting the analysis processes of all the data on each platform;
the big data acquisition platform 802 is used for creating standard data interfaces aiming at different data types, adopting different data acquisition tools, preprocessing acquired data and storing the data to a big data center in different categories;
a big data computing platform 803, which stores the acquired data in a hybrid storage manner and provides data analysis and mining for the stored data by using a Spark Streaming computing tool;
the big data service platform 804 is used for uniformly managing and monitoring the information of the big data center, wherein the information of the big data center comprises the total number of the database, the total number of the data, the total capacity, the whole information of the basic table and the synthetic table;
a big data exchange platform 805 which manages information exchange between a big data center of a hospital and an in-hospital business system and other information platforms outside the hospital in a unified manner;
and a big data application platform 806, which is used for directly querying the information of the big data center and displaying the analysis result in a plurality of different charts.
Specifically, the big data framework-based medical data management method as described in any one of the above method embodiments may be performed in the big data framework-based medical data management system provided in the present embodiment.
For specific limitations of the big data framework-based medical data management system, reference may be made to the above limitations of the big data framework-based medical data management method, which are not described herein again.
In one embodiment, a computer device is provided, the internal structure of which may be as shown in FIG. 9. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a big data architecture based medical data management method.
Those skilled in the art will appreciate that the architecture shown in fig. 9 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the above method embodiments when executing the computer program.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the above respective method embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (9)
1. A medical data management method based on big data architecture, which is characterized by comprising the following steps:
a data lake in data is set up, all collected data, analysis process data, analysis models, analysis result data and operation scheduling are managed, and the analysis process of all data on the whole platform is supported to be smoothly carried out;
building a big data acquisition platform, wherein the big data acquisition platform expands different read-write plug-ins aiming at data interfaces created by different data types based on an open-source heterogeneous data synchronization engine, and stores acquired data to a big data center in a classified manner after preprocessing;
building a big data computing platform, wherein the big data computing platform stores the acquired data in a hybrid storage mode, and provides data analysis and mining for the stored data by using a Spark computing engine;
the method comprises the steps that a big data service platform is built, the big data service platform is used for carrying out unified management and monitoring on information of a big data center, and the information of the big data center comprises the total number of a database, the total number of data, the total capacity, a basic table and the integral information of a synthetic table;
building a big data exchange platform, wherein the big data exchange platform uniformly manages information exchange between a big data center of a hospital and an in-hospital business system and other information platforms outside the hospital;
and building a big data application platform, wherein the big data application platform is used for directly inquiring the information of the big data center and displaying the analysis result in a plurality of different charts.
2. The big-data framework-based medical data management method according to claim 1, wherein the big data acquisition platform is further configured to:
the method supports the collection of data sources of different types of databases including Mysql, Postgresql, Clickhouse, Oracle and Sqlserver;
supporting writing collected data into hive, Mysql, Postgresql, Clickhouse, Oracle and Sqlserver;
the read-write speed is supported, and the read-write task is prevented from occupying too many server resources and influencing service application;
the method comprises the steps that a data source acquired through a visual interface is appointed, a data table needing to be extracted is selected in a check mode to preview the data table, and a data range needing to be extracted is locked through data fields under the appointed data table in a check mode;
the method supports self-defining setting of the primary key for the data source table without the primary key, and the big data center removes the duplicate according to the primary key;
the method supports the fine setting of the acquisition conditions on the basis of the established data range by utilizing the SQL sentence;
and the mode of setting the workflow through a visual interface is supported, and the whole flow of data acquisition and processing is directly set.
3. The big data architecture-based medical data management method according to claim 1, wherein the big data service platform is further configured to:
the method supports that a new synthesis table is rapidly generated by compiling the SQL statement on the basis of the basic table, and the blood relationship between the synthesis table and the basic table can be checked;
unified monitoring on scheduling tasks initiated by a data center is supported, and the overall situation is displayed in a chart form;
the method supports the timely early warning of illegal operation in the data acquisition process;
the data verification and data monitoring of the collected data are supported, and the accuracy of the data center information is guaranteed.
4. The big-data framework-based medical data management method according to claim 1, wherein the big data application platform is further configured to:
previewing data from a data source by compiling an SQL query statement, and dragging and pulling to generate a data model;
supporting the simultaneous drilling of a plurality of data and the direct tracing of generating SQL statistical statements of the report on the interface;
the access of various data sources and various charts is supported;
the user-defined data isolation is supported, the data isolation is controlled by taking the dimension as the minimum granularity, and statistical data can be filtered according to different users.
5. The big data framework-based medical data management method according to any one of claims 1 to 4, wherein the step of building a Taidailake data lake in data further comprises the steps of:
the API of the dataLake data lake provides interface service for the UI layer, and workflow configuration of system configuration, resource center, data processing and data management is stored and updated into a database;
the API layer obtains the meta-information of the butted data source, stores the meta-information in the es full-text retrieval engine, facilitates subsequent rapid positioning of data assets, message pushing and cluster monitoring, and the Master and the Worker nodes can register temporary nodes on the ZooKeeper when deployment and creation are carried out, and store server information in the heartbeat package to the temporary nodes.
6. The big data framework-based medical data management method according to claim 5, wherein a Master cluster supports high availability configuration, and a plurality of daemon threads are maintained in the Master cluster for monitoring the states of other masters and Workers.
7. The medical data management method based on big data framework according to claim 6, characterized in that the Worker cluster is used for acquiring and executing tasks to be executed from the distributed queue, and updating the execution result into the database.
8. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 7 are implemented when the computer program is executed by the processor.
9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011413510.3A CN112463765A (en) | 2020-12-04 | 2020-12-04 | Medical data management method and system based on big data framework |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011413510.3A CN112463765A (en) | 2020-12-04 | 2020-12-04 | Medical data management method and system based on big data framework |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112463765A true CN112463765A (en) | 2021-03-09 |
Family
ID=74800100
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011413510.3A Pending CN112463765A (en) | 2020-12-04 | 2020-12-04 | Medical data management method and system based on big data framework |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112463765A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113192614A (en) * | 2021-04-22 | 2021-07-30 | 广州中康数字科技有限公司 | Medical information management system based on big data |
| CN113192624A (en) * | 2021-07-01 | 2021-07-30 | 京东方科技集团股份有限公司 | Emergency treatment system, emergency treatment method and electronic device |
| CN113836235A (en) * | 2021-09-29 | 2021-12-24 | 平安医疗健康管理股份有限公司 | Data processing method based on data center and related equipment thereof |
| CN115579094A (en) * | 2022-11-16 | 2023-01-06 | 神州医疗科技股份有限公司 | Multi-mode medical data lake construction method and system |
| CN115827922A (en) * | 2022-12-08 | 2023-03-21 | 华润网络(深圳)有限公司 | Visual analysis processing method and system based on wind power data and computer equipment |
| CN117909322A (en) * | 2024-03-19 | 2024-04-19 | 广东琴智科技研究院有限公司 | Unified data management application system and method based on intelligent computing platform |
| CN117909322B (en) * | 2024-03-19 | 2024-05-31 | 广东琴智科技研究院有限公司 | Unified data management application system and method based on intelligent computing platform |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110415831A (en) * | 2019-07-18 | 2019-11-05 | 天宜(天津)信息科技有限公司 | A kind of medical treatment big data cloud service analysis platform |
| CN111209269A (en) * | 2019-12-16 | 2020-05-29 | 云赛智联股份有限公司 | Big data management system of wisdom city |
-
2020
- 2020-12-04 CN CN202011413510.3A patent/CN112463765A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110415831A (en) * | 2019-07-18 | 2019-11-05 | 天宜(天津)信息科技有限公司 | A kind of medical treatment big data cloud service analysis platform |
| CN111209269A (en) * | 2019-12-16 | 2020-05-29 | 云赛智联股份有限公司 | Big data management system of wisdom city |
Non-Patent Citations (2)
| Title |
|---|
| 罗繁明;袁俊;赵恒煜;: "基于大数据的特色新型智库平台建设研究――以广东智库信息化平台为例", 情报资料工作, vol. 41, no. 05, 25 September 2020 (2020-09-25), pages 42 - 48 * |
| 许文韵: "健康医疗大数据中心建设实践与思考", 《医学信息学杂志》, vol. 41, no. 8, 25 August 2020 (2020-08-25), pages 49 - 50 * |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113192614B (en) * | 2021-04-22 | 2024-02-13 | 广州中康数字科技有限公司 | Medical information management system based on big data |
| CN113192614A (en) * | 2021-04-22 | 2021-07-30 | 广州中康数字科技有限公司 | Medical information management system based on big data |
| CN113192624A (en) * | 2021-07-01 | 2021-07-30 | 京东方科技集团股份有限公司 | Emergency treatment system, emergency treatment method and electronic device |
| CN113192624B (en) * | 2021-07-01 | 2022-05-31 | 京东方科技集团股份有限公司 | Emergency treatment system, emergency treatment method and electronic device |
| WO2023274402A1 (en) * | 2021-07-01 | 2023-01-05 | 京东方科技集团股份有限公司 | Emergency treatment system, emergency treatment method, and electronic device |
| CN113836235A (en) * | 2021-09-29 | 2021-12-24 | 平安医疗健康管理股份有限公司 | Data processing method based on data center and related equipment thereof |
| CN113836235B (en) * | 2021-09-29 | 2024-04-09 | 平安医疗健康管理股份有限公司 | Data processing method based on data center and related equipment thereof |
| CN115579094B (en) * | 2022-11-16 | 2023-02-28 | 神州医疗科技股份有限公司 | Multi-mode medical data lake construction method and system |
| CN115579094A (en) * | 2022-11-16 | 2023-01-06 | 神州医疗科技股份有限公司 | Multi-mode medical data lake construction method and system |
| CN115827922A (en) * | 2022-12-08 | 2023-03-21 | 华润网络(深圳)有限公司 | Visual analysis processing method and system based on wind power data and computer equipment |
| CN115827922B (en) * | 2022-12-08 | 2024-02-27 | 华润网络(深圳)有限公司 | Visual analysis processing method, system and computer equipment based on wind power data |
| CN117909322A (en) * | 2024-03-19 | 2024-04-19 | 广东琴智科技研究院有限公司 | Unified data management application system and method based on intelligent computing platform |
| CN117909322B (en) * | 2024-03-19 | 2024-05-31 | 广东琴智科技研究院有限公司 | Unified data management application system and method based on intelligent computing platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112463765A (en) | Medical data management method and system based on big data framework | |
| US11847574B2 (en) | Systems and methods for enriching modeling tools and infrastructure with semantics | |
| CN107918600B (en) | Report development system and method, storage medium and electronic equipment | |
| Rosen et al. | Commit guru: analytics and risk prediction of software commits | |
| Begoli et al. | Design principles for effective knowledge discovery from big data | |
| US8671084B2 (en) | Updating a data warehouse schema based on changes in an observation model | |
| JP6505123B2 (en) | Processing Data Sets in Big Data Repository | |
| CN111917887A (en) | System for realizing data governance under big data environment | |
| Suriarachchi et al. | Crossing analytics systems: a case for integrated provenance in data lakes | |
| CN114925045B (en) | PaaS platform for big data integration and management | |
| US11574075B2 (en) | Distributed machine learning technique used for data analysis and data computation in distributed environment | |
| Bao et al. | Activityspace: a remembrance framework to support interapplication information needs | |
| CN108829505A (en) | A kind of distributed scheduling system and method | |
| CN115374102A (en) | Data processing method and system | |
| Yadav et al. | Big data analytics for health systems | |
| KR20210008545A (en) | Method for providing business process management system based on automatic report generation | |
| Kalvit | Application of an innovative MBSE (SysML-1D) co-simulation in healthcare | |
| US20140136274A1 (en) | Providing multiple level process intelligence and the ability to transition between levels | |
| Neto et al. | Disease surveillance big data platform for large scale event processing | |
| JP2007524886A (en) | System and method for managing a distributed design network | |
| CN112633834A (en) | Method and system for project declaration demand analysis | |
| Huang et al. | Data storage accounting and verification at LHC experiments | |
| CN112579705A (en) | Metadata acquisition method and device, computer equipment and storage medium | |
| US9459939B2 (en) | In-memory approach to extend semantic event processing with domain insights | |
| Ataei et al. | Towards a domain-driven distributed reference architecture for big data systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |