CN113377851A - Data processing system and method - Google Patents
Data processing system and method Download PDFInfo
- Publication number
- CN113377851A CN113377851A CN202110653844.6A CN202110653844A CN113377851A CN 113377851 A CN113377851 A CN 113377851A CN 202110653844 A CN202110653844 A CN 202110653844A CN 113377851 A CN113377851 A CN 113377851A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- data processing
- acquired
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 61
- 238000000034 method Methods 0.000 title claims abstract description 31
- 238000011161 development Methods 0.000 claims abstract description 18
- 238000003672 processing method Methods 0.000 claims description 19
- 230000010354 integration Effects 0.000 claims description 17
- 230000005540 biological transmission Effects 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 11
- 238000007405 data analysis Methods 0.000 claims description 8
- 238000004140 cleaning Methods 0.000 claims description 6
- 238000004458 analytical method Methods 0.000 abstract description 17
- 238000013480 data collection Methods 0.000 abstract description 9
- 238000011282 treatment Methods 0.000 abstract description 4
- 230000006870 function Effects 0.000 description 26
- 230000008569 process Effects 0.000 description 18
- 238000004891 communication Methods 0.000 description 13
- 238000010586 diagram Methods 0.000 description 12
- 238000005516 engineering process Methods 0.000 description 5
- 238000007726 management method Methods 0.000 description 5
- 230000000007 visual effect Effects 0.000 description 4
- 238000013523 data management Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 230000005236 sound signal Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 238000013079 data visualisation Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000000586 desensitisation Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000010079 rubber tapping Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000010897 surface acoustic wave method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/256—Integrating or interfacing systems involving database management systems in federated or virtual databases
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H40/20—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms
Abstract
The invention relates to a data processing system and a method, wherein the system comprises: the data acquisition module is used for acquiring data from a data source through a JDBC database interface packaged by a webpage interface; the data modeling module is used for establishing a data table model according to the data table name, the field name and the constraint condition of the field type, importing a table establishing statement generated by the data table model into a database, and generating an entity table, wherein the entity table is used for storing the acquired data according to a uniform data format; and the data processing module is used for developing and analyzing the acquired data. Through the application, the problem that the large data platform solution integrating processing and analysis cannot be realized in the related art is solved, the universe data collection is realized, a standard unified inlet is provided for data acquisition and data pushing, the function of combining development and treatment is realized, and the large data platform integrating processing and analysis is realized.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data processing system, a data processing method, a computer device, and a computer-readable storage medium.
Background
In the technical field of clinical research, data sources of different businesses have corresponding acquisition modes and acquisition tools, and different companies usually establish data models suitable for the insides of the companies by taking business standards as guidance. The big data platform has the storage and calculation capacity and can store and calculate the collected data. Through the data pushing function, the collected data can be pushed to a BI database for data visualization. The analysts may obtain data and perform data analysis via the big data platform or the BI database.
The above data processing method has the following disadvantages: the global data collection of a company is complicated, and a uniform acquisition method and an entrance are not provided; data management is difficult, punctual output of data cannot be guaranteed, unified data modeling and table building processes do not exist, and the table structure and field types cannot be standardized; the data use cost is high, data pushing synchronization is needed, and a large data platform solution integrating processing and analysis cannot be realized; data asset management is difficult, and an effective authority management and data desensitization process cannot be performed; the above problems cannot be centralized on one platform for uniform processing.
At present, no effective solution is provided for the problem that a large data platform solution integrating processing and analysis cannot be realized in the related art.
Disclosure of Invention
The present application aims to overcome the defects in the prior art, and provides a data processing system, a data processing method, a computer device and a computer readable storage medium, so as to at least solve the problem that the related art cannot realize a large data platform solution integrating processing and analysis.
In order to achieve the purpose, the technical scheme adopted by the application is as follows:
in a first aspect, an embodiment of the present application provides a data processing system, including:
the data acquisition module is used for acquiring data from a data source through a JDBC database interface packaged by a webpage interface;
the data modeling module is connected with the data acquisition module and used for establishing a data table model according to the data table name, the field name and the constraint condition of the field type, importing a table establishing statement generated by the data table model into a database and generating a entity table, wherein the entity table is used for storing acquired data according to a uniform data format;
and the data processing module is respectively connected with the data acquisition module and the data modeling module and is used for developing and analyzing the acquired data.
In some of these embodiments, the data acquisition module comprises:
the reading data source plug-in is connected with the data source and used for reading data from the data source under the conditions that the identity authentication and the data authority authentication of a user pass and the data source exists;
the data transmission channel plug-in is connected with the read data source plug-in and is used for caching data;
and the writing data source plug-in is connected with the data transmission channel plug-in and is used for writing data into the target end database.
In some of these embodiments, the data processing module comprises:
the data development component is used for developing data;
the data report component is used for developing and visually displaying reports;
and the data query module is used for performing data query.
In some of these embodiments, the data processing module further comprises:
and the data sharing module is used for sharing the acquired data and/or the data analysis result of the acquired data.
In some of these embodiments, the data processing system further comprises:
the data integration module is connected with the data acquisition module and used for performing data integration on the acquired data, wherein the data integration comprises at least one of the following steps: the method comprises the steps of cleaning and collecting data of a plurality of data sources, extracting business commonalities from the data of the plurality of data sources, and collecting the fields to which the plurality of data sources belong.
In some of these embodiments, the data processing system further comprises:
and the authority control module is used for controlling the operation authority of the acquired data.
In a second aspect, an embodiment of the present application provides a data processing method, including:
collecting data from a data source through a JDBC database interface packaged by a webpage interface;
establishing a data table model according to the data table name, the field name and the constraint condition of the field type, and importing a table establishing statement generated by the data table model into a database to generate an entity table, wherein the entity table is used for storing the acquired data according to a uniform data format;
and developing and analyzing the collected data.
In some embodiments, collecting data from the data source via the JDBC database interface encapsulated by the web interface comprises:
reading data under the conditions that the identity authentication and the data authority authentication of the user pass and the existence of a data source is determined;
caching the read data;
and writing the read data into a target end database.
In a third aspect, an embodiment of the present application provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the data processing method according to the second aspect is implemented.
In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the data processing method according to the second aspect.
By adopting the technical scheme, compared with the prior art, the data processing system provided by the embodiment of the application acquires data from a data source through the JDBC database interface packaged by the data acquisition module through the webpage interface; establishing a data table model according to the data table name, the field name and the constraint condition of the field type through a data modeling module, importing a table establishing statement generated by the data table model into a database, and generating an entity table, wherein the entity table is used for storing acquired data according to a uniform data format; the data processing module is used for developing and analyzing the acquired data, so that the problem that a large data platform solution integrating processing and analysis cannot be realized in the related technology is solved, the universe data collection is realized, a standard and uniform entrance is provided for data acquisition and data push, and the complexity of a traditional data synchronization mode is avoided; the table name, the field name and the field type of each topic domain are restricted in the data modeling stage, and the phenomenon that the field names are the same and the data types are different is avoided; the function of combining development and treatment is realized, and the high-quality output of company data is ensured; the big data platform integrating processing and analysis is realized.
The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the application.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
fig. 1 is a block diagram of a mobile terminal according to an embodiment of the present application;
FIG. 2 is a block diagram of a data processing system according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a data collection process according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a data modeling flow according to an embodiment of the present application;
FIG. 5 is a schematic diagram of a data processing flow according to an embodiment of the present application;
FIG. 6 is a flow chart of a data processing method according to an embodiment of the present application;
fig. 7 is a hardware structure diagram of a computer device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application.
It is obvious that the drawings in the following description are only examples or embodiments of the present application, and that it is also possible for a person skilled in the art to apply the present application to other similar contexts on the basis of these drawings without inventive effort. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.
Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Reference to "connected," "coupled," and the like in this application is not intended to be limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" as referred to herein means two or more. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. Reference herein to the terms "first," "second," "third," and the like, are merely to distinguish similar objects and do not denote a particular ordering for the objects.
The embodiment provides a mobile terminal. Fig. 1 is a block diagram of a mobile terminal according to an embodiment of the present application. As shown in fig. 1, the mobile terminal includes: a Radio Frequency (RF) circuit 110, a memory 120, an input unit 130, a display unit 140, a sensor 150, an audio circuit 160, a wireless fidelity (WiFi) module 170, a processor 180, and a power supply 190. Those skilled in the art will appreciate that the mobile terminal architecture shown in fig. 1 is not intended to be limiting of mobile terminals and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
The following describes each constituent element of the mobile terminal in detail with reference to fig. 1:
the RF circuit 110 may be used for receiving and transmitting signals during information transmission and reception or during a call, and in particular, receives downlink information of a base station and then processes the received downlink information to the processor 180; in addition, the data for designing uplink is transmitted to the base station. In general, RF circuits include, but are not limited to, an antenna, at least one Amplifier, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, and the like. In addition, the RF circuitry 110 may also communicate with networks and other devices via wireless communications. The wireless communication may use any communication standard or protocol, including but not limited to Global System for Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Message Service (SMS), and the like.
The memory 120 may be used to store software programs and modules, and the processor 180 executes various functional applications and data processing of the mobile terminal by operating the software programs and modules stored in the memory 120. The memory 120 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the mobile terminal, and the like. Further, the memory 120 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
The input unit 130 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the mobile terminal. Specifically, the input unit 130 may include a touch panel 131 and other input devices 132. The touch panel 131, also referred to as a touch screen, may collect touch operations of a user on or near the touch panel 131 (e.g., operations of the user on or near the touch panel 131 using any suitable object or accessory such as a finger or a stylus pen), and drive the corresponding connection device according to a preset program. Alternatively, the touch panel 131 may include two parts, i.e., a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor 180, and can receive and execute commands sent by the processor 180. In addition, the touch panel 131 may be implemented by various types such as a resistive type, a capacitive type, an infrared ray, and a surface acoustic wave. The input unit 130 may include other input devices 132 in addition to the touch panel 131. In particular, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.
The display unit 140 may be used to display information input by a user or information provided to the user and various menus of the mobile terminal. The Display unit 140 may include a Display panel 141, and optionally, the Display panel 141 may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like. Further, the touch panel 131 can cover the display panel 141, and when the touch panel 131 detects a touch operation on or near the touch panel 131, the touch operation is transmitted to the processor 180 to determine the type of the touch event, and then the processor 180 provides a corresponding visual output on the display panel 141 according to the type of the touch event. Although the touch panel 131 and the display panel 141 are shown in fig. 1 as two separate components to implement the input and output functions of the mobile terminal, in some embodiments, the touch panel 131 and the display panel 141 may be integrated to implement the input and output functions of the mobile terminal.
The mobile terminal may also include at least one sensor 150, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor that may adjust the brightness of the display panel 141 according to the brightness of ambient light, and a proximity sensor that may turn off the display panel 141 and/or the backlight when the mobile terminal is moved to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally, three axes), detect the magnitude and direction of gravity when stationary, and can be used for applications (such as horizontal and vertical screen switching, related games, magnetometer attitude calibration) for recognizing the attitude of the mobile terminal, and related functions (such as pedometer and tapping) for vibration recognition; as for other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, and an infrared sensor, which can be configured on the mobile terminal, further description is omitted here.
A speaker 161 and a microphone 162 in the audio circuit 160 may provide an audio interface between the user and the mobile terminal. The audio circuit 160 may transmit the electrical signal converted from the received audio data to the speaker 161, and convert the electrical signal into a sound signal for output by the speaker 161; on the other hand, the microphone 162 converts the collected sound signal into an electric signal, converts the electric signal into audio data after being received by the audio circuit 160, and then outputs the audio data to the processor 180 for processing, and then transmits the audio data to, for example, another mobile terminal via the RF circuit 110, or outputs the audio data to the memory 120 for further processing.
WiFi belongs to a short-distance wireless transmission technology, and the mobile terminal can help a user to send and receive e-mails, browse webpages, access streaming media and the like through the WiFi module 170, and provides wireless broadband internet access for the user. Although fig. 1 shows the WiFi module 170, it is understood that it does not belong to the essential components of the mobile terminal, and it can be omitted or replaced with other short-range wireless transmission modules, such as Zigbee module or WAPI module, etc., as required within the scope not changing the essence of the invention.
The processor 180 is a control center of the mobile terminal, connects various parts of the entire mobile terminal using various interfaces and lines, and performs various functions of the mobile terminal and processes data by operating or executing software programs and/or modules stored in the memory 120 and calling data stored in the memory 120, thereby performing overall monitoring of the mobile terminal. Alternatively, processor 180 may include one or more processing units; preferably, the processor 180 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 180.
The mobile terminal also includes a power supply 190 (e.g., a battery) for powering the various components, which may preferably be logically coupled to the processor 180 via a power management system that may be configured to manage charging, discharging, and power consumption.
Although not shown, the mobile terminal may further include a camera, a bluetooth module, and the like, which will not be described herein.
In this embodiment, the processor 180 is configured to: collecting data from a data source through a JDBC database interface packaged by a webpage interface; establishing a data table model according to the data table name, the field name and the constraint condition of the field type, and importing a table establishing statement generated by the data table model into a database to generate an entity table, wherein the entity table is used for storing the acquired data according to a uniform data format; and developing and analyzing the collected data.
In some of these embodiments, the processor 180 is further configured to: reading data under the conditions that the identity authentication and the data authority authentication of the user pass and the existence of a data source is determined; caching the read data; and writing the read data into a target end database.
In some of these embodiments, the processor 180 is further configured to: developing the collected data; carrying out report development and visual display on the acquired data; and carrying out data query on the acquired data.
In some of these embodiments, the processor 180 is further configured to: and sharing the collected data and/or the data analysis result of the collected data.
In some of these embodiments, the processor 180 is further configured to: performing data integration on the acquired data, wherein the data integration comprises at least one of the following: the method comprises the steps of cleaning and collecting data of a plurality of data sources, extracting business commonalities from the data of the plurality of data sources, and collecting the fields to which the plurality of data sources belong.
In some of these embodiments, the processor 180 is further configured to: and controlling the operation authority of the acquired data.
The embodiment provides a data processing system. Fig. 2 is a block diagram of a data processing system according to an embodiment of the present application, as described in fig. 2, the data processing system including:
the data acquisition module 201 is used for acquiring data from a data source through a JDBC database interface packaged by a webpage interface;
the data modeling module 202 is connected to the data acquisition module 201, and is configured to establish a data table model according to constraint conditions of a data table name, a field name, and a field type, and import a table establishment statement generated by the data table model into a database to generate an entity table, where the entity table is used to store acquired data according to a uniform data format;
and the data processing module 203 is respectively connected with the data acquisition module 201 and the data modeling module 202, and is used for developing and analyzing the acquired data.
In some of these embodiments, the data acquisition module 201 includes:
the reading data source plug-in is connected with the data source and used for reading data from the data source under the conditions that the identity authentication and the data authority authentication of a user pass and the data source exists;
the data transmission channel plug-in is connected with the read data source plug-in and is used for caching data;
and the writing data source plug-in is connected with the data transmission channel plug-in and is used for writing data into the target end database.
The embodiment of the application uses a unified data synchronization process to solve a complicated data synchronization mode.
The data format and data characteristics of the data generated by different service systems are different. For example, the data generated by the APP system is mainly a log file, and the data generated by the OLTP online processing system is mainly stored in the Mysql database as structured data. The data acquisition module of the data processing system supports acquisition of semi-structured data such as log files and the like, and also supports acquisition of structured data in the Mysql database.
As shown in fig. 3, the data collection flow of the data collection module may include the following processes:
the first step is as follows: reading data sources
The Reader is a plug-in for reading a data source, plays a role of connecting the data source, can perform identity authentication and data authority authentication of a user in the connection process, can check whether the data source exists after the authentication is passed, and performs data reading after the current work is finished, and the Reader is an initial end of data acquisition.
The second step is that: data transmission channel
The Channel is a data transmission Channel plug-in, is a buffer area between the Reader and the Writer and is used for buffering data, and the existence of the Channel allows the Reader and the Writer to flexibly read and write.
The third step: write data source
The Writer is a write data source plug-in, connects the data buffer area with the target data source, and when the Writer plug-in is used, the data source connection needs to be configured first, and a user can select to store the data in a file system or directly write the data into a target end database.
In some of these embodiments, the normalization of the data is ensured by the modeling functionality of the data modeling module 202. After the data are synchronized to the data processing system, standard data table development is carried out by using a modeling function, wherein the standard data table development comprises the unification of field names and the unification of field types contained in a plurality of business modules. Through a standard modeling process, a uniform abstract framework is provided for the table building and the operation of the multi-bin system.
As shown in fig. 4, the physical modeling is implemented on a specific physical medium based on the logical model, for example, a database product is used to build a database. Corresponding keywords can be abstracted according to the business process in the data modeling process, a table model is established by matching with a standard field naming mode, and finally a table establishing statement generated by the model is led into a database to generate a entity table. In the application process of the table, because of standardized modeling and table building, a user can conveniently know the table content when seeing the table name and know the information stored in the field when seeing the field name, and the working efficiency is greatly improved.
According to the embodiment of the application, data generated by a source end is collected to a target end through a JDBC database interface packaged by a Web interface. The collected data are processed, and the processing process strictly complies with the modeling specification: selecting a business process, declaring granularity, confirming dimension and confirming fact, and selecting a star model in dimension modeling divides the bins into four levels of ODS, DWD, DWS and ADS.
In some of these embodiments, the data processing system further comprises:
a data integration module connected to the data acquisition module 201 and configured to perform data integration on the acquired data, where the data integration includes at least one of: the method comprises the steps of cleaning and collecting data of a plurality of data sources, extracting business commonalities from the data of the plurality of data sources, and collecting the fields to which the plurality of data sources belong.
Data integration can be viewed from multiple perspectives, and is data level integration, for example, complex and complicated data in each system is centrally processed; secondly, the integration of the service level is more concerned about the commonality of data among different systems, and the extraction of the commonality can enable the user station to see problems in higher dimensionality. And finally, integrating the subject domains, and completing the unification of the data in all the fields of the company by cleaning and collecting the data and extracting the commonalities of data services. A data warehouse is established that aggregates corporate universe data.
In some embodiments, the data processing module 203 may include:
the data development component is used for developing data;
the data report component is used for developing and visually displaying reports;
and the data query module is used for performing data query.
In some of these embodiments, the data processing module further comprises:
and the data sharing module is used for sharing the acquired data and/or the data analysis result of the acquired data.
In some of these embodiments, the data processing system further comprises:
and the authority control module is used for controlling the operation authority of the acquired data.
As shown in fig. 5, the authority control module may control data authority, such as addition, deletion, modification, and check of data; access rights, such as role, specific table, etc. access can be controlled; the operation authority can be controlled, such as data sharing, downloading, uploading and the like; and the authority of using and managing the intelligent template can be controlled. The data processing module 203 is developed on the data processing system, a data query function is provided, an analyst can edit query sentences in the component template, data query and analysis are conveniently carried out, a data sharing function is provided, data analysis results are shared to other personnel or are led in and led out, operations such as data export in a traditional mode are avoided, and safety and efficiency of data use are improved.
The data processing system of the embodiment of the application, namely the big data platform, is open to all department personnel of a company, and developers can use the development function module of the platform to develop data and rely on the data management function of the platform to arrange and maintain bottom data. The BI engineer can use a report function module of the platform to edit a data source according to the business condition of a company, and use a simple dragging function to develop a report so as to complete visual display. The analysts can use the analysis function modules provided by the platform to newly build a query panel and self-define query sentences. Meanwhile, an analyst is supported to custom create a form, and the whole analysis process is assisted. The analysis result can be shared to other personnel by using a data sharing function, so that the whole analysis process has mobility and is online. Through a large data platform combining functions of integration, development, treatment, analysis and the like, the data processing efficiency is improved, the data safety is improved, the use cost is reduced, and the business development of companies is assisted.
In a one-stop development and analysis big data platform, a plurality of modules are provided for different personnel to use, and developers can use data source synchronization, data iteration and task scheduling dependence functions; analysts can conveniently inquire and analyze on the platform, so that inconsistency of data apertures is avoided; BI personnel can conveniently use various graphic plug-ins to visually display data on the platform, need not to use third party visualization tool, reduce data use cost.
The data processing system provided by the embodiment of the application can realize company global data collection, provides a standard unified entrance for company data acquisition and data push, and avoids the complex and complicated traditional data synchronization mode. And in the data modeling stage, the table name, the field name and the field type of each topic domain are restricted, so that the phenomenon that the field names are the same and the data types are different is avoided. The function of combining development and management is realized, for example, employees can directly develop codes on a platform, whether business logic behind the codes is reasonable or not and whether data standards are in compliance or not, and after developers submit the codes, the data are processed by the data management function of the platform, so that high-quality output of company data is ensured. After data collection and processing, a unified interface is provided for an analyst, and company data are conveniently, quickly, efficiently and safely used through reasonable authority management and control.
The embodiment of the application provides a unified method and an entrance for acquiring and pushing the data source, standardizes a modeling and table building mode, and ensures that the data of each theme service has standard property and consistency. The data development function of the data platform helps company developers to save a large amount of tedious work and focus on logic writing in the aspect of business. The platform visualization capability effectively improves the data availability and provides data support for company services more conveniently. The data analysis requirements can be connected with a unified data outlet, and the inflow of repeatedly acquired data and dirty data is avoided. The data platform gets through the whole-process data link of the company, and company data assets are protected and applied better.
The embodiment provides a data processing method. Fig. 6 is a flowchart of a data processing method according to an embodiment of the present application, and as shown in fig. 6, the flowchart includes the following steps:
step S601, reading data under the condition that the identity authentication and the data authority authentication of the user pass and the existence of a data source is determined;
step S602, caching the read data;
step S603, writing the read data into the target database.
Through the steps, data are collected from a data source through a JDBC database interface packaged by a webpage interface; establishing a data table model according to the data table name, the field name and the constraint condition of the field type, and importing a table establishing statement generated by the data table model into a database to generate an entity table, wherein the entity table is used for storing the acquired data according to a uniform data format; the data development and analysis are carried out on the acquired data, so that the problem that a large data platform solution integrating processing and analysis cannot be realized in the related technology is solved, the universe data collection is realized, a standard and uniform entrance is provided for data acquisition and data push, and the complexity of a traditional data synchronization mode is avoided; the table name, the field name and the field type of each topic domain are restricted in the data modeling stage, and the phenomenon that the field names are the same and the data types are different is avoided; the function of combining development and treatment is realized, and the high-quality output of company data is ensured; the big data platform integrating processing and analysis is realized.
In some of these embodiments, step S601 may include: reading data under the conditions that the identity authentication and the data authority authentication of the user pass and the existence of a data source is determined; caching the read data; and writing the read data into a target end database.
In some embodiments, step S603 may include: developing the collected data; carrying out report development and visual display on the acquired data; and carrying out data query on the acquired data.
In some embodiments, step S603 may further include: and sharing the collected data and/or the data analysis result of the collected data.
In some embodiments, the data processing method further comprises: performing data integration on the acquired data, wherein the data integration comprises at least one of the following: the method comprises the steps of cleaning and collecting data of a plurality of data sources, extracting business commonalities from the data of the plurality of data sources, and collecting the fields to which the plurality of data sources belong.
In some embodiments, the data processing method further comprises: and controlling the operation authority of the acquired data.
It should be noted that the steps illustrated in the above-described flow diagrams or in the flow diagrams of the figures may be performed in a computer system, such as a set of computer-executable instructions, and that, although a logical order is illustrated in the flow diagrams, in some cases, the steps illustrated or described may be performed in an order different than here.
An embodiment provides a computer device. The data processing method combined with the embodiment of the application can be realized by computer equipment. Fig. 7 is a hardware structure diagram of a computer device according to an embodiment of the present application.
The computer device may comprise a processor 71 and a memory 72 in which computer program instructions are stored.
Specifically, the processor 71 may include a Central Processing Unit (CPU), or A Specific Integrated Circuit (ASIC), or may be configured to implement one or more Integrated circuits of the embodiments of the present Application.
The memory 72 may be used to store or cache various data files that need to be processed and/or used for communication, as well as possible computer program instructions executed by the processor 71.
The processor 71 reads and executes computer program instructions stored in the memory 72 to implement any of the data processing methods in the above-described embodiments.
In some of these embodiments, the computer device may also include a communication interface 73 and a bus 70. As shown in fig. 7, the processor 71, the memory 72, and the communication interface 73 are connected via the bus 70 to complete mutual communication.
The communication interface 73 is used for realizing communication among modules, devices, units and/or equipment in the embodiment of the present application. The communication interface 73 may also enable communication with other components such as: the data communication is carried out among external equipment, image/data acquisition equipment, a database, external storage, an image/data processing workstation and the like.
The bus 70 comprises hardware, software, or both that couple the components of the computer device to one another. Bus 70 includes, but is not limited to, at least one of the following: data Bus (Data Bus), Address Bus (Address Bus), Control Bus (Control Bus), Expansion Bus (Expansion Bus), and Local Bus (Local Bus). By way of example, and not limitation, Bus 70 may include an Accelerated Graphics Port (AGP) or other Graphics Bus, an Enhanced Industry Standard Architecture (EISA) Bus, a Front-Side Bus (FSB), a Hyper Transport (HT) Interconnect, an ISA (ISA) Bus, an InfiniBand (InfiniBand) Interconnect, a Low Pin Count (LPC) Bus, a memory Bus, a microchannel Architecture (MCA) Bus, a PCI (Peripheral Component Interconnect) Bus, a PCI-Express (PCI-X) Bus, a Serial Advanced Technology Attachment (SATA) Bus, a Video Electronics Bus (audio Electronics Association), abbreviated VLB) bus or other suitable bus or a combination of two or more of these. Bus 70 may include one or more buses, where appropriate. Although specific buses are described and shown in the embodiments of the application, any suitable buses or interconnects are contemplated by the application.
In addition, in combination with the data processing method in the foregoing embodiments, the embodiments of the present application may provide a computer-readable storage medium to implement. The computer readable storage medium having stored thereon computer program instructions; the computer program instructions, when executed by a processor, implement any of the data processing methods in the above embodiments.
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A data processing system, comprising:
the data acquisition module is used for acquiring data from a data source through a JDBC database interface packaged by a webpage interface;
the data modeling module is connected with the data acquisition module and used for establishing a data table model according to the data table name, the field name and the constraint condition of the field type, importing a table establishing statement generated by the data table model into a database and generating a entity table, wherein the entity table is used for storing acquired data according to a uniform data format;
and the data processing module is respectively connected with the data acquisition module and the data modeling module and is used for developing and analyzing the acquired data.
2. The data processing system of claim 1, wherein the data acquisition module comprises:
the reading data source plug-in is connected with the data source and used for reading data from the data source under the conditions that the identity authentication and the data authority authentication of a user pass and the data source exists;
the data transmission channel plug-in is connected with the read data source plug-in and is used for caching data;
and the writing data source plug-in is connected with the data transmission channel plug-in and is used for writing data into the target end database.
3. The data processing system of claim 1, wherein the data processing module comprises:
the data development component is used for developing data;
the data report component is used for developing and visually displaying reports;
and the data query module is used for performing data query.
4. The data processing system of claim 3, wherein the data processing module further comprises:
and the data sharing module is used for sharing the acquired data and/or the data analysis result of the acquired data.
5. The data processing system of any of claims 1 to 4, further comprising:
the data integration module is connected with the data acquisition module and used for performing data integration on the acquired data, wherein the data integration comprises at least one of the following steps: the method comprises the steps of cleaning and collecting data of a plurality of data sources, extracting business commonalities from the data of the plurality of data sources, and collecting the fields to which the plurality of data sources belong.
6. The data processing system of any of claims 1 to 4, further comprising:
and the authority control module is used for controlling the operation authority of the acquired data.
7. A data processing method, comprising:
collecting data from a data source through a JDBC database interface packaged by a webpage interface;
establishing a data table model according to the data table name, the field name and the constraint condition of the field type, and importing a table establishing statement generated by the data table model into a database to generate an entity table, wherein the entity table is used for storing the acquired data according to a uniform data format;
and developing and analyzing the collected data.
8. The data processing method of claim 7, wherein collecting data from a data source via a web interface encapsulated JDBC database interface comprises:
reading data under the conditions that the identity authentication and the data authority authentication of the user pass and the existence of a data source is determined;
caching the read data;
and writing the read data into a target end database.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the data processing method according to claim 7 or 8 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data processing method of claim 7 or 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110653844.6A CN113377851A (en) | 2021-06-11 | 2021-06-11 | Data processing system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110653844.6A CN113377851A (en) | 2021-06-11 | 2021-06-11 | Data processing system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113377851A true CN113377851A (en) | 2021-09-10 |
Family
ID=77573983
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110653844.6A Pending CN113377851A (en) | 2021-06-11 | 2021-06-11 | Data processing system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113377851A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030014335A1 (en) * | 2001-02-08 | 2003-01-16 | Output Technology Solutions, Inc. | Historical data warehousing system |
| CN107958028A (en) * | 2017-11-16 | 2018-04-24 | 平安科技（深圳）有限公司 | Method, apparatus, storage medium and the terminal of data acquisition |
| CN110032592A (en) * | 2019-03-06 | 2019-07-19 | 国网辽宁省电力有限公司信息通信分公司 | A kind of energy big data integration system and method |
| CN110866005A (en) * | 2019-09-26 | 2020-03-06 | 上海东方富联科技有限公司 | Internet of things data acquisition management method and system, storage medium and terminal |
| CN112100257A (en) * | 2020-08-21 | 2020-12-18 | 杭州安恒信息安全技术有限公司 | Data processing method, data processing device, computer equipment and storage medium |
| CN112699100A (en) * | 2020-12-31 | 2021-04-23 | 天津浪淘科技股份有限公司 | Management and analysis system based on metadata |
| CN112765184A (en) * | 2021-04-07 | 2021-05-07 | 四川新网银行股份有限公司 | Real-time acquisition method based on Mysql database and table division |
| CN112883118A (en) * | 2021-03-31 | 2021-06-01 | 浪潮云信息技术股份公司 | Method and system for synchronously acquiring incremental data based on sql |
-
2021
- 2021-06-11 CN CN202110653844.6A patent/CN113377851A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030014335A1 (en) * | 2001-02-08 | 2003-01-16 | Output Technology Solutions, Inc. | Historical data warehousing system |
| CN107958028A (en) * | 2017-11-16 | 2018-04-24 | 平安科技（深圳）有限公司 | Method, apparatus, storage medium and the terminal of data acquisition |
| CN110032592A (en) * | 2019-03-06 | 2019-07-19 | 国网辽宁省电力有限公司信息通信分公司 | A kind of energy big data integration system and method |
| CN110866005A (en) * | 2019-09-26 | 2020-03-06 | 上海东方富联科技有限公司 | Internet of things data acquisition management method and system, storage medium and terminal |
| CN112100257A (en) * | 2020-08-21 | 2020-12-18 | 杭州安恒信息安全技术有限公司 | Data processing method, data processing device, computer equipment and storage medium |
| CN112699100A (en) * | 2020-12-31 | 2021-04-23 | 天津浪淘科技股份有限公司 | Management and analysis system based on metadata |
| CN112883118A (en) * | 2021-03-31 | 2021-06-01 | 浪潮云信息技术股份公司 | Method and system for synchronously acquiring incremental data based on sql |
| CN112765184A (en) * | 2021-04-07 | 2021-05-07 | 四川新网银行股份有限公司 | Real-time acquisition method based on Mysql database and table division |
Non-Patent Citations (2)
| Title |
|---|
| 吉祥鸟: "基于python的数据采集以及可视化脚本的开发(超详细)", 《知乎》 * |
| 苏力萍: "《中台战略 中台建设与数字商业》" * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103929796B (en) | The method and device of shared access point | |
| CN108345543B (en) | Data processing method, device, equipment and storage medium | |
| RU2628488C2 (en) | Method, terminal device and server for nfc data transfer | |
| CN108156508B (en) | Barrage information processing method and device, mobile terminal, server and system | |
| CN106708554B (en) | Program operating method and device | |
| CN107204964B (en) | Authority management method, device and system | |
| CN103678502B (en) | Information collection method and device | |
| CN103543988A (en) | Method for processing array information, method and device of controlling information to enter arrays | |
| CN104426963A (en) | Terminal associating method and terminal | |
| CN105530239B (en) | Multi-medium data acquisition methods and device | |
| CN103631625B (en) | A kind of method of data acquisition, user terminal, server and system | |
| CN104901991A (en) | Methods, devices and system for transferring virtual resource | |
| CN104679381A (en) | Method and device for switching chat box | |
| CN104699501B (en) | A kind of method and device for running application program | |
| CN107040610A (en) | Method of data synchronization, device, storage medium, terminal and server | |
| CN107707618B (en) | Method and Related product based on position adjustment download | |
| CN114999604A (en) | Visit report generation method and device for clinical research project | |
| CN109246233A (en) | Data processing method, device, equipment and storage medium based on on-line monitoring | |
| CN104063400A (en) | Data search method and data search device | |
| CN112862432A (en) | Generation method and device of clinical research project management process | |
| CN104391629A (en) | Method for sending message in orientation manner, method for displaying message, server and terminal | |
| CN103617036B (en) | A kind of method, apparatus and terminal device for showing plug-in unit | |
| CN108205568A (en) | Method and device based on label selection data | |
| CN113377851A (en) | Data processing system and method | |
| CN104134044B (en) | A kind of detection method, device and system of Information Security |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210910 |