CN113849549A - Data loading method, device, equipment and medium - Google Patents
Data loading method, device, equipment and medium Download PDFInfo
- Publication number
- CN113849549A CN113849549A CN202111107818.XA CN202111107818A CN113849549A CN 113849549 A CN113849549 A CN 113849549A CN 202111107818 A CN202111107818 A CN 202111107818A CN 113849549 A CN113849549 A CN 113849549A
- Authority
- CN
- China
- Prior art keywords
- data
- service
- database
- loading
- service data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000011068 loading method Methods 0.000 title claims abstract description 50
- 238000004140 cleaning Methods 0.000 claims abstract description 37
- 238000003860 storage Methods 0.000 claims abstract description 24
- 238000006243 chemical reaction Methods 0.000 claims abstract description 22
- 238000000034 method Methods 0.000 claims abstract description 17
- 238000009826 distribution Methods 0.000 claims description 61
- 230000002159 abnormal effect Effects 0.000 claims description 27
- 238000004590 computer program Methods 0.000 claims description 6
- 238000004519 manufacturing process Methods 0.000 claims description 6
- 238000007781 pre-processing Methods 0.000 abstract description 4
- 238000013523 data management Methods 0.000 description 10
- 230000003287 optical effect Effects 0.000 description 6
- 238000007405 data analysis Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000007726 management method Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/256—Integrating or interfacing systems involving database management systems in federated or virtual databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- Public Health (AREA)
- Water Supply & Treatment (AREA)
- General Health & Medical Sciences (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a data loading method, a data loading device, data loading equipment and a data loading medium, wherein the method comprises the following steps: connecting a plurality of data sources through a plurality of preset data source connectors, and acquiring service data of the plurality of data sources; performing data format conversion on the service data according to a preset data format, and performing data cleaning on the format-converted service data; and loading and storing the service data subjected to data cleaning into a Hadoop database. According to the technical scheme, the problem that efficiency of query, storage and use cannot be guaranteed when original data of multiple sources are directly uploaded to the database in the prior art is solved, preprocessing of the heterogeneous data of the multiple sources is achieved, then data loading and storage are conducted, and management of the database on the data is facilitated.
Description
Technical Field
The embodiment of the invention relates to the technical field of big data processing, in particular to a data loading method, a device, equipment and a medium.
Background
With the development of power grid management informatization, various information systems are built by various power supply enterprises from professional dimensions with different services, and a large amount of special data is accumulated by the power supply enterprises from various professional aspects along with the passing of time and the deepening of informatization construction. However, these data have no uniform standard, system implementation manner and data storage manner, and the dispersed data cannot form decision data support in the management layer of the power grid enterprise, and cannot improve the efficiency and benefit of the enterprise at a higher level. When business data in various professional aspects are integrated, the original data are directly uploaded to a database due to the huge data volume, and the efficiency of query, storage and use cannot be guaranteed.
Disclosure of Invention
The embodiment of the invention provides a data loading method, a data loading device, data loading equipment and a data loading medium, so that multi-source heterogeneous data can be efficiently loaded and conveniently managed by a database.
In a first aspect, an embodiment of the present invention provides a data loading method, where the method includes:
connecting a plurality of data sources through a plurality of preset data source connectors, and acquiring service data of the plurality of data sources;
performing data format conversion on the service data according to a preset data format, and performing data cleaning on the format-converted service data;
and loading and storing the service data subjected to data cleaning into a Hadoop database.
Optionally, the connecting a plurality of data sources through a plurality of preset data source connectors includes:
calculating expected data transmitting power of each data source, and determining the type of a database of each data source;
and establishing connection with the corresponding data sources according to the sequence of the numerical values of the expected data transmitting power from large to small through the preset data source connectors matched with the database types of the data sources.
Optionally, the data cleaning of the format-converted service data includes:
matching abnormal data in the service data after format conversion according to preset abnormal values of all data fields in the service data, and cleaning the abnormal data;
and when the data of each data field in the service data is numerical data, detecting abnormal data through quantiles, and cleaning the abnormal data.
Optionally, the service data includes a distribution transformer station identifier, a distribution transformer station name, and a plurality of service data in a subordinate distribution substation, a power supply station, a distribution transformer type, a subordinate substation, a distribution network line, a distribution transformer state, a distribution transformer production commissioning life, and a distribution transformer model, which are associated with the distribution transformer station identifier and the distribution transformer station name.
Optionally, the data cleaning the format-converted service data further includes:
and deleting the data of which the distribution substation identification does not match with the distribution substation name.
Optionally, the preset data source connectors include:
the system comprises a data source connector connected with a data source taking MySQL as a database, a data source connector connected with a data source taking ORACLE as a database and a data source connector connected with a data source taking MONGODB as a database.
Optionally, the performing data format conversion on the service data according to a preset data format includes:
when more than two data characteristics are included in the service data of one data source, the data characteristics are separated by characters to generate a plurality of pieces of data.
In a second aspect, an embodiment of the present invention further provides a data loading apparatus, where the apparatus includes:
the data acquisition module is used for connecting a plurality of data sources through a plurality of preset data source connectors and acquiring service data of the plurality of data sources;
the data processing module is used for carrying out data format conversion on the service data according to a preset data format and carrying out data cleaning on the service data after format conversion;
and the data loading module is used for loading and storing the service data subjected to data cleaning into the Hadoop database.
Optionally, the data obtaining module is specifically configured to:
calculating expected data transmitting power of each data source, and determining the type of a database of each data source;
and establishing connection with the corresponding data sources according to the sequence of the numerical values of the expected data transmitting power from large to small through the preset data source connectors matched with the database types of the data sources.
Optionally, the data processing module is specifically configured to:
matching abnormal data in the service data after format conversion according to preset abnormal values of all data fields in the service data, and cleaning the abnormal data;
and when the data of each data field in the service data is numerical data, detecting abnormal data through quantiles, and cleaning the abnormal data.
Optionally, the service data includes a distribution transformer station identifier, a distribution transformer station name, and a plurality of service data in a subordinate distribution substation, a power supply station, a distribution transformer type, a subordinate substation, a distribution network line, a distribution transformer state, a distribution transformer production commissioning life, and a distribution transformer model, which are associated with the distribution transformer station identifier and the distribution transformer station name.
Optionally, the data processing module is further configured to:
and deleting the data of which the distribution substation identification does not match with the distribution substation name.
Optionally, the preset data source connectors include:
the system comprises a data source connector connected with a data source taking MySQL as a database, a data source connector connected with a data source taking ORACLE as a database and a data source connector connected with a data source taking MONGODB as a database.
Optionally, the data processing module is further configured to:
when more than two data characteristics are included in the service data of one data source, the data characteristics are separated by characters to generate a plurality of pieces of data.
In a third aspect, an embodiment of the present invention further provides a computer device, where the computer device includes:
one or more processors;
a memory for storing one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement a data loading method as provided by any embodiment of the invention.
In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a data loading method as provided in any embodiment of the present invention.
The embodiment of the invention has the following advantages or beneficial effects:
in the embodiment of the invention, a plurality of preset data source connectors are connected with a plurality of data sources, and service data of the plurality of data sources are obtained; then, performing data format conversion on the service data according to a preset data format, and performing data cleaning on the format-converted service data; and finally, loading and storing the service data subjected to data cleaning into a Hadoop database. According to the technical scheme of the embodiment in the market, the problem that the efficiency of query, storage and use cannot be guaranteed by directly uploading multi-source original data to the database in the prior art is solved, preprocessing of the multi-source heterogeneous data is achieved, then data loading and storage are conducted, the database can manage the data conveniently, and the business data management and analysis efficiency of each data source is improved.
Drawings
Fig. 1 is a flowchart of a data loading method according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a data loading apparatus according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a computer device according to a third embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a flowchart of a data loading method according to an embodiment of the present invention, which is applicable to a situation of performing data management of multiple data sources. The method can be executed by a data loading device, which can be implemented by software and/or hardware, and is integrated in a computer device with application development function.
As shown in fig. 1, the data loading method includes the following steps:
s110, connecting a plurality of data sources through a plurality of preset data source connectors, and acquiring service data of the plurality of data sources.
In the context of grid data management, the data source may be any one of the service parties of the grid, such as various locations and various levels of power management units. When power supply service data management is performed in each unit, data items of service data are not completely the same, and format standards of stored data and database systems adopted by the data are different. Therefore, it is difficult to manage the grid data by higher-level managers. In this embodiment, first, a connection is established with each data source, and service data of each data source is accessed.
Specifically, for data sources using different databases, connections can be established with the data sources through different data interfaces. The data interface is a preset data source connector. The data source connector is used as a middleware of the data source and the data management system, on one hand, data can be read from the data source and uploaded to the target database, and on the other hand, a data processing instruction sent by the data management system can be issued to the data source. Exemplary, commonly used databases include MySQL, ORACLE, and MONGODB databases. Correspondingly, the data connector comprises a data source connector connected with a data source taking MySQL as a database, a data source connector connected with a data source taking ORACLE as a database, a data source connector connected with a data source taking MONGODB as a database and other connectors.
In particular, a data source connection rule may be set in the data connector for selecting a data source to be preferentially connected when one data source connector needs to connect a plurality of data sources. For example, the expected transmission power of the data source may be calculated by the following formula, and a data source with a higher value of the expected transmission power may be determined as a data source with a priority connection.
Wherein h isk＝LK/Davg，LkFor the kth data source data volume, DavgFor data upload delay, R is data upload expected rate, PmMaximum work of transmission for mth data sourceRate, hmChannel power gain for the mth data source, N0Is channel white Gaussian noise, PkE is a natural constant for the expected transmit power of the kth data source.
Furthermore, by establishing connection with each data source, the acquired service data can be structured data or unstructured data, and include scheduling, power transmission, power transformation, power distribution and power utilization service data generated in each link of electric power. Exemplary, the service data include a plurality of service data in distribution substation identification, a distribution substation name, a subordinate distribution substation associated with the distribution substation identification and the distribution substation name, a power supply station, a distribution transformer type, a subordinate substation, a distribution network line, a distribution transformer state, a distribution transformer production commissioning life and a distribution transformer model. Production real-time and historical data may also be included; external environment, weather and geographic information data, which affect the safe and stable operation of the power grid, relate to various services and have various data types.
And S120, performing data format conversion on the service data according to a preset data format, and performing data cleaning on the service data after format conversion.
The acquired data can be firstly cached in a cache region of the data management system, and after the data is processed, the data is loaded into a target database (Hadoop database) for data storage.
Firstly, the data formats and/or representation forms of all data sources are converted, data with different formats are converted into data with the same standard format, and for some data needing model analysis, different data items can be normalized according to data analysis requirements to obtain standard data. When more than two data characteristics are included in the service data of one data source, the data characteristics are separated by characters to generate a plurality of pieces of data. The data characteristics correspond to data fields in different data tables.
And then, performing data cleaning on the service data subjected to data conversion. The data cleaning comprises an abnormal data cleaning method and a data filtering method. For example, for some data items whose values are not negative, the positive and negative of the data under the data item are determined, and if the values are negative, the data are abnormal data. Or, by setting a quantile rule, abnormal values in the service data are monitored, and when the data exceeding the upper limit of the abnormal values are replaced by the upper limit of the abnormal values, the data lower than the lower limit of the abnormal values are replaced by the lower limit of the abnormal values. Or, during data filtering, directly eliminating a piece of data with a problem in a data item, for example, eliminating data with a mismatch between a distribution substation identifier and a distribution substation name, wherein the type of data belongs to invalid data.
And S130, loading and storing the service data subjected to data cleaning into a Hadoop database.
After data cleaning, the data meeting the conditions can be loaded into a Hadoop database for storage. In addition, data which do not meet the conditions or are wrong can be recorded, so that a user can correct the data according to the recorded error information and reload the data into the Hadoop database according to the loading rule after the data are corrected by the user. Thereby ensuring data integrity.
Through the steps, data processing is carried out on the data in advance before the data are loaded and stored, so that subsequent data analysis can be facilitated, and the efficiency of data management analysis is improved.
According to the technical scheme of the embodiment, a plurality of preset data source connectors are connected with a plurality of data sources, and service data of the plurality of data sources are obtained; then, performing data format conversion on the service data according to a preset data format, and performing data cleaning on the format-converted service data; and finally, loading and storing the service data subjected to data cleaning into a Hadoop database. According to the technical scheme of the embodiment in the market, the problem that the efficiency of query, storage and use cannot be guaranteed by directly uploading multi-source original data to the database in the prior art is solved, preprocessing of the multi-source heterogeneous data is achieved, then data loading and storage are conducted, the database can manage the data conveniently, and the business data management and analysis efficiency of each data source is improved.
Example two
Fig. 2 is a schematic structural diagram of a data loading apparatus according to a second embodiment of the present invention, where this embodiment is applicable to a case of managing data of multiple data sources, and the apparatus may be implemented in a software and/or hardware manner and integrated in a computer device with an application development function.
As shown in fig. 2, the data loading apparatus includes: a data acquisition module 210, a data processing module 220, and a data loading module 230.
The data obtaining module 210 is configured to connect a plurality of data sources through a plurality of preset data source connectors, and obtain service data of the plurality of data sources; the data processing module 220 is configured to perform data format conversion on the service data according to a preset data format, and perform data cleaning on the format-converted service data; and the data loading module 230 is configured to load and store the data-cleaned service data into the Hadoop database.
According to the technical scheme of the embodiment, a plurality of preset data source connectors are connected with a plurality of data sources, and service data of the plurality of data sources are obtained; then, performing data format conversion on the service data according to a preset data format, and performing data cleaning on the format-converted service data; and finally, loading and storing the service data subjected to data cleaning into a Hadoop database. According to the technical scheme of the embodiment in the market, the problem that the efficiency of query, storage and use cannot be guaranteed by directly uploading multi-source original data to the database in the prior art is solved, preprocessing of the multi-source heterogeneous data is achieved, then data loading and storage are conducted, the database can manage the data conveniently, and the business data management and analysis efficiency of each data source is improved.
Optionally, the data obtaining module 210 is specifically configured to:
calculating expected data transmitting power of each data source, and determining the type of a database of each data source;
and establishing connection with the corresponding data sources according to the sequence of the numerical values of the expected data transmitting power from large to small through the preset data source connectors matched with the database types of the data sources.
Optionally, the data processing module 220 is specifically configured to:
matching abnormal data in the service data after format conversion according to preset abnormal values of all data fields in the service data, and cleaning the abnormal data;
and when the data of each data field in the service data is numerical data, detecting abnormal data through quantiles, and cleaning the abnormal data.
Optionally, the service data includes a distribution transformer station identifier, a distribution transformer station name, and a plurality of service data in a subordinate distribution substation, a power supply station, a distribution transformer type, a subordinate substation, a distribution network line, a distribution transformer state, a distribution transformer production commissioning life, and a distribution transformer model, which are associated with the distribution transformer station identifier and the distribution transformer station name.
Optionally, the data processing module 220 is further configured to:
and deleting the data of which the distribution substation identification does not match with the distribution substation name.
Optionally, the preset data source connectors include:
the system comprises a data source connector connected with a data source taking MySQL as a database, a data source connector connected with a data source taking ORACLE as a database and a data source connector connected with a data source taking MONGODB as a database.
Optionally, the data processing module 220 is further configured to:
when more than two data characteristics are included in the service data of one data source, the data characteristics are separated by characters to generate a plurality of pieces of data.
The data loading device provided by the embodiment of the invention can execute the data loading method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a computer device according to a third embodiment of the present invention. FIG. 3 illustrates a block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention. The computer device 12 shown in FIG. 3 is only an example and should not impose any limitation on the scope of use or functionality of embodiments of the present invention. The computer device 12 may be any terminal device with computing capability, such as a terminal device of an intelligent controller, a server, a mobile phone, and the like.
As shown in FIG. 3, computer device 12 is in the form of a general purpose computing device. The components of computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. Computer device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 3, and commonly referred to as a "hard drive"). Although not shown in FIG. 3, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.
The processing unit 16 executes various functional applications and data processing by executing programs stored in the system memory 28, for example, implementing a data loading method provided by the present embodiment, the method includes:
connecting a plurality of data sources through a plurality of preset data source connectors, and acquiring service data of the plurality of data sources;
performing data format conversion on the service data according to a preset data format, and performing data cleaning on the format-converted service data;
and loading and storing the service data subjected to data cleaning into a Hadoop database.
Example four
A fourth embodiment provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a data loading method according to any embodiment of the present invention, and the method includes:
connecting a plurality of data sources through a plurality of preset data source connectors, and acquiring service data of the plurality of data sources;
performing data format conversion on the service data according to a preset data format, and performing data cleaning on the format-converted service data;
and loading and storing the service data subjected to data cleaning into a Hadoop database.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer-readable storage medium may be, for example but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It will be understood by those skilled in the art that the modules or steps of the invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of computing devices, and optionally they may be implemented by program code executable by a computing device, such that it may be stored in a memory device and executed by a computing device, or it may be separately fabricated into various integrated circuit modules, or it may be fabricated by fabricating a plurality of modules or steps thereof into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A method for loading data, the method comprising:
connecting a plurality of data sources through a plurality of preset data source connectors, and acquiring service data of the plurality of data sources;
performing data format conversion on the service data according to a preset data format, and performing data cleaning on the format-converted service data;
and loading and storing the service data subjected to data cleaning into a Hadoop database.
2. The method of claim 1, wherein connecting a plurality of data sources via a plurality of default data source connectors comprises:
calculating expected data transmitting power of each data source, and determining the type of a database of each data source;
and establishing connection with the corresponding data sources according to the sequence of the numerical values of the expected data transmitting power from large to small through the preset data source connectors matched with the database types of the data sources.
3. The method of claim 1, wherein the performing data cleansing on the format-converted service data comprises:
matching abnormal data in the service data after format conversion according to preset abnormal values of all data fields in the service data, and cleaning the abnormal data;
and when the data of each data field in the service data is numerical data, detecting abnormal data through quantiles, and cleaning the abnormal data.
4. The method of claim 1, wherein the service data comprises a plurality of service data selected from a distribution substation identifier, a distribution substation name, a subordinate distribution substation associated with the distribution substation identifier and the distribution substation name, a power supply station, a distribution transformer type, a subordinate substation, a distribution network line, a distribution transformer status, a distribution transformer production commissioning age, and a distribution transformer model.
5. The method of claim 4, wherein the performing data cleansing on the format-converted service data further comprises:
and deleting the data of which the distribution substation identification does not match with the distribution substation name.
6. The method of claim 1, wherein the plurality of default data source connectors comprises:
the system comprises a data source connector connected with a data source taking MySQL as a database, a data source connector connected with a data source taking ORACLE as a database and a data source connector connected with a data source taking MONGODB as a database.
7. The method of claim 1, wherein the performing data format conversion on the service data according to a preset data format comprises:
when more than two data characteristics are included in the service data of one data source, the data characteristics are separated by characters to generate a plurality of pieces of data.
8. A data loading apparatus, characterized in that the apparatus comprises:
the data acquisition module is used for connecting a plurality of data sources through a plurality of preset data source connectors and acquiring service data of the plurality of data sources;
the data processing module is used for carrying out data format conversion on the service data according to a preset data format and carrying out data cleaning on the service data after format conversion;
and the data loading module is used for loading and storing the service data subjected to data cleaning into the Hadoop database.
9. A computer device, characterized in that the computer device comprises:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a data loading method as claimed in any one of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data loading method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111107818.XA CN113849549A (en) | 2021-09-22 | 2021-09-22 | Data loading method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111107818.XA CN113849549A (en) | 2021-09-22 | 2021-09-22 | Data loading method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113849549A true CN113849549A (en) | 2021-12-28 |
Family
ID=78974856
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111107818.XA Pending CN113849549A (en) | 2021-09-22 | 2021-09-22 | Data loading method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113849549A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109558400A (en) * | 2018-11-28 | 2019-04-02 | 北京锐安科技有限公司 | Data processing method, device, equipment and storage medium |
| CN110457256A (en) * | 2019-08-01 | 2019-11-15 | 大众问问(北京)信息科技有限公司 | Date storage method, device, computer equipment and storage medium |
| CN111897875A (en) * | 2020-07-31 | 2020-11-06 | 平安科技（深圳）有限公司 | Fusion processing method and device for urban multi-source heterogeneous data and computer equipment |
| CN112308436A (en) * | 2020-11-04 | 2021-02-02 | 国网江苏省电力有限公司扬州市江都区供电分公司 | Power distribution network evaluation diagnosis analysis method, device, equipment and storage medium |
| CN112330108A (en) * | 2020-10-22 | 2021-02-05 | 贵州电网有限责任公司 | Auxiliary decision making system for distribution transformer data management |
-
2021
- 2021-09-22 CN CN202111107818.XA patent/CN113849549A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109558400A (en) * | 2018-11-28 | 2019-04-02 | 北京锐安科技有限公司 | Data processing method, device, equipment and storage medium |
| CN110457256A (en) * | 2019-08-01 | 2019-11-15 | 大众问问(北京)信息科技有限公司 | Date storage method, device, computer equipment and storage medium |
| CN111897875A (en) * | 2020-07-31 | 2020-11-06 | 平安科技（深圳）有限公司 | Fusion processing method and device for urban multi-source heterogeneous data and computer equipment |
| CN112330108A (en) * | 2020-10-22 | 2021-02-05 | 贵州电网有限责任公司 | Auxiliary decision making system for distribution transformer data management |
| CN112308436A (en) * | 2020-11-04 | 2021-02-02 | 国网江苏省电力有限公司扬州市江都区供电分公司 | Power distribution network evaluation diagnosis analysis method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109446274B (en) | Method and device for managing BI metadata of big data platform | |
| WO2021057198A1 (en) | Big data-based cross-domain service whole-process routing and penetration method and apparatus | |
| CN110688828A (en) | File processing method and device, file processing system and computer equipment | |
| CN111038906A (en) | Order sorting method and device | |
| WO2021057064A1 (en) | Data interaction conversion method and apparatus based on artificial intelligence, device, and medium | |
| CN111782672B (en) | Multi-field data management method and related device | |
| CN113590437B (en) | Alarm information processing method, device, equipment and medium | |
| CN112925584A (en) | Scene-based file configuration method, device, storage medium, and program product | |
| CN112579632A (en) | Data verification method, device, equipment and medium | |
| CN113849549A (en) | Data loading method, device, equipment and medium | |
| CN116383207A (en) | Data tag management method and device, electronic equipment and storage medium | |
| CN115330540A (en) | Method and device for processing transaction data | |
| CN114612212A (en) | Business processing method, device and system based on risk control | |
| CN113849508A (en) | Data storage method, device, equipment and medium | |
| CN111625524B (en) | Data processing method, device, equipment and storage medium | |
| US11842077B2 (en) | Method, device, and computer program product for transmitting data for object storage | |
| CN117951466B (en) | Data management method, device, medium and electronic equipment | |
| CN112669015B (en) | Power dispatching micro-service construction system and method | |
| WO2024140058A1 (en) | Data table life cycle determination method and apparatus, electronic device, and storage medium | |
| CN114897464A (en) | Method for establishing product flow list, electronic equipment and storage medium | |
| CN116319716A (en) | Information processing method, no-service system, electronic device, and storage medium | |
| CN115309728A (en) | Method and device for determining missing data of equipment | |
| CN114356928A (en) | Risk analysis method and device, electronic equipment and storage medium | |
| CN116842065A (en) | Service-based dynamic self-defined report method and system | |
| CN115964351A (en) | Log data writing and inquiring method, device, server and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |