CN109949877A - A kind of data fusion method and system based on Experiment of Material Science - Google Patents
A kind of data fusion method and system based on Experiment of Material Science Download PDFInfo
- Publication number
- CN109949877A CN109949877A CN201910197620.1A CN201910197620A CN109949877A CN 109949877 A CN109949877 A CN 109949877A CN 201910197620 A CN201910197620 A CN 201910197620A CN 109949877 A CN109949877 A CN 109949877A
- Authority
- CN
- China
- Prior art keywords
- data
- constructed
- experiment
- template
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides a kind of data fusion method and system based on Experiment of Material Science, can automatically come out the Experiment of Material Science data pick-up for belonging to a storage template and be constructed, is converted into specified data set.The described method includes: obtaining the storage template of Experiment of Material Science data, the storage template is parsed, all fields for including in the storage template are obtained, the field to be constructed are extracted from all fields, and obtain the target file type to be constructed；It is connected to pre-set database or file system, the data of all fields in the storage template are all extracted；According to the field to be constructed that extraction obtains, the data of extraction are filtered；According to the target file type of acquisition to be constructed, target data set is converted by filtered data.The present invention relates to Material Fields.
Description
Technical field
The present invention relates to Material Fields, particularly relate to a kind of data fusion method and system based on Experiment of Material Science.
Background technique
The arrival of big data era and artificial intelligence changes the Industry of all trades and professions.Material Field is no exception,
Artificial intelligence can speed up research and development new material and Rapid Science experiment.The basis of artificial intelligence is data, Experiment of Material Science number
Include a variety of data formats according to being all unstructured and changeable, fusion is stored and extracted to data and brings huge challenge.
And lack the data set of specification, so that application of the artificial intelligence in materials science field is limited significantly.
In order to cope with challenges, numerous scholars develop the storage mode of unstructured data, and state key researches and develops special material
Material genetic engineering develops material genetic engineering storage system, and there are mainly two types of existing technologies, first is that fixed storage format, allows
User fills data according to regulation format, for example certain material private database, another kind are flexible and changeable mode, Yong Huke
With customized storage format, such as material genetic engineering storage system, it is subsequently filled data.The storage format of two ways can be with
It is referred to as storage template, although this storage mode based on storage template can store the changeable material science data of structure,
But these data still can not be utilized well, how these data are used becomes problem, in the prior art, one
As by way of manually extracting, generate the target data set for scientific analysis, the artificial mode for extracting data needs to expend
A large amount of manpower and material resources, efficiency is also very low, can not adapt to the data of rapid development.
Summary of the invention
The technical problem to be solved in the present invention is to provide a kind of data fusion method and system based on Experiment of Material Science,
To solve the problems, such as that artificial extraction data present in the prior art are time-consuming, laborious, inefficiency.
In order to solve the above technical problems, the embodiment of the present invention provides a kind of data fusion side based on Experiment of Material Science
Method, comprising:
The storage template for obtaining Experiment of Material Science data, parses the storage template, obtains the storage mould
All fields for including in plate extract the field to be constructed from all fields, and obtain the file destination class to be constructed
Type；
It is connected to pre-set database or file system, the data of all fields in the storage template are whole
It extracts；
According to the field to be constructed that extraction obtains, the data of extraction are filtered；
According to the target file type of acquisition to be constructed, target data set is converted by filtered data.
Further, the storage template for obtaining Experiment of Material Science data, parses the storage template, obtains
Include: to all fields for including in template that store
From pre-set template system, the storage template of Experiment of Material Science data is obtained, wherein template system,
Storage template for storage material scientific experimental data；Or, receiving depositing for the Experiment of Material Science data of user's manual creation
Store up template；
Determine the Format Type of the storage template；
According to the Format Type of the storage template, data parsing operation is carried out to the storage template, obtains described deposit
All fields for including in storage template.
Further, after all fields for including in obtaining the storage template, the method also includes:
All fields are uniformly switched to the json format of key-value pair description.
Further, described to be connected to pre-set database or file system, will own in the storage template
The data of field, which all extract, includes:
It is connected to pre-set database or file system, retrieves all numbers based on the storage template storage
According to the data of all fields in the storage template are all extracted using withdrawal device；
Wherein, the withdrawal device includes: file system connector and DB connector；
The file system connector, for being connected to the file system of operating system；
The DB connector, for connecting database.
Further, the DB connector includes: Postgresql connector, MySQL connector, MongoDB company
Connect one or more of device, Oracle connector, Redis connector；
Postgresql connector is for connecting postgresql database；
MySQL connector is for connecting MySQL database；
MongoDB connector is for connecting MongoDB database；
Oracle connector is for connecting oracle database；
Redis connector is for connecting Redis database.
Further, during extracting data, data cleansing operation is carried out to data, handles exceptional value and missing
Value.
Further, described according to the obtained field to be constructed of extraction, the data of extraction are filtered include:
According to the field to be constructed that extraction obtains, the data after cleaning are filtered, the field to be constructed is extracted
Data carry out data recombination, construct new stream object.
Further, the acquisition target file type to be constructed includes:
It is concentrated from pre-set target file type, obtains the target file type to be constructed；
Wherein, the target file type collection includes: the one or more file destinations of xml, json, excel, csv, txt
Type.
Further, the target file type to be constructed according to acquisition, converts target for filtered data
Data set includes:
It is corresponding using the target file type to be constructed according to the mapping relations between converter and target file type
The stream object of construction is converted target data set by converter；
Wherein, the mapping relations between converter and target file type include:
Xml converter is corresponding with xml document type；
Json converter is corresponding with json file type；
Excel converter is corresponding with Excel file type；
Csv converter is corresponding with csv file type；
Txt converter is corresponding with txt file type.
The embodiment of the present invention also provides a kind of data fusion system based on Experiment of Material Science, comprising:
Interactive module parses the storage template, obtains for obtaining the storage template of Experiment of Material Science data
All fields for including into the storage template, extract the field to be constructed from all fields, and obtain and to construct
Target file type；
Abstraction module is connected to pre-set database or file system, by all fields in the storage template
Data all extract；
Module is constructed, the field to be constructed for obtaining according to extraction is filtered the data of extraction；
Conversion module converts target for filtered data for the target file type to be constructed according to acquisition
Data set.
The advantageous effects of the above technical solutions of the present invention are as follows:
In above scheme, the storage template of Experiment of Material Science data is obtained, the storage template is parsed, is obtained
What all fields for including in the storage template, the extraction field to be constructed from all fields, and acquisition to be constructed
Target file type；It is connected to pre-set database or file system, by the number of all fields in the storage template
It is extracted according to whole；According to the field to be constructed that extraction obtains, the data of extraction are filtered；Structure is wanted according to acquisition
Filtered data are converted target data set by the target file type built.In such manner, it is possible to which a storage will be belonged to automatically
The Experiment of Material Science data of template are extracted from database or file system to be constructed, is converted into and can be directly used for
The specified data set of scientific analysis, saves time and manpower and material resources cost.
Detailed description of the invention
Fig. 1 is the flow diagram of the data fusion method provided in an embodiment of the present invention based on Experiment of Material Science；
Fig. 2 is data process of analysis schematic diagram provided in an embodiment of the present invention；
Fig. 3 is withdrawal device structural schematic diagram provided in an embodiment of the present invention；
Fig. 4 is the structural schematic diagram of the data fusion system provided in an embodiment of the present invention based on Experiment of Material Science；
Fig. 5 is that the detailed construction of the data fusion system provided in an embodiment of the present invention based on Experiment of Material Science is illustrated
Figure.
Specific embodiment
To keep the technical problem to be solved in the present invention, technical solution and advantage clearer, below in conjunction with attached drawing and tool
Body embodiment is described in detail.
The present invention is time-consuming, laborious existing artificial extraction data, inefficiency aiming at the problem that, provide a kind of based on material
The data fusion method and system of scientific experiment.
Embodiment one
As shown in Figure 1, the data fusion method provided in an embodiment of the present invention based on Experiment of Material Science, comprising:
S1 obtains the storage template of Experiment of Material Science data, parses to the storage template, obtain the storage
All fields for including in template extract the field to be constructed from all fields, and obtain the file destination to be constructed
Type；
S2 is connected to pre-set database or file system, by the data of all fields in the storage template
All extract；
S3 is filtered the data of extraction according to the field to be constructed that extraction obtains；
S4 converts target data set for filtered data according to the target file type of acquisition to be constructed.
Based on the data fusion method of Experiment of Material Science described in the embodiment of the present invention, Experiment of Material Science data are obtained
Storage template, the storage template is parsed, all fields for including in the storage template are obtained, from described all
The field to be constructed is extracted in field, and obtains the target file type to be constructed；Be connected to pre-set database or
File system all extracts the data of all fields in the storage template；The word to be constructed obtained according to extraction
Section, is filtered the data of extraction；According to the target file type of acquisition to be constructed, mesh is converted by filtered data
Mark data set.In such manner, it is possible to which the Experiment of Material Science data of a storage template will be belonged to automatically from database or file
It is extracted in system and is constructed, is converted into the specified data set that can be directly used for scientific analysis, save time and manpower object
Power cost.
In the present embodiment, by taking titanium alloy stretching test as an example, which is novel in order to test for Experiment of Material Science
The tensile property of titanium alloy material.
In the present embodiment, the material that the titanium alloy stretching test is selected is alpha and beta titanium alloy, the corresponding trade mark are as follows:
Ti-6Al-1.5Cr-2.5Mo-0.5Fe-0.3Si；Chemical component is as follows:
{
Ti, surplus
Al, 5.5~7.0
Mo, 2.0~3.0
Cr, 0.8~2.3
Fe, 0.2~0.7
Si, 0.15~0.4
C,0.08
N,0.05
H,0.015
O,0.18
Other compositions, 0.5
}
The processing technology that the titanium alloy stretching test uses are as follows:
{
Process code name: MI
Title: isothermal annealing
Illustrate: the state after isothermal annealing
}
Experimental result are as follows:
{
Performance class: chemical property
Title: tensile property
Kind: forging stick
Sample direction: L
Temperature: 500
δ or other specifications: 60
Performance number: 657 (this experimental result for being)
}
In the present embodiment, for titanium alloy stretching test, titanium alloy can be gone out according to storage template extraction and drawn
The chemical component, processing technology and experimental data for stretching performance test, obtain target data set, and target data set is applied to number
The drawing of titanium alloy can directly be predicted for given titanium alloy material composition, processing technology and experimental data according to analysis field
Performance is stretched, without manually carrying out measured data of experiment by stretching experiment platform, can directly save time and manpower and material resources
Cost.
Data fusion method provided in this embodiment based on Experiment of Material Science can be used for Experiment of Material Science result and deposit
Storage system, solves that artificial extraction data present in current material scientific experiment result storage system are time-consuming, laborious, low efficiency
Under problem there is fast and stable and at low cost and based on the data fusion method of Experiment of Material Science described in the present embodiment
The advantages of.
It is further, described to obtain in the specific embodiment of the aforementioned data fusion method based on Experiment of Material Science
Draw materials the storage template of scientific experimental data, the storage template parsed, obtain include in the storage template
All fields include:
From pre-set template system, the storage template of Experiment of Material Science data is obtained, wherein template system,
Storage template for storage material scientific experimental data；Or, receiving depositing for the Experiment of Material Science data of user's manual creation
Store up template；
Determine the Format Type of the storage template；
According to the Format Type of the storage template, data parsing operation is carried out to the storage template, obtains described deposit
All fields for including in storage template.
In the present embodiment, the template system is the template system of storage system, and the storage template in template system is to use
Json or xml document are described.In a particular application, can connect to pre-set template system by using
The storage template of storage system obtains interface, gets the storage template of specified json xml format.
In the present embodiment, the storage template of json xml format can be created with manual mode, and Template Information is filled out
It writes whole.
In the present embodiment, after obtaining storage template, the Format Type of the storage template need to be determined, for example, being
Json format or xml format；And according to the Format Type of the storage template, data parsing is carried out to the storage template
Operation, data process of analysis figure by the way of recursive iteration as shown in Fig. 2, traverse storage all fields of template, by all words
Duan Tongyi switchs to the json format of key-value pair description, obtains a Template_field, to identify, Template_field
Are as follows:
In the present embodiment, material science data usually include: numerical value, character string, image and video these types, by material
Expect that scientific experiment generates, all fields all include default field, for filling missing values automatically when missing values, in addition, right
In numeric type field, there is range instruction；For character string data, there is maximum length limitation.
In the present embodiment, after obtaining Template_field, all field names (FieldName) is also needed to extract
Come to select, selection will construct the field that data set includes, also to be concentrated from pre-set target file type, selection is wanted
The target file type of data set is constructed, Target_field is generated:
The field name that Target_field contains target file type and will extract (is referred to as: target word
Section).
In the present embodiment, the target file type collection includes: the one or more mesh of xml, json, excel, csv, txt
Mark file type.In practical applications, target file type collection can be determined according to practical application scene.
In the specific embodiment of the aforementioned data fusion method based on Experiment of Material Science, further, the company
It is connected to pre-set database or file system, the data of all fields in the storage template are all extracted into packet
It includes:
It is connected to pre-set database or file system, retrieves all numbers based on the storage template storage
According to,；
Wherein, the withdrawal device includes: file system connector and DB connector；
The file system connector, for being connected to the file system of operating system；
The DB connector, for connecting database.
In the present embodiment, it is connected to pre-set database or file system, in the storage template that will acquire
All fields as data query conditions, all data based on the storage template storage are retrieved, using withdrawal device by institute
The data for stating all fields in storage template all extract, wherein during extracting data, carry out data to data
Cleaning operation handles exceptional value and missing values.
In the present embodiment, query sentence of database is generated according to Template_field object, from the database having connected
All data based on storage template storage are retrieved, or the store path of file system is provided, batch reads material science
Experimental result file all checked and cleaned to every data after reading data, can be with from Template_field
Obtain the limitation of each column data attribute.The integrality for first checking data uses default in Template_field for missing values
Instead of；Then check the legitimacy (whether being exceptional value) of data, it is main check data type and limitation whether and Template_
It is consistent in field, illegal content default value is filled, the integrality and legitimacy of data is checked out, is done
Net complete data object.
In the present embodiment, Database Systems indicate the database of storage system, and file system indicates Experiment of Material Science
As a result file system is stored.
In the present embodiment, file system connector is used for the file system of attended operation system, is integrated with some files batch
The interface that amount reads, is written.
In the specific embodiment of the aforementioned data fusion method based on Experiment of Material Science, further, the number
It include: Postgresql connector, MySQL connector, MongoDB connector, Oracle connector, Redis according to library connector
One or more of connector；
Postgresql connector is for connecting postgresql database；
MySQL connector is for connecting MySQL database；
MongoDB connector is for connecting MongoDB database；
Oracle connector is for connecting oracle database；
Redis connector is for connecting Redis database.
In the present embodiment, the DB connector includes: Postgresql connector, MySQL connector, MongoDB
The connector in this 5 different frequently-used data libraries of connector, Oracle connector, Redis connector, as shown in Figure 3；In reality
In, the DB connector can carry out customized expansion, keep consistent with the Database Systems of storage system.
In the present embodiment, Postgresql connector is for connecting postgresql database, integrating and having encapsulated often
It is instructed with SQL query.MySQL connector is for connecting MySQL database, integrating and having encapsulated common SQL query instruction.
MongoDB connector is for connecting MongoDB database, integrating and having encapsulated common mongodb data base querying instruction.
Oracle connector is for connecting oracle database, integrating and having encapsulated common SQL query instruction.Redis connector is used
In connection Redis database, integrates and encapsulated common Redis data base querying instruction.
In the present embodiment, DB connector needs input database address and port, user name, password, database name
The information such as title are attached, and withdrawal device has been internally integrated the operational order of each database, are abstracted as unified query interface, are shielded
Each database language inconsistent problem.
In the specific embodiment of the aforementioned data fusion method based on Experiment of Material Science, further, described
According to the obtained field to be constructed of extraction, the data of extraction are filtered include:
According to the field to be constructed that extraction obtains, the data after cleaning are filtered, the field to be constructed is extracted
Data carry out data recombination, construct new stream object, the stream object is that a data frame (DataFrame) is right
As.
In the present embodiment, the data after cleaning are carried out screening and filtering according to Target_field object, extract Target_
The field for including in field object gives up other unwanted fields, the word for including in the Target_field object of extraction
Segment data is reassembled into a stream object, wherein the stream object includes: index and aiming field.
In the specific embodiment of the aforementioned data fusion method based on Experiment of Material Science, further, described
According to the target file type of acquisition to be constructed, converting target data set for filtered data includes:
It is corresponding using the target file type to be constructed according to the mapping relations between converter and target file type
The stream object of construction is converted target data set by converter；
Wherein, the mapping relations between converter and target file type include:
Xml converter is corresponding with xml document type；
Json converter is corresponding with json file type；
Excel converter is corresponding with Excel file type；
Csv converter is corresponding with csv file type；
Txt converter is corresponding with txt file type.
In the present embodiment, converter includes five sub- converters, can convert five Doctypes.Five sub- converters with
Five target file types correspond, specific:
Excel converter, for stream compression to be turned to excel document.Optionally, the pandas data of python can be used
Handling implement is converted, which realizes the method that DataFrame is transformed into excel file, the excel converter collection
At this method.
Xml converter, for stream compression to be turned to xml document.Optionally, it can be realized with the ready-made library xml, xml turns
Change device to be integrated with the library xml of python and realize high-level interface.
Json converter, for stream compression to be turned to json document.Optionally, there are numerous open source json tools, json
The library json that converter is integrated with python realizes the conversion from DataFrame to json.
Csv converter, for stream compression to be turned to csv file.Optionally, the pandas data processing of python can be used
Tool is converted, and csv converter realizes the method that DataFrame is transformed into csv file.
Txt converter, for stream compression to be turned to txt file.Read and write text file major part language it is all built-in this
A function.
In the present embodiment, according to the target file type to be constructed using corresponding converter DataFrame data pair
As switching to for target data set.
Embodiment two
The present invention also provides a kind of specific embodiments of data fusion system based on Experiment of Material Science, due to this hair
The data fusion system based on Experiment of Material Science of bright offer and the aforementioned data fusion method based on Experiment of Material Science
Specific embodiment is corresponding, and being somebody's turn to do the data fusion system based on Experiment of Material Science can be specifically real by executing the above method
The process step in mode is applied to achieve the object of the present invention, therefore the above-mentioned data fusion method tool based on Experiment of Material Science
Explanation in body embodiment is also applied for the tool of the data fusion system provided by the invention based on Experiment of Material Science
Body embodiment will not be described in great detail in present invention specific embodiment below.
As shown in figure 4, the embodiment of the present invention also provides a kind of data fusion system based on Experiment of Material Science, comprising:
Interactive module 201 solves the storage template for obtaining the storage template of Experiment of Material Science data
Analysis obtains all fields for including in the storage template, the field to be constructed is extracted from all fields, and obtain and want
The target file type of building；
Abstraction module 204 is connected to pre-set database or file system, by all words in the storage template
The data of section all extract；
Module 203 is constructed, the field to be constructed for obtaining according to extraction is filtered the data of extraction；
Conversion module 202 converts mesh for filtered data for the target file type to be constructed according to acquisition
Mark data set.
Based on the data fusion system of Experiment of Material Science described in the embodiment of the present invention, Experiment of Material Science data are obtained
Storage template, the storage template is parsed, all fields for including in the storage template are obtained, from described all
The field to be constructed is extracted in field, and obtains the target file type to be constructed；Be connected to pre-set database or
File system all extracts the data of all fields in the storage template；The word to be constructed obtained according to extraction
Section, is filtered the data of extraction；According to the target file type of acquisition to be constructed, mesh is converted by filtered data
Mark data set.In such manner, it is possible to which the Experiment of Material Science data of a storage template will be belonged to automatically from database or file
It is extracted in system and is constructed, is converted into the specified data set that can be directly used for scientific analysis, save time and manpower object
Power cost.
Fig. 5 is the detailed construction schematic diagram of the data fusion system provided in this embodiment based on Experiment of Material Science, such as
Shown in Fig. 5, the system also includes: template system 205, Database Systems 206 and file system 207.
In the present embodiment, interactive module 201 is mainly used for obtaining for realizing the interactive function of system and extraneous input and output
Family input is taken finally to be exported as requested, for example, obtain user's selection/the Experiment of Material Science data that input deposit
Store up template；It is also used to parse the storage template of acquisition, obtains all fields for including in the storage template, it can
It is as follows to be described as Template_field:
In the present embodiment, interactive module 201 is also used to after obtaining Template_field, is also needed all fields
Name (FieldName) is extracted to select, and selection will construct the field that data set includes, will also be from pre-set target
File type is concentrated, and the target file type that construct data set is chosen, and generates Target_field:
The field name that Target_field contains target file type and will extract (is referred to as: target word
Section).
In the present embodiment, Template_field is sent to abstraction module 204 by interactive module 201, Target_
Field is sent to building module 203 and conversion module 202.
In the present embodiment, abstraction module 204 receives the Template_field object from interactive module 201, according to
Template_field object generates query sentence of database, is deposited from the database retrieval having connected is all based on the storage template
The data of storage, or provide file system store path, batch read Experiment of Material Science destination file, read data it
Every data is all checked and cleaned afterwards, the available each column data attribute limitation from Template_field.First examine
The integrality for looking into data replaces missing values using default in Template_field；Then the legitimacy of data is checked
(whether being exceptional value), it is main to check whether data type and limitation are consistent in Template_field, for illegal
Content is filled with default value, checks out the integrality and legitimacy of data, obtains clean complete data object, and will count
Building module 203 is sent to according to object.
In the present embodiment, constructs module 203 object sended over according to abstraction module 204 and what is received be selfed
The Target_field object of mutual module 201 carries out screening and filtering, extracts the field for including in Target_field object, gives up
The field data for including in the Target_field object of extraction is reassembled into one by other unwanted fields
DataFrame object is sent conversion module 202 by DataFrame object.
In the present embodiment, the file destination to be constructed in Target_field of the conversion module 202 according to interactive module 201
Type is switched to DataFrame object for target data set using corresponding converter；Conversion module 202 will obtain target data
Collection is sent to interactive module 201, and interactive module provides unified download interface, file is downloaded to specified directory.
In the present embodiment, template system 205, this refers to the template system of storage system.
In the present embodiment, Database Systems 206 refer to the database module of storage material scientific experiment result.
In the present embodiment, the result is that storing in the form of a file, file system 207 is used for table for some Experiments of Material Science
Show the file system of storage result file.
The above is a preferred embodiment of the present invention, it is noted that for those skilled in the art
For, without departing from the principles of the present invention, several improvements and modifications can also be made, these improvements and modifications
It should be regarded as protection scope of the present invention.
Claims (10)
1. a kind of data fusion method based on Experiment of Material Science characterized by comprising
The storage template for obtaining Experiment of Material Science data, parses the storage template, obtains in the storage template
All fields for including extract the field to be constructed from all fields, and obtain the target file type to be constructed；
It is connected to pre-set database or file system, the data of all fields in the storage template are all extracted
Out；
According to the field to be constructed that extraction obtains, the data of extraction are filtered；
According to the target file type of acquisition to be constructed, target data set is converted by filtered data.
2. the data fusion method according to claim 1 based on Experiment of Material Science, which is characterized in that the acquisition material
Expect scientific experimental data storage template, the storage template is parsed, obtain it is described storage template in include own
Field includes:
From pre-set template system, the storage template of Experiment of Material Science data is obtained, wherein template system is used for
The storage template of storage material scientific experimental data；Or, receiving the storage mould of the Experiment of Material Science data of user's manual creation
Plate；
Determine the Format Type of the storage template；
According to the Format Type of the storage template, data parsing operation is carried out to the storage template, obtains the storage mould
All fields for including in plate.
3. the data fusion method according to claim 2 based on Experiment of Material Science, which is characterized in that described in obtaining
After all fields for including in storage template, the method also includes:
All fields are uniformly switched to the json format of key-value pair description.
4. the data fusion method according to claim 1 based on Experiment of Material Science, which is characterized in that described to be connected to
The data of all fields in the storage template are all extracted and include: by pre-set database or file system
It is connected to pre-set database or file system, retrieves all data based on the storage template storage, benefit
The data of all fields in the storage template are all extracted with withdrawal device；
Wherein, the withdrawal device includes: file system connector and DB connector；
The file system connector, for being connected to the file system of operating system；
The DB connector, for connecting database.
5. the data fusion method according to claim 4 based on Experiment of Material Science, which is characterized in that the database
Connector includes: Postgresql connector, MySQL connector, MongoDB connector, Oracle connector, Redis connection
One or more of device；
Postgresql connector is for connecting postgresql database；
MySQL connector is for connecting MySQL database；
MongoDB connector is for connecting MongoDB database；
Oracle connector is for connecting oracle database；
Redis connector is for connecting Redis database.
6. the data fusion method according to claim 4 based on Experiment of Material Science, which is characterized in that extracting data
During, data cleansing operation is carried out to data, handles exceptional value and missing values.
7. the data fusion method according to claim 6 based on Experiment of Material Science, which is characterized in that the basis mentions
The field to be constructed obtained, is filtered the data of extraction and includes:
According to the field to be constructed that extraction obtains, the data after cleaning are filtered, the field data to be constructed is extracted
Data recombination is carried out, new stream object is constructed.
8. the data fusion method according to claim 7 based on Experiment of Material Science, which is characterized in that the acquisition is wanted
The target file type of building includes:
It is concentrated from pre-set target file type, obtains the target file type to be constructed；
Wherein, the target file type collection includes: the one or more target file types of xml, json, excel, csv, txt.
9. the data fusion method according to claim 8 based on Experiment of Material Science, which is characterized in that the basis obtains
The target file type to be constructed taken, converting target data set for filtered data includes:
According to the mapping relations between converter and target file type, the corresponding conversion of the target file type to be constructed is utilized
The stream object of construction is converted target data set by device；
Wherein, the mapping relations between converter and target file type include:
Xml converter is corresponding with xml document type；
Json converter is corresponding with json file type；
Excel converter is corresponding with Excel file type；
Csv converter is corresponding with csv file type；
Txt converter is corresponding with txt file type.
10. a kind of data fusion system based on Experiment of Material Science characterized by comprising
Interactive module parses the storage template, obtains institute for obtaining the storage template of Experiment of Material Science data
All fields for including in storage template are stated, the field to be constructed are extracted from all fields, and obtain the mesh to be constructed
Mark file type；
Abstraction module is connected to pre-set database or file system, by the number of all fields in the storage template
It is extracted according to whole；
Module is constructed, the field to be constructed for obtaining according to extraction is filtered the data of extraction；
Conversion module converts target data for filtered data for the target file type to be constructed according to acquisition
Collection.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910197620.1A CN109949877A (en) | 2019-03-15 | 2019-03-15 | A kind of data fusion method and system based on Experiment of Material Science |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910197620.1A CN109949877A (en) | 2019-03-15 | 2019-03-15 | A kind of data fusion method and system based on Experiment of Material Science |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109949877A true CN109949877A (en) | 2019-06-28 |
Family
ID=67010094
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910197620.1A Pending CN109949877A (en) | 2019-03-15 | 2019-03-15 | A kind of data fusion method and system based on Experiment of Material Science |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109949877A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111240714A (en) * | 2019-12-29 | 2020-06-05 | 南京云帐房网络科技有限公司 | Financial data initialization method and system based on template intelligent learning |
| CN112231524A (en) * | 2020-10-22 | 2021-01-15 | 北京天融信网络安全技术有限公司 | Data fusion method and device, storage medium and electronic equipment |
| CN113030734A (en) * | 2021-02-03 | 2021-06-25 | 智光研究院(广州)有限公司 | Method and device for identifying parameters of electrical model |
| CN113505527A (en) * | 2021-06-24 | 2021-10-15 | 中国科学院计算机网络信息中心 | Material property prediction method and system based on data driving |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102521292A (en) * | 2011-11-29 | 2012-06-27 | 西安交通大学 | Template-based analytic method for integrated data of heterogeneous pollution source |
| CN103150380A (en) * | 2013-03-13 | 2013-06-12 | 河海大学 | Table format customizable Excel table analysis method |
| CN106528880A (en) * | 2016-12-14 | 2017-03-22 | 云南电网有限责任公司电力科学研究院 | Normalizing method and system for data structure format of multi-source power service data |
| CN109033319A (en) * | 2018-07-18 | 2018-12-18 | 长扬科技（北京）有限公司 | A kind of big data log method for normalizing and tool |
| CN109086444A (en) * | 2018-08-17 | 2018-12-25 | 吉林亿联银行股份有限公司 | A kind of data normalization method, apparatus and electronic equipment |
-
2019
- 2019-03-15 CN CN201910197620.1A patent/CN109949877A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102521292A (en) * | 2011-11-29 | 2012-06-27 | 西安交通大学 | Template-based analytic method for integrated data of heterogeneous pollution source |
| CN103150380A (en) * | 2013-03-13 | 2013-06-12 | 河海大学 | Table format customizable Excel table analysis method |
| CN106528880A (en) * | 2016-12-14 | 2017-03-22 | 云南电网有限责任公司电力科学研究院 | Normalizing method and system for data structure format of multi-source power service data |
| CN109033319A (en) * | 2018-07-18 | 2018-12-18 | 长扬科技（北京）有限公司 | A kind of big data log method for normalizing and tool |
| CN109086444A (en) * | 2018-08-17 | 2018-12-25 | 吉林亿联银行股份有限公司 | A kind of data normalization method, apparatus and electronic equipment |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111240714A (en) * | 2019-12-29 | 2020-06-05 | 南京云帐房网络科技有限公司 | Financial data initialization method and system based on template intelligent learning |
| CN111240714B (en) * | 2019-12-29 | 2024-01-05 | 云帐房网络科技有限公司 | Financial data initialization method and system based on template intelligent learning |
| CN112231524A (en) * | 2020-10-22 | 2021-01-15 | 北京天融信网络安全技术有限公司 | Data fusion method and device, storage medium and electronic equipment |
| CN113030734A (en) * | 2021-02-03 | 2021-06-25 | 智光研究院(广州)有限公司 | Method and device for identifying parameters of electrical model |
| CN113030734B (en) * | 2021-02-03 | 2023-10-20 | 智光研究院(广州)有限公司 | Identification method and device for electrical model parameters |
| CN113505527A (en) * | 2021-06-24 | 2021-10-15 | 中国科学院计算机网络信息中心 | Material property prediction method and system based on data driving |
| CN113505527B (en) * | 2021-06-24 | 2022-10-04 | 中国科学院计算机网络信息中心 | Material property prediction method and system based on data driving |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109949877A (en) | A kind of data fusion method and system based on Experiment of Material Science | |
| US9507811B2 (en) | Compressed data page with uncompressed data fields | |
| CN104685497B (en) | The hardware realization of the polymerization/packet operated by filter method | |
| CN106055584B (en) | Manage data query | |
| CN102004744B (en) | Data extraction system and method from one source table to table of at least one object database | |
| CN111091876A (en) | DNA storage method, system and electronic equipment | |
| US20060282452A1 (en) | System and method for mapping structured document to structured data of program language and program for executing its method | |
| US8316034B2 (en) | Analyzing binary data streams to identify embedded record structures | |
| CN109766085B (en) | Method and device for processing enumeration type codes | |
| CN107784026A (en) | A kind of ETL data processing methods and device | |
| KR101535703B1 (en) | Apparatus and method for converting Value Object | |
| CN112163025A (en) | Database data exporting method and device, computer equipment and storage medium | |
| Dou et al. | Scientific workflow design 2.0: Demonstrating streaming data collections in Kepler | |
| US6697817B2 (en) | Variable-length database apparatus and method for accessing the same | |
| CN104537012B (en) | Data processing method and device | |
| CN101770367A (en) | Compressing method and compressing device of .NET file | |
| CN116775599A (en) | Data migration method, device, electronic equipment and storage medium | |
| US11036616B2 (en) | Tracing the data processing activities of a data processing apparatus | |
| CN108846059A (en) | OpenFOAM limited bulk analysis result data format and its conversion method towards result post-processing | |
| CN110807092B (en) | Data processing method and device | |
| CN110825846B (en) | Data processing method and device | |
| CN108121807A (en) | The implementation method of multi-dimensional index structures OBF-Index under Hadoop environment | |
| CN116521063B (en) | Efficient test data reading and writing method and device for HDF5 | |
| NasiriGerdeh et al. | Root files for computer scientists | |
| JP2001331353A (en) | Data input system to database, and recording medium in which its program is stored |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190628 |
|
| RJ01 | Rejection of invention patent application after publication |