CN108255953A - Data processing method and processing device - Google Patents
Data processing method and processing device Download PDFInfo
- Publication number
- CN108255953A CN108255953A CN201711382098.1A CN201711382098A CN108255953A CN 108255953 A CN108255953 A CN 108255953A CN 201711382098 A CN201711382098 A CN 201711382098A CN 108255953 A CN108255953 A CN 108255953A
- Authority
- CN
- China
- Prior art keywords
- metadata
- business datum
- data
- database
- specified
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 48
- 238000003672 processing method Methods 0.000 title claims abstract description 15
- 238000004140 cleaning Methods 0.000 claims description 11
- 238000004519 manufacturing process Methods 0.000 claims description 11
- 239000000284 extract Substances 0.000 claims description 2
- 238000000034 method Methods 0.000 description 15
- 230000008569 process Effects 0.000 description 8
- 210000004080 milk Anatomy 0.000 description 5
- 235000013336 milk Nutrition 0.000 description 5
- 239000004094 surface-active agent Substances 0.000 description 5
- 230000002159 abnormal effect Effects 0.000 description 4
- 239000006071 cream Substances 0.000 description 4
- 239000002453 shampoo Substances 0.000 description 4
- 239000000344 soap Substances 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 239000008267 milk Substances 0.000 description 3
- 238000003860 storage Methods 0.000 description 3
- 238000005406 washing Methods 0.000 description 3
- 238000010276 construction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 238000004904 shortening Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
Abstract
The invention provides a data processing method and a data processing device, which comprise the following steps: predetermining at least one database; for each database in the at least one database, acquiring at least two service data from the database; determining metadata corresponding to each service data; searching at least one piece of specified metadata corresponding to a preset data mart from each piece of determined metadata; and determining the business data corresponding to each piece of specified metadata as the specified business data corresponding to the data mart. The scheme can improve the data processing efficiency.
Description
Technical field
The present invention relates to network communication technology field, more particularly to a kind of data processing method and processing unit.
Background technology
With the rapid development of Internet, the network data increasingly expanded make Internet user gradually got lost in information you
Ocean among, the data for how handling magnanimity have become the emphasis of each enterprises pay attention.
At present, effective information is searched from the data of magnanimity by enterprise, is mainly realized by way of manually setting script.
But when there is new business demand in enterprise, staff then needs to change corresponding script, and work can be expended by changing script
Personnel's more time, therefore the treatment effeciency of data can be reduced.
Invention content
An embodiment of the present invention provides a kind of data processing method and processing units, can improve the treatment effeciency of data.
In a first aspect, an embodiment of the present invention provides a kind of data processing method, including：
Predefine at least one database；
For each described database at least one database, at least two are obtained from the database
Business datum；
Determine the corresponding metadata of each described business datum；
It is searched and the corresponding at least one specified metadata of preset Data Mart from each determining metadata；
Determine that each corresponding described business datum of specified metadata is and the corresponding specified industry of the Data Mart
Business data.
Preferably, it is described that at least two business datums are obtained from the database, including：
According to preset data acquisition range and each acquisition quantity, corresponding at least two are obtained from the database
A business datum；
It is described obtain at least two business datums from the database after, it is described determine each described business
Before the corresponding metadata of data, further comprise：
For database each described, record obtains the batch of the business datum from the database each time
Number, obtain the time and production serial number.
Preferably, it is described obtain at least two business datums from the database after, it is described determine each
Before the corresponding metadata of the business datum, further comprise：
Go the redundant data in the business datum unless each；
The data requirement of the business datum after each unified removal redundant data.
Preferably, it is described determine each cleaning after the corresponding metadata of the business datum after, it is described from
It is found out in the business datum after cleaning with before the corresponding specified services data of preset type of service, further wrapping
It includes：
Go redundancy metadata in the metadata unless each；
According to preset metadata type, at least one missing is extracted from the metadata after removal redundancy metadata
Metadata；
It is described to be searched and the corresponding at least one specified member of preset Data Mart from each determining metadata
Data, including：
From the metadata after each removal redundancy metadata and each described missing metadata, search and pre-
If the corresponding at least one specified metadata of Data Mart.
Preferably, determine that each corresponding described business datum of specified metadata is and the Data Mart phase described
Before corresponding specified services data, further comprise：
Determine each theme in preset Data Mart；
It is described to be searched and the corresponding at least one specified member of preset Data Mart from each determining metadata
Data, including：
For theme each described, searched and the corresponding specified metadata of the theme from each metadata.
Second aspect, an embodiment of the present invention provides a kind of data processing equipment, including：
Processing unit, for predefining at least one database；Determine the corresponding first number of each business datum
According to and determine that each specified corresponding described business datum of metadata is and the corresponding specified services number of Data Mart
According to.
Acquiring unit, for each the described number being directed at least one database that the processing unit determines
According to library, at least two business datums are obtained from the database；
Searching unit, for being searched and preset Data Mart phase from each metadata that the processing unit determines
Corresponding at least one specified metadata.
Preferably, further comprise：Recording unit；
The acquiring unit, for according to preset data acquisition range and each acquisition quantity, from the database
It is middle to obtain corresponding at least two business datum；
The recording unit for being directed to each described database, records the acquiring unit each time from the number
According to batch number, acquisition time and the production serial number that the business datum is obtained in library.
Preferably, the processing unit, the redundant data being further used in the business datum unless each；It is unified every
The data requirement of the business datum after one removal redundant data.
Preferably, the processing unit is further used for redundancy metadata in the metadata unless each；According to default
Metadata type, from removal redundancy metadata after the metadata in extract at least one missing metadata；
The searching unit, the metadata being further used for after each removal redundancy metadata and each institute
It states in missing metadata, searches and the corresponding at least one specified metadata of preset Data Mart.
Preferably, the processing unit is further used for determining each theme in preset Data Mart；
The searching unit is further used for for each described theme, searched from each metadata with it is described
The corresponding specified metadata of theme.
In embodiments of the present invention, it after business datum is got from database, needs first to determine for each use
The metadata of the features such as content, quality and the situation of bright business datum, then determine the preset corresponding specified member of Data Mart
Data, and when there is new business demand, it only need to be according to the corresponding specified metadata of Data Mart, you can determine that new business needs
Required specified services data are sought, it can thus be avoided staff changes preset script when there is new business demand, from
And the time that business personnel handles data can be reduced, so as to improve the efficiency of data processing.
Description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, to embodiment or will show below
There is attached drawing needed in technology description to be briefly described, it should be apparent that, the accompanying drawings in the following description is the present invention
Some embodiments, for those of ordinary skill in the art, without creative efforts, can also basis
These attached drawings obtain other attached drawings.
Fig. 1 is a kind of flow chart for data processing method that one embodiment of the invention provides；
Fig. 2 is the flow chart for another data processing method that one embodiment of the invention provides；
Fig. 3 is a kind of structure diagram for data processing equipment that one embodiment of the invention provides；
Fig. 4 is the structure diagram for another data processing equipment that one embodiment of the invention provides.
Specific embodiment
Purpose, technical scheme and advantage to make the embodiment of the present invention are clearer, below in conjunction with the embodiment of the present invention
In attached drawing, the technical solution in the embodiment of the present invention is clearly and completely described, it is clear that described embodiment is
Part of the embodiment of the present invention, instead of all the embodiments, based on the embodiments of the present invention, those of ordinary skill in the art
The all other embodiments obtained under the premise of creative work is not made, shall fall within the protection scope of the present invention.
As shown in Figure 1, an embodiment of the present invention provides a kind of data processing methods, which is characterized in that including：
Step 101：Predefine at least one database；
Step 102：For each described database at least one database, obtained from the database
At least two business datums；
Step 103：Determine the corresponding metadata of each described business datum；
Step 104：It is searched and the corresponding at least one finger of preset Data Mart from each determining metadata
Determine metadata；
Step 105：It is corresponding with the Data Mart to determine each corresponding described business datum of specified metadata
Specified services data.
In embodiments of the present invention, it after business datum is got from database, needs first to determine for each use
The metadata of the features such as content, quality and the situation of bright business datum, then determine the preset corresponding specified member of Data Mart
Data, and when there is new business demand, it only need to be according to the corresponding specified metadata of Data Mart, you can determine that new business needs
Required specified services data are sought, it can thus be avoided staff changes preset script when there is new business demand, from
And the time that business personnel handles data can be reduced, so as to improve the efficiency of data processing.
In an embodiment of the present invention, it is described that at least two business datums are obtained from the database, including：
According to preset data acquisition range and each acquisition quantity, corresponding at least two are obtained from the database
A business datum；
It is described obtain at least two business datums from the database after, it is described determine each described business
Before the corresponding metadata of data, further comprise：
For database each described, record obtains the batch of the business datum from the database each time
Number, obtain the time and production serial number.
In embodiments of the present invention, the business datum of acquisition is being gathered to sub corresponding preposition from each database
After in library, the batch number, acquisition time and the production serial number that are obtained for each data-base recording business datum are needed, is come real
Now the business datum of acquisition and business data processing process are identified, so that during business datum appearance exception, it can be to industry
The source for data of being engaged in, the processing procedure of business datum are traced.
In an embodiment of the present invention, it is described obtain at least two business datums from the database after, in institute
It states before determining the corresponding metadata of each described business datum, further comprises：
Go the redundant data in the business datum unless each；
The data requirement of the business datum after each unified removal redundant data.
In embodiments of the present invention, it after the business datum obtained well is gathered in original library, needs to getting
Business datum is cleaned, that is, is removed redundant data, uniform traffic data requirement, the mistake for correcting business datum, corrected business
The logic of data, the construction of transformation service data and the business datum that incompleteness is supplied according to the business datum got, and remember
Cleaning process, wash result, operating personnel and the context of business datum variation of business datum are recorded, so that business number
It can trace to the source according to when occurring abnormal after processing.
In an embodiment of the present invention, it is described determine each cleaning after the corresponding metadata of the business datum it
Afterwards, found out in the business datum after cleaning with the corresponding specified services data of preset type of service it
Before, further comprise：
Go redundancy metadata in the metadata unless each；
According to preset metadata type, at least one missing is extracted from the metadata after removal redundancy metadata
Metadata；
It is described to be searched and the corresponding at least one specified member of preset Data Mart from each determining metadata
Data, including：
From the metadata after each removal redundancy metadata and each described missing metadata, search and pre-
If the corresponding at least one specified metadata of Data Mart.
In embodiments of the present invention, it after the corresponding metadata of business datum is determined, needs to advise according to preset metadata
The metadata that model cleaning is determined, the i.e. synonymous metadata of metadata changing metadata title, different name to the same name different defining are only stayed
One, the unit of unified metadata, then to existing metadata refine the metadata of the additional missing of adjustment, so as to improve member
The statement of data, finally corresponding to each metadata according to the integrality of business datum, consistency, accuracy and promptness
Business datum is assessed and is marked, so as to ensure the levels of precision of business data processing result and reliable journey to the greatest extent
Degree.
In an embodiment of the present invention, it is described determine each specified corresponding described business datum of metadata for institute
Before stating the corresponding specified services data of Data Mart, further comprise：
Determine each theme in preset Data Mart；
It is described to be searched and the corresponding at least one specified member of preset Data Mart from each determining metadata
Data, including：
For theme each described, searched and the corresponding specified metadata of the theme from each metadata.
In embodiments of the present invention, by determining each theme in the Data Mart based on sciemtifec and technical sphere model, i.e.,
It can be from the corresponding business datum of fixed metadata lookup, the i.e. associated business datum of attribute, then the business that will be found
Data are sent to each enterprise (for example, application library), so that each enterprise specifies from the business datum received and implements section
Business decision.
In order to more clearly illustrate technical scheme of the present invention and advantage, to a kind of data provided in an embodiment of the present invention
Processing method is described in detail, as shown in Fig. 2, this method may comprise steps of：
Step 201：Business datum is obtained from predetermined each database.
Specifically, when obtaining business datum from disparate databases, the preset concurrentization of Main Basiss, not
With area deployment data exchange client, further according to preset data acquisition range and the quantity obtained every time, from each data
Corresponding business datum is obtained in library.
For example, the database b of Beijing and the database j of Jinan City are predefined；
The quantity of first quarter business datum in 2017 is 160GB in database b, the quantity of second quarter business datum is
90GB, third season business datum quantity be 60GB；
The quantity of first quarter business datum in 2017 is 120GB in database j, the quantity of second quarter business datum is
40GB, third season business datum quantity be 50GB；
The business datum for ranging from obtaining the third season in 2017 is obtained according to business datum and obtains business datum every time
Quantity for 30GB, need to obtain the business datum of the third season in 2017, and the industry that will be got in two times from database b
Business data are gathered in the corresponding front damming b of Beijing database b；
And obtain the business datum of the third season in 2017, the business datum that will be got in two times from database j
It gathers in the corresponding front damming j of Jinan City database j.
Step 202：The batch number of business datum obtained from each database is recorded, obtain the time and produces flowing water
Number.
Specifically, the batch number of record traffic data, acquisition time and production serial number, can cause business datum
Processing procedure is clearer and more definite, and can trace to the source when finding business datum exception, so as to improve the business datum of acquisition
Safety.
For example, the batch number of business datum obtained for the first time in record front damming b is bj2017112901, is put in storage
Time be 12 minutes 14 points of on November 29th, 2017, second of business datum obtained batch number be bj2017112902, storage
Time is 35 minutes 14 points of on November 29th, 2017 and production serial number is bj4jdsj, and the business datum of acquisition is pumped into
In original library.
The batch number of business datum obtained for the first time in record front damming j is jn2017112901, and entry time is
15 minutes 14 points of on November 29th, 2017, second of business datum obtained batch number be jn2017112902, entry time is
25 minutes 14 points of on November 29th, 2017 and production serial number are jn4jdsj, and the business datum of acquisition is pumped into original library
In.
Step 203：Remove the redundant data in each business datum obtained.
Specifically, there can be the business datum largely repeated in the business datum of acquisition, and reducing redundant data can contract
Weakness manages the time of business datum, so as to be conducive to improve the treatment effeciency of business datum.
For example, it is from the front damming b business datums extracted：
User a is in Shanghai City Dongchuan road XX universities；
The order amount of money of user a is 15 times below 100；
The order amount of money of user a is 3 times in 100 yuan of -200 yuan of sections；
The order amount of money of user a is 3 times in 200 yuan of -300 yuan of sections；
The perfumed soap 3 times of the shampoo 7 times of bought 60 yuan of user a, 90 yuan of shower cream 5 times and 50 yuan；
The B brands clothes for 120 yuan of the A brands clothes 1 time and 180 yuan that user a is bought 2 times；
The B brands surfactant for 260 yuan of A brands skin care milks 2 times and 298 yuan that user a is bought 1 time；
Removing the business datum after redundant data is：
The order amount of money of user a is 15 times below 100,100 yuan of -200 yuan of sections are 3 times and 200 yuan of -300 yuan of sections
It is 3 times；
60 yuan of shampoo 7 times of purchase, 90 yuan of shower cream 5 times, 50 yuan of perfumed soap 3 times, 120 yuan of A brands clothes 1
Secondary, 180 yuan of 2 times, 260 yuan A brands skin care milks 2 times and 298 yuan of B brands clothes B brands surfactant 1 time.
Step 204：The data requirement of business datum after each unified removal redundant data.
Specifically, in order to avoid semantic conflict occurs in the business datum obtained from disparate databases, definition integrity is needed
The data requirement of business datum for constraining uniformly to obtain, and the cleaning process of record traffic data, wash result, operator
Member and the context of business datum variation, so as to can trace to the source when occurring abnormal after business data processing.
For example, the business datum font obtained from front damming b be " Song typeface ", the business obtained from front damming j
The font of data is " lishu ", and the business datum font in original library is unified for " regular script ".
Step 205：Determine the corresponding metadata of business datum after each removal redundant data.
In particular it is required that the features such as the content, quality and situation of the business datum of acquisition are described with metadata, it can
To support such as to indicate storage location, historical data, resource lookup, file record work(, and then reach to assist business datum retrieval
Purpose.
For example, the corresponding metadata of Shanghai City Dongchuan road XX universities is address；
It is 15 times below 100, in 100 yuan of -200 yuan of sections is 3 times and is 3 correspondences in 200 yuan of -300 yuan of sections
Metadata be consumption number of times；
60 yuan, 90 yuan, 50 yuan, 120 yuan, 180 yuan and 260 yuan corresponding metadata are spending amount；
Business datum shampoo, shower cream and the corresponding metadata of perfumed soap are purchase washing product；
Business datum A brands clothes and the corresponding metadata of B brand clothes are purchase clothing；
Business datum A brands skin care milk and the corresponding metadata of B brand surfactant are purchase skin care item.
Step 206：Go redundancy metadata in metadata unless each.
Specifically, after the corresponding metadata of each business datum is determined, it is understood that there may be the metadata of the same name different defining is different
The synonymous metadata of name, so needing to refine existing metadata, so that shortening the corresponding business of processing metadata
The time of data.
For example, metadata is removed for purchase skin care item as redundancy metadata, by A brands skin care milk and B brands
It is to buy in the business datum of washing product that surfactant, which is added to metadata,.
Step 207：According to preset metadata type, extracted from the metadata after removal redundancy metadata at least one
Lack metadata.
Specifically, metadata is the summary to the content expressed by business datum, the corresponding business datum of metadata can
It can be there are multiple contents, so needing, according to preset metadata type, i.e. metadata specification, to carry out the metadata after refinement
The metadata of additional missing, so as to improve the statement of metadata.
For example, the business datum obtained from front damming j is that corresponding metadata is：
Address=Shanghai City Dongchuan road XX universities, it is possible to extract missing metadata：School=XX universities.
Step 208：Determine each theme in Data Mart.
Specifically, the Data Mart based on sciemtifec and technical sphere model is integrated, subject-oriented a data acquisition system, can be with
Meet the needs of different departments or user, determine theme needed for Data Mart, in order to which corresponding business datum is deposited
It stores up in Data Mart.
Specifically, the theme a in Data Mart is shopping environment, theme b is enterprise getting profit situation.
Step 209：For each theme, searched and theme phase from each removal redundancy metadata and missing metadata
Corresponding specified metadata.
Specifically, each theme corresponds to certain data attribute, and can be determined according to data attribute corresponding
Specified metadata.
For example, the corresponding specified metadata of the shopping environment of theme a is purchase washing product, purchase clothing, consumes
The amount of money and consumption number of times.
Step 210：Determine that each corresponding business datum of specified metadata is and the corresponding specified industry of Data Mart
Business data.
Specifically, after the specified metadata that the theme in Data Mart is determined, you can correspond to specified metadata
Business datum be sent to each enterprise, i.e. application library so that each enterprise specifies and implements according to the business datum received
The business decision of science.
For example, be less than 100 with the corresponding specified services data of Data Mart it is 15 times, at 100 yuan -200
First section be 3 times and be 3 times, 60 yuan, 90 yuan, 50 yuan, 120 yuan, 180 yuan and 260 yuan in 200 yuan of -300 yuan of sections, shampoo,
Shower cream, perfumed soap, A brands clothes, B brands clothes, A brands skin care milk and B brand surfactant.
As shown in figure 3, the embodiment of the present invention provides a kind of data processing equipment, including：
Processing unit 301, for predefining at least one database；Determine the corresponding member of each business datum
Data and determine that each specified corresponding described business datum of metadata is and the corresponding specified services number of Data Mart
According to.
Acquiring unit 302, for being directed to each in the determining at least one database of the processing unit 301
The database obtains at least two business datums from the database；
Searching unit 303, for being searched and preset data from each metadata that the processing unit 301 determines
The corresponding at least one specified metadata in fairground.
As shown in figure 4, the processing unit, further comprises：Recording unit 401；
The acquiring unit 302, for according to preset data acquisition range and each acquisition quantity, from the data
Corresponding at least two business datum is obtained in library；
The recording unit 401, for being directed to each described database, record the acquiring unit 302 each time from
The batch number of the business datum is obtained in the database, obtains time and production serial number.
In an embodiment of the present invention, the processing unit is further used for superfluous in the business datum unless each
Remainder evidence；The data requirement of the business datum after each unified removal redundant data.
In an embodiment of the present invention, the processing unit is further used for redundancy member in the metadata unless each
Data；According to preset metadata type, at least one missing member is extracted from the metadata after removal redundancy metadata
Data；
The searching unit, the metadata being further used for after each removal redundancy metadata and each institute
It states in missing metadata, searches and the corresponding at least one specified metadata of preset Data Mart.
In an embodiment of the present invention, the processing unit is further used for determining each in preset Data Mart
A theme；
The searching unit is further used for for each described theme, searched from each metadata with it is described
The corresponding specified metadata of theme.
The each embodiment of the present invention at least has the advantages that：
1st, in an embodiment of the present invention, it after business datum is got from database, needs first to determine each use
To illustrate the metadata of the features such as the content of business datum, quality and situation, then determine the preset corresponding finger of Data Mart
Determine metadata, and when there is new business demand, it only need to be according to the corresponding specified metadata of Data Mart, you can determine new industry
Specified services data needed for business demand, it can thus be avoided staff changes preset foot when there is new business demand
This, so as to reduce the time that business personnel handles data, so as to improve the efficiency of data processing.
2nd, in an embodiment of the present invention, the business datum of acquisition is being gathered to sub corresponding from each database
After in front damming, the batch number, acquisition time and the production serial number that are obtained for each data-base recording business datum are needed,
Realize that business datum and business data processing process to acquisition are identified, it, can be with during so that business datum occurring abnormal
The processing procedure in source, business datum to business datum traces.
3rd, in an embodiment of the present invention, it after the business datum obtained well is gathered in original library, needs to obtaining
To business datum cleaned, that is, remove redundant data, uniform traffic data requirement, correct business datum mistake, correct
The logic of business datum, the construction of transformation service data and the business datum that incompleteness is supplied according to the business datum got,
And cleaning process, wash result, operating personnel and the context of business datum variation of record traffic data, so that industry
It can trace to the source when occurring abnormal after business data processing.
4th, in an embodiment of the present invention, it after the corresponding metadata of business datum is determined, needs according to preset first number
According to the metadata that specification cleaning is determined, the i.e. synonymous metadata of metadata changing metadata title, different name to the same name different defining
One, the unit of unified metadata are only stayed, then to existing metadata refine the metadata of the additional missing of adjustment, so that complete
The statement of kind metadata, finally according to the integrality of business datum, consistency, accuracy and promptness to each metadata pair
The business datum answered is assessed and is marked, so as to ensure the levels of precision of business data processing result and reliable to the greatest extent
Degree.
5th, in an embodiment of the present invention, by determining each master in the Data Mart based on sciemtifec and technical sphere model
Topic, you can from the corresponding business datum of fixed metadata lookup, the i.e. associated business datum of attribute, then will find
Business datum is sent to each enterprise (for example, application library), so that each enterprise specifies and real from the business datum received
Apply the business decision of science.
It should be noted that herein, such as first and second etc relational terms are used merely to an entity
Or operation is distinguished with another entity or operation, is existed without necessarily requiring or implying between these entities or operation
Any actual relationship or order.Moreover, term " comprising ", "comprising" or its any other variant be intended to it is non-
It is exclusive to include, so that process, method, article or equipment including a series of elements not only include those elements,
But also it including other elements that are not explicitly listed or further includes solid by this process, method, article or equipment
Some elements.In the absence of more restrictions, the element limited by sentence " including a 〃 〃 ", it is not excluded that
Also there is other identical factor in the process, method, article or apparatus that includes the element.
It is last it should be noted that：The foregoing is merely presently preferred embodiments of the present invention, is merely to illustrate the skill of the present invention
Art scheme, is not intended to limit the scope of the present invention.Any modification for being made all within the spirits and principles of the present invention,
Equivalent replacement, improvement etc., are all contained in protection scope of the present invention.
Claims (10)
1. a kind of data processing method, which is characterized in that including：
Predefine at least one database；
For each described database at least one database, at least two business are obtained from the database
Data；
Determine the corresponding metadata of each described business datum；
It is searched and the corresponding at least one specified metadata of preset Data Mart from each determining metadata；
Determine that each corresponding described business datum of specified metadata is and the corresponding specified services number of the Data Mart
According to.
2. processing method according to claim 1, which is characterized in that
It is described that at least two business datums are obtained from the database, including：
According to preset data acquisition range and each acquisition quantity, corresponding at least two industry is obtained from the database
Business data；
It is described obtain at least two business datums from the database after, it is described determine each described business datum
Before corresponding metadata, further comprise：
For database each described, record obtains the batch number of the business datum from the database, obtains each time
Take time and production serial number.
3. processing method according to claim 1, which is characterized in that
It is described obtain at least two business datums from the database after, it is described determine each described business datum
Before corresponding metadata, further comprise：
Go the redundant data in the business datum unless each；
The data requirement of the business datum after each unified removal redundant data.
4. processing method according to claim 1, which is characterized in that
It is described determine each cleaning after the corresponding metadata of the business datum after, it is described after cleaning described in
It is found out in business datum with before the corresponding specified services data of preset type of service, further comprising：
Go redundancy metadata in the metadata unless each；
According to preset metadata type, the first number of at least one missing is extracted from the metadata after removal redundancy metadata
According to；
The lookup from each determining metadata and the corresponding at least one specified metadata of preset Data Mart,
Including：
From each removal redundancy metadata after described metadata and each it is described missing metadata in, search with it is preset
The corresponding at least one specified metadata of Data Mart.
5. according to the processing method any in Claims 1-4, which is characterized in that
Determine that each corresponding described business datum of specified metadata is corresponding specified with the Data Mart described
Before business datum, further comprise：
Determine each theme in preset Data Mart；
The lookup from each determining metadata and the corresponding at least one specified metadata of preset Data Mart,
Including：
For theme each described, searched and the corresponding specified metadata of the theme from each metadata.
6. a kind of data processing equipment, which is characterized in that including：
Processing unit, for predefining at least one database；Determine the corresponding metadata of each business datum, with
And determine that each corresponding described business datum of specified metadata is and the corresponding specified services data of Data Mart.
Acquiring unit, for each the described data being directed at least one database that the processing unit determines
Library obtains at least two business datums from the database；
Searching unit, it is corresponding with preset Data Mart for being searched from each metadata that the processing unit determines
At least one specified metadata.
7. processing unit according to claim 6, which is characterized in that further comprise：Recording unit；
The acquiring unit, for according to preset data acquisition range and each acquisition quantity, being obtained from the database
Take corresponding at least two business datum；
The recording unit for being directed to each described database, records the acquiring unit each time from the database
The middle batch number for obtaining the business datum obtains time and production serial number.
8. processing unit according to claim 6, which is characterized in that
The processing unit, the redundant data being further used in the business datum unless each；Each unified removal is superfluous
The data requirement of the business datum of the remainder after.
9. processing unit according to claim 6, which is characterized in that
The processing unit is further used for redundancy metadata in the metadata unless each；According to preset metadata category
Type extracts at least one missing metadata from the metadata after removal redundancy metadata；
It is described scarce with each to be further used for the metadata after each removal redundancy metadata for the searching unit
It loses in metadata, searches and the corresponding at least one specified metadata of preset Data Mart.
10. according to the processing unit any in claim 6 to 9, which is characterized in that
The processing unit is further used for determining each theme in preset Data Mart；
The searching unit is further used for, for each described theme, searching and the theme from each metadata
Corresponding specified metadata.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711382098.1A CN108255953A (en) | 2017-12-20 | 2017-12-20 | Data processing method and processing device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711382098.1A CN108255953A (en) | 2017-12-20 | 2017-12-20 | Data processing method and processing device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108255953A true CN108255953A (en) | 2018-07-06 |
Family
ID=62723378
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711382098.1A Pending CN108255953A (en) | 2017-12-20 | 2017-12-20 | Data processing method and processing device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108255953A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112311686A (en) * | 2020-09-27 | 2021-02-02 | 长沙市到家悠享网络科技有限公司 | Data processing method and device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101739454A (en) * | 2009-12-29 | 2010-06-16 | 用友软件股份有限公司 | Data processing system |
| CN102118315A (en) * | 2011-02-28 | 2011-07-06 | 华为技术有限公司 | Method for fluidizing, recording and reading data and system adopting same |
| CN103942245A (en) * | 2014-02-19 | 2014-07-23 | 浪潮软件股份有限公司 | Data extracting method based on metadata |
| CN106294492A (en) * | 2015-06-08 | 2017-01-04 | 深圳中兴网信科技有限公司 | Data cleaning method and cleaning engine |
-
2017
- 2017-12-20 CN CN201711382098.1A patent/CN108255953A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101739454A (en) * | 2009-12-29 | 2010-06-16 | 用友软件股份有限公司 | Data processing system |
| CN102118315A (en) * | 2011-02-28 | 2011-07-06 | 华为技术有限公司 | Method for fluidizing, recording and reading data and system adopting same |
| CN103942245A (en) * | 2014-02-19 | 2014-07-23 | 浪潮软件股份有限公司 | Data extracting method based on metadata |
| CN106294492A (en) * | 2015-06-08 | 2017-01-04 | 深圳中兴网信科技有限公司 | Data cleaning method and cleaning engine |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112311686A (en) * | 2020-09-27 | 2021-02-02 | 长沙市到家悠享网络科技有限公司 | Data processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10824675B2 (en) | Resource-efficient generation of a knowledge graph | |
| US20180165712A1 (en) | Method and apparatus for composing search phrases, distributing ads and searching product information | |
| JP5869662B2 (en) | System, method and computer program for managing user bookmark data | |
| CN108108426A (en) | Understanding method, device and the electronic equipment that natural language is putd question to | |
| CN104090886A (en) | Method and device for constructing real-time portrayal of user | |
| WO2014008139A2 (en) | Generating search results | |
| CN109299219A (en) | Data query method, apparatus, electronic equipment and computer readable storage medium | |
| WO2020098315A1 (en) | Information matching method and terminal | |
| CN108428166A (en) | The clothes commending system of figure and features feature recognition classification based on convolutional neural networks | |
| US9239863B2 (en) | Method and apparatus for graphic code database updates and search | |
| CN109885772A (en) | The education content personalized recommendation system of knowledge based map | |
| CN110059177A (en) | A kind of activity recommendation method and device based on user's portrait | |
| CN112163160A (en) | Knowledge graph-based sensitive identification method | |
| CN105468649A (en) | Method and apparatus for determining matching of to-be-displayed object | |
| KR102301663B1 (en) | Identifying physical objects using visual search query | |
| CN108255953A (en) | Data processing method and processing device | |
| CN106156260B (en) | Method and device for repairing missing data | |
| CN111967970B (en) | Bank product recommendation method and device based on spark platform | |
| CN104298786B (en) | A kind of image search method and device | |
| CN107818117A (en) | A kind of method for building up of tables of data, online query method and relevant apparatus | |
| Liu et al. | srvpa: A multi-domain conversational service recommendation approach | |
| WO2021062959A1 (en) | Data processing method and apparatus for business objects | |
| CN110751511A (en) | Integral processing method and device based on user attributes | |
| CN107704105A (en) | Input reminding method, device, electronic equipment and computer-readable recording medium | |
| CN113269616B (en) | Multi-layer shopping recommendation method oriented to graphic neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180706 |