CN110750568A - Data recombination method and system for big data platform - Google Patents
Data recombination method and system for big data platform Download PDFInfo
- Publication number
- CN110750568A CN110750568A CN201910948061.3A CN201910948061A CN110750568A CN 110750568 A CN110750568 A CN 110750568A CN 201910948061 A CN201910948061 A CN 201910948061A CN 110750568 A CN110750568 A CN 110750568A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- acquisition
- output port
- port
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 32
- 230000006798 recombination Effects 0.000 title claims description 5
- 238000005215 recombination Methods 0.000 title claims description 5
- 238000013461 design Methods 0.000 claims abstract description 69
- 238000004140 cleaning Methods 0.000 claims abstract description 38
- 230000008521 reorganization Effects 0.000 claims abstract description 26
- 238000013500 data storage Methods 0.000 claims description 10
- 238000003860 storage Methods 0.000 description 15
- 238000012545 processing Methods 0.000 description 12
- 238000004891 communication Methods 0.000 description 9
- 238000007726 management method Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000004590 computer program Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000012795 verification Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000007405 data analysis Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000003825 pressing Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 239000004566 building material Substances 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000000556 factor analysis Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2246—Trees, e.g. B+trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/16—Real estate
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Tourism & Hospitality (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Economics (AREA)
- General Health & Medical Sciences (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Stored Programmes (AREA)
Abstract
The application provides a data reorganization method of a big data platform, which comprises the following steps: acquiring target acquisition data through a target acquisition port, wherein the target acquisition data comprises design acquisition data, cost acquisition data, purchase acquisition data and engineering acquisition data; cleaning the target acquisition data to obtain cleaning data; and recombining the cleaning data into target service data, and outputting the target service data through a target output port, wherein the target service data comprises design service data, cost service data and engineering service data.
Description
Technical Field
The application relates to the technical field of big data, in particular to a data reorganization method and system of a big data platform.
Background
With the development of information technology, big data becomes a product of deep integration of information technology and novel industries such as electronic commerce, intelligent manufacturing and the like. In this time-of-day context, big data also triggers the informatization management of the real estate industry.
At present, the informationized management of the real estate industry based on a large data platform mainly has the following problems that firstly, various platform systems are complicated, data segmentation is caused, data cannot be analyzed in multiple angles, and an enterprise lacks accurate and efficient data support in the aspects of decision making, investment, operation and maintenance and service; secondly, in the process of using the big data platform, corresponding configuration needs to be made according to different business requirements, and a large amount of manpower is consumed. Therefore, how to realize the fine operation of enterprise data and further promote the management of the real estate industry is still a hot issue to be solved urgently.
Disclosure of Invention
The application provides a data reorganization method and a data reorganization system of a big data platform, which can be used for carrying out effective information management on the real estate industry.
In a first aspect, the present application provides a data reorganization method for a big data platform, including:
acquiring target acquisition data through a target acquisition port, wherein the target acquisition data is design acquisition data under the condition that the target acquisition port is a design acquisition port, the target acquisition data is cost acquisition data under the condition that the target acquisition port is a cost acquisition port, the target acquisition data is purchase acquisition data under the condition that the target acquisition port is a purchase acquisition port, and the target acquisition data is engineering acquisition data under the condition that the target acquisition port is an engineering acquisition port;
cleaning the target acquisition data to obtain cleaning data;
and recombining the cleaning data into target service data, and outputting the target service data through a target output port, wherein when the target output port is a design output port, the target service data is design service data, when the target output port is a cost output port, the target service data is cost service data, when the target acquisition port is a purchase acquisition port, the target acquisition data is purchase acquisition data, and when the target output port is an engineering output port, the target output data is engineering service data.
In some possible embodiments, the design acquisition port, the cost acquisition port, the procurement acquisition port, and the engineering acquisition port are isolated from one another; the design output port, the cost output port, the purchase output port and the engineering output port are isolated from each other.
In some possible embodiments, the method further comprises: receiving a query request input by a user, wherein the query request carries an identity of the user, and the query request is used for querying the target service data; determining that the user has the authority to query the target service data; under the condition that the user is determined to have the authority of inquiring the target service data, acquiring the target service data from the target output port; and rejecting the query request under the condition that the user is determined not to have the authority of querying the target service data.
In some possible embodiments, the method further comprises: when the target output port is the design output port, pushing the target service data into a first B-tree, wherein the first B-tree is only used for storing the design service data; when the target output port is the cost output port, the target service data is pressed into a second B-tree, wherein the second B-tree is only used for storing the cost service data; when the target output port is the purchase output port, the target business data is pressed into a third B tree, wherein the third B tree is only used for storing the purchase business data; and when the target output port is the engineering output port, the target service data is pressed into a fourth B-tree, wherein the fourth B-tree is only used for storing the engineering service data.
In some possible embodiments, the first, second, third and fourth B-trees are isolated from each other.
According to the method, the target acquisition data is acquired through the target acquisition port, then the target acquisition data is cleaned to obtain the cleaning data, the cleaning data is recombined to obtain the target service data, and the target service data is output through the target output port, so that a user can directly inquire data information of related services according to service requirements, the safety of data storage is improved, the calculation performance and the response speed are greatly optimized, the workload of manual participation is reduced, the error probability is reduced, and the inquiry efficiency is improved.
In a second aspect, the present application provides a data reorganization system of a big data platform, including:
the data acquisition unit is used for acquiring target acquisition data through a target acquisition port, wherein the target acquisition data is design acquisition data under the condition that the target acquisition port is a design acquisition port, the target acquisition data is cost acquisition data under the condition that the target acquisition port is a cost acquisition port, the target acquisition data is purchase acquisition data under the condition that the target acquisition port is a purchase acquisition port, and the target acquisition data is engineering acquisition data under the condition that the target acquisition port is an engineering acquisition port;
the data cleaning unit is used for cleaning the target data to obtain cleaning data;
and the data recombination unit is used for recombining the cleaning data into target service data and outputting the target service data through a target output port, wherein the target service data is design service data under the condition that the target output port is a design output port, the target service data is cost service data under the condition that the target output port is a cost output port, the target acquisition data is purchase acquisition data under the condition that the target output port is a purchase acquisition port, and the target output data is engineering service data under the condition that the target output port is an engineering output port.
In some possible embodiments, the design acquisition port, the cost acquisition port, the procurement acquisition port, and the engineering acquisition port are isolated from one another; the design output port, the cost output port, the purchase output port and the engineering output port are isolated from each other.
In some possible embodiments, the system further comprises: a data query unit, configured to receive a query request input by a user, where the query request carries an identity of the user, and the query request is used to query the target service data; the identity authentication unit is used for determining that the user has the authority to inquire the target service data; under the condition that the user is determined to have the authority of inquiring the target service data, acquiring the target service data from the target output port; and rejecting the query request under the condition that the user is determined not to have the authority of querying the target service data.
In some possible embodiments, the system further includes a data storage unit, where the data storage unit is specifically configured to, in a case where the target output port is the design output port, push the target traffic data into a first B-tree, where the first B-tree is only used for storing the design traffic data; when the target output port is the cost output port, the target service data is pressed into a second B-tree, wherein the second B-tree is only used for storing the cost service data; when the target output port is the purchase output port, the target business data is pressed into a third B tree, wherein the third B tree is only used for storing the purchase business data; and when the target output port is the engineering output port, the target service data is pressed into a fourth B-tree, wherein the fourth B-tree is only used for storing the engineering service data.
In some possible embodiments, the first, second, third and fourth B-trees are isolated from each other.
In a third aspect, the present application provides a data reorganization system of a big data platform, where the system includes a processor, a communication interface, and a memory; the memory is configured to store instructions, the processor is configured to execute the instructions, and the communication interface is configured to receive or transmit data; wherein the processor executes the instructions to perform the method of the first aspect or any specific implementation manner of the first aspect.
In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program for execution by hardware (e.g., a processor, etc.) to implement the method described in the first aspect or any specific implementation manner of the first aspect.
In a fifth aspect, the present application provides a computer program product comprising instructions for causing a data reorganization system of a big data platform to perform the method as described in the first aspect above or any specific implementation manner of the first aspect above when the computer program runs on the data reorganization system of the big data platform.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic flowchart of a data reorganization method of a big data platform according to an embodiment of the present application;
FIG. 2 is a schematic diagram of an architecture of a big data platform provided in an embodiment of the present application;
fig. 3 is a schematic structural diagram of a data reorganization system of a big data platform according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a data reorganization system of a large data platform according to another embodiment provided in an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the present application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the specification of the present application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
In particular implementations, the terminals described in embodiments of the invention include, but are not limited to, other portable devices such as mobile phones, laptop computers, or tablet computers having touch sensitive surfaces (e.g., touch screen displays and/or touch pads). It should be understood that in some embodiments, the device is not a portable communication device, but is a desktop computer having a touch-sensitive surface (e.g., a touch screen display and/or touchpad).
In the discussion that follows, a terminal that includes a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and/or joystick.
The terminal supports various applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disc burning application, a spreadsheet application, a gaming application, a telephone application, a video conferencing application, an email application, an instant messaging application, an exercise support application, a photo management application, a digital camera application, a web browsing application, a digital music player application, and/or a digital video player application.
Various applications that may be executed on the terminal may use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the terminal can be adjusted and/or changed between applications and/or within respective applications. In this way, a common physical architecture (e.g., touch-sensitive surface) of the terminal can support various applications with user interfaces that are intuitive and transparent to the user.
In this embodiment of the present invention, the terminal device may include various terminal devices such as a mobile phone, a tablet computer, a Personal Digital Assistant (PDA), a Mobile Internet Device (MID), a notebook computer, and an intelligent wearable device (e.g., an intelligent watch and an intelligent bracelet), which is not limited in this embodiment of the present invention.
The terminal supports various human-computer interaction modes, and the mode of acquiring information and instructions can be contact type or non-contact type. The touch mode may be that a user inputs information and instructions to the terminal through physical keys, or inputs information and instructions through virtual keys and/or touch keys, or inputs information and instructions through single-point or multi-point control by touching a terminal screen, for example, a user inputs instructions to the terminal through a physical keyboard. The terminal may acquire information and instructions by recognizing gestures or voices of the user, or the terminal may acquire information and instructions by tracking and positioning the movement process of the eyes or the head of the user, for example, the user may start a voice recognition function, and the terminal acquires the voice of the user to process and recognize the acquired instructions.
The embodiment of the application is based on a big data platform architecture, and a new management solution is brought to structured, semi-structured and unstructured multi-source data by utilizing a big data clustering and storing scheme combining Hadoop and a relational database. Meanwhile, the collected data is recombined into new data information meeting the requirements of users based on the business requirements of enterprises, and a rapid and accurate data retrieval service is provided for manual data query and system work application.
As shown in fig. 1, fig. 1 is a schematic flow chart of a data reorganization method of a big data platform provided in the present application. The data reorganization method of the big data platform of the embodiment includes, but is not limited to, the following steps:
s101, acquiring target acquisition data through a target acquisition port.
Under the condition that the target acquisition port is the design acquisition port, the target acquisition data is the design acquisition data; under the condition that the target acquisition port is the cost acquisition port, the target acquisition data is cost acquisition data; under the condition that the target acquisition port is a purchase acquisition port, the target acquisition data is purchase acquisition data; and under the condition that the target acquisition port is the engineering acquisition port, the target acquisition data is engineering acquisition data.
In the embodiment of the application, the design acquisition port, the cost acquisition port, the purchase acquisition port and the engineering acquisition port are isolated from each other.
In one possible embodiment, for the information management of the real estate industry, the design acquisition data acquired by the design acquisition port comprises house type design data, landscape design data and the like of design workers of an enterprise aiming at house construction; the cost acquisition data acquired by the cost acquisition port comprises cost budget data of projects and the like; the purchase acquisition data acquired by the purchase acquisition port comprises various building material data and the like; the engineering acquisition data acquired by the engineering acquisition port comprises data such as engineering progress and the like.
In some possible embodiments, the target collection data may be a form, a three-dimensional image, or web page information dynamically acquired by using a crawler technology, which is not limited herein.
And S102, cleaning the target collected data to obtain cleaning data.
In a possible embodiment, the acquired target collected data may be subjected to data analysis by different data formats or repeated through data analysis on the target collected data, and a preset cleaning rule is used to perform data cleaning on the target collected data, so as to obtain standard clean and continuous data, which is provided for subsequent data reconstruction for use, where the preset cleaning rule includes missing value processing, abnormal value processing, repeated value processing, null value filling, uniform unit, normalization processing, factor analysis, and the like, and the embodiment of the present application is not particularly limited.
S103, recombining the cleaning data into target service data, and outputting the target service data through a target output port.
Wherein, in the case that the target output port is a design output port, the target service data is design service data; when the target output port is a cost output port, the target service data is cost service data; under the condition that the target output port is a purchase acquisition port, the target acquisition data is purchase acquisition data; and under the condition that the target output port is an engineering output port, the target output data is engineering service data.
In the embodiment of the application, the design output port, the cost output port, the purchase output port and the engineering output port are isolated from each other.
In one possible embodiment, the characteristic attribute of the cleaning data is obtained by analyzing the cleaning data, the hash table is searched according to the characteristic attribute value, and the pointer of the cleaning data is inserted into the data structure indicated by the hash address corresponding to the keyword under the condition that the key value matched with the characteristic attribute value is found; and under the condition that a key value matched with the characteristic attribute value is not found, generating a new hash address corresponding to the characteristic attribute value and a corresponding data structure in the hash table, inserting a pointer of the cleaning data into the data structure indicated by the new hash address, and finally arranging the data in the same data structure in sequence to obtain the target service data.
And S104, storing the target service data into the corresponding B tree according to the target output port.
In this embodiment, when the target output port is a design output port, the target service data is pushed into a first B-tree, where the first B-tree is only used for storing design service data; when the target output port is a cost output port, the target service data is pressed into a second B-tree, wherein the second B-tree is only used for storing the cost service data; when the target output port is a purchase output port, the target business data is pressed into a third B tree, wherein the third B tree is only used for storing purchase business data; and when the target output port is an engineering output port, the target service data is pressed into a fourth B-tree, wherein the fourth B-tree is only used for storing the engineering service data.
In the embodiment of the present application, the first B-tree, the second B-tree, the third B-tree and the fourth B-tree are isolated from each other.
S105, receiving a query request input by a user, wherein the query request carries the identity of the user, and the query request is used for querying the target service data.
S106, determining that the user has the authority to inquire the target service data. If the user is determined to have the authority to query the target service data, executing S107; if it is determined that the user does not have the authority to query the target service data, S108 is performed.
And S107, acquiring the target service data from the target output port and sending the target service data to the user terminal.
And S108, sending the query rejection information to the user terminal.
In a possible embodiment, the terminal device sends a viewing request triggered by a user to a data reorganization system of the big data platform, and the data reorganization system determines whether the user can view the target service data according to user identity identification information in the viewing request, when the user can view the target service data. And the data reorganization system of the big data platform sends the target service data to the terminal equipment. For example, a design query entry, a cost query entry, a purchase query entry and an engineering query entry are displayed on the terminal device, and when a user needs to query the recent progress of a certain building engineering, the user can timely know the progress condition of the engineering by clicking the engineering query entry, so that the project responsible person can conveniently make corresponding adjustment.
In a possible embodiment, different users can check different business data, for example, a worker in the design department can only check design business data, a worker in the material purchasing department can only check purchasing business data, and a project manager can check design business data, purchasing business data, cost business data and engineering business data at the same time, so that the security of enterprise data information is guaranteed, and data leakage is prevented.
Based on the data reorganization method of the big data platform shown in fig. 1, fig. 2 shows an architecture diagram of the big data platform. The big data platform in the embodiment of the application at least comprises: a data acquisition layer 210, a data processing layer 220, and a data access layer 230. Wherein,
a data acquisition layer 210, configured to acquire target acquisition data through a target acquisition port, where the target acquisition data is design acquisition data when the target acquisition port is a design acquisition port; under the condition that the target acquisition port is a cost acquisition port, the target acquisition data is cost acquisition data; under the condition that the target acquisition port is a purchase acquisition port, the target acquisition data is purchase acquisition data; and under the condition that the target acquisition port is an engineering acquisition port, the target acquisition data is engineering acquisition data.
The data processing layer 220 at least includes a data cleaning unit 221, a data reorganizing unit 222, and a data storage unit 223. The data cleaning unit 221 is configured to clean the target data to obtain cleaning data; a data restructuring unit 222, configured to restructure the cleaning data into target service data and output the target service data through a target output port, specifically, in a case that the target output port is a design output port, the target service data is design service data; when the target output port is a cost output port, the target service data is cost service data; under the condition that the target output port is a purchase acquisition port, the target acquisition data is purchase acquisition data; and under the condition that the target output port is an engineering output port, the target output data is engineering service data. The data storage unit 223 is specifically configured to, when the target output port is a design output port, push the target service data into a first B-tree, where the first B-tree is only used for storing design service data; when the target output port is a cost output port, the target service data is pressed into a second B-tree, wherein the second B-tree is only used for storing the cost service data; when the target output port is a purchase output port, the target business data is pressed into a third B tree, wherein the third B tree is only used for storing purchase business data; and when the target output port is an engineering output port, the target service data is pressed into a fourth B-tree, wherein the fourth B-tree is only used for storing the engineering service data.
The data access layer 230 is configured to provide a user with a query for the service data, and at least includes a data query unit 231 and an identity verification unit 232. The data query unit 231 is configured to receive a query request input by a user, where the query request carries an identity of the user, and the query request is used to query the target service data; an identity authentication unit 232, configured to determine that the user has the right to query the target service data; under the condition that the user is determined to have the authority of inquiring the target service data, acquiring the target service data from the target output port; and rejecting the query request under the condition that the user is determined not to have the authority of querying the target service data.
Referring to fig. 3, fig. 3 is a schematic structural diagram of a data reorganization system of a large data platform in an embodiment provided in the present application. The data reorganization system of the big data platform in the embodiment of the application at least comprises: a data acquisition unit 310, a data cleansing unit 320, and a data reorganization unit 330. Wherein,
a data acquisition unit 310, which is equivalent to the data acquisition layer 210 in the big data platform shown in fig. 2, configured to acquire target acquisition data through a target acquisition port, where the target acquisition data is design acquisition data in a case where the target acquisition port is the design acquisition port; under the condition that the target acquisition port is the cost acquisition port, the target acquisition data is cost acquisition data; under the condition that the target acquisition port is a purchase acquisition port, the target acquisition data is purchase acquisition data; and under the condition that the target acquisition port is the engineering acquisition port, the target acquisition data is engineering acquisition data.
The data cleansing unit 320, which is identical to the data cleansing unit 221 of the large data platform data processing layer 220 shown in fig. 2, is used for cleansing the target data to obtain cleansed data.
A data reorganizing unit 330, which is identical to the data reorganizing unit 222 in the big data platform data processing layer 220 shown in fig. 2, configured to reorganize the cleaning data into target service data, and output the target service data through a target output port, where the target service data is design service data in a case where the target output port is the design output port; under the condition that the target output port is the cost output port, the target service data is cost service data; under the condition that the target output port is a purchase acquisition port, the target acquisition data is purchase acquisition data; and under the condition that the target output port is the engineering output port, the target output data is engineering service data.
In this embodiment of the present application, the data reorganization system of the big data platform further includes a data query unit 340 and an identity verification unit 350. The data querying unit 340 is equivalent to the data querying unit 231 of the big data platform data access layer 230 shown in fig. 2, and is configured to receive a query request input by a user, where the query request carries an identity of the user, and the query request is used to query the target service data; an identity verification unit 350, which is identical to the identity verification unit 232 of the big data platform data access layer 230 shown in fig. 2, configured to determine that a user has an authority to query target service data, and in a case that it is determined that the user has the authority to query the target service data, obtain the target service data from the target output port; and rejecting the query request under the condition that the user is determined not to have the authority of querying the target service data.
In this embodiment of the present application, the data reorganization system of the big data platform further includes a data storage unit 360, where the data storage unit 360 is identical to the data storage unit 223 of the big data platform data processing layer 220 shown in fig. 2, and is specifically configured to, in a case that a target output port is a design output port, push target service data into a first B-tree, where the first B-tree is only used to store design service data; when the target output port is the cost output port, pressing target service data into a second B-tree, wherein the second B-tree is only used for storing the cost service data; under the condition that the target output port is a purchase output port, pressing target service data into a third B-tree, wherein the third B-tree is only used for storing the purchase service data; and when the target output port is the engineering output port, pushing the target service data into a fourth B-tree, wherein the fourth B-tree is only used for storing the engineering service data.
In the embodiment of the application, a design acquisition port, a cost acquisition port, a purchase acquisition port and an engineering acquisition port are isolated from each other; the design output port, the cost output port, the purchase output port and the engineering output port are isolated from each other.
In the embodiment of the present application, the first B-tree, the second B-tree, the third B-tree and the fourth B-tree are isolated from each other.
Referring to fig. 4, the present application provides a schematic structural diagram of a data reorganization system 400 of a big data platform, including: a processor 410, a communication interface 420, and a memory 430, wherein the processor 410, the communication interface 420, and the memory 430 are coupled by a bus 440. Wherein,
the processor 410 may be a Central Processing Unit (CPU), a general purpose processor, a Digital Signal Processor (DSP), an application-specific integrated circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable Logic Device (PLD), a transistor logic device, a hardware component, or any combination thereof. The processor 410 described above may implement or perform the various exemplary methods described in connection with the present disclosure. Specifically, the processor 410 reads the program codes stored in the memory 430, and cooperates with the communication interface 420 to execute some or all of the steps of the method executed by the data collating system 300 of the big data platform in the above embodiments of the present application.
The communication interface 420 may be a wired interface, such as an ethernet interface, a controller area network interface, a Local Interconnect Network (LIN) interface, and a FlexRay interface, or a wireless interface, such as a cellular network interface or a wireless lan interface, for communicating with other modules or devices.
The memory 430 may include volatile memory, such as Random Access Memory (RAM); the memory may also include a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a Hard Disk Drive (HDD), or a solid-state drive (SSD) memory, which may also include a combination of the above types of memory. Memory 430 may store program codes as well as program data. The program code comprises a code of a data acquisition unit, a code of a data cleaning unit, a code of a data recombination unit, a code of a data storage unit and the like. The program data includes: design collected data, cost collected data, purchase collected data, engineering collected data, user identity identification information and the like.
The bus 440 may be a Controller Area Network (CAN) or other internal bus that enables interconnection between various systems or devices within the vehicle. The bus 440 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
Wherein the processor 410 is configured to execute the following steps by calling the program code in the memory 430:
acquiring target acquisition data through a target acquisition port, wherein the target acquisition data comprises design acquisition data, cost acquisition data, purchase acquisition data and engineering acquisition data;
cleaning the target acquisition data to obtain cleaning data;
recombining the cleaning data into target service data, outputting the target service data through a target output port, and storing the target service data into a corresponding B tree, wherein the target service data comprises design service data, cost service data, purchase service data, engineering service data and the like;
receiving a query request input by a user, wherein the query request carries the identity of the user and is used for querying the target service data;
determining that a user has the authority to query target service data;
under the condition that the user is determined to have the authority of inquiring the target service data, acquiring the target service data from a target output port;
and in the case that the user is determined not to have the authority to query the target service data, rejecting the query request.
The data marshalling system of the large data platform described above may contain more or fewer components than shown in fig. 4, or have a different arrangement of components.
In another embodiment of the present application, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program comprising program instructions that when executed by a processor implement:
the computer readable storage medium may be an internal storage unit as described in any of the foregoing embodiments, for example, a hard disk or a memory of a terminal. The computer readable storage medium may also be an external storage device of the terminal, such as a plug-in hard disk, a Smart Memory Card (SMC), a Secure Digital (SD) card, a flash memory card, and the like, provided on the terminal. Further, the computer-readable storage medium may also include both an internal storage unit and an external storage device of the terminal. The computer-readable storage medium is used for storing the computer program and other programs and data required by the terminal. The computer readable storage medium may also be used to temporarily store data that has been output or is to be output.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the terminal and the unit described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the embodiments provided in the present application, it should be understood that the disclosed method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may also be an electric, mechanical or other form of connection.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiments of the present application.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially or partially contributed by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: u disk, removable hard disk, read only memory, random access memory, magnetic or optical disk, etc. for storing program codes.
While the invention has been described with reference to specific embodiments, the scope of the invention is not limited thereto, and those skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the invention. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A data reorganization method of a big data platform is characterized by comprising the following steps:
acquiring target acquisition data through a target acquisition port, wherein the target acquisition data is design acquisition data under the condition that the target acquisition port is a design acquisition port, the target acquisition data is cost acquisition data under the condition that the target acquisition port is a cost acquisition port, the target acquisition data is purchase acquisition data under the condition that the target acquisition port is a purchase acquisition port, and the target acquisition data is engineering acquisition data under the condition that the target acquisition port is an engineering acquisition port;
cleaning the target acquisition data to obtain cleaning data;
and recombining the cleaning data into target service data, and outputting the target service data through a target output port, wherein the target service data is design service data under the condition that the target output port is a design output port, the target service data is cost service data under the condition that the target output port is a cost output port, the target acquisition data is purchase acquisition data under the condition that the target output port is a purchase acquisition port, and the target output data is engineering service data under the condition that the target output port is an engineering output port.
2. The method of claim 1,
the design acquisition port, the cost acquisition port, the purchase acquisition port and the engineering acquisition port are isolated from each other;
the design output port, the cost output port, the purchase output port and the engineering output port are isolated from each other.
3. The method according to claim 1 or 2, characterized in that the method further comprises:
receiving a query request input by a user, wherein the query request carries an identity of the user, and the query request is used for querying the target service data;
determining that the user has the authority to query the target service data;
under the condition that the user is determined to have the authority of inquiring the target service data, acquiring the target service data from the target output port;
and rejecting the query request under the condition that the user is determined not to have the authority of querying the target service data.
4. The method of claim 1, further comprising:
when the target output port is the design output port, pushing the target service data into a first B-tree, wherein the first B-tree is only used for storing the design service data;
when the target output port is the cost output port, the target service data is pressed into a second B-tree, wherein the second B-tree is only used for storing the cost service data;
when the target output port is the purchase output port, the target business data is pressed into a third B tree, wherein the third B tree is only used for storing the purchase business data;
and when the target output port is the engineering output port, the target service data is pressed into a fourth B-tree, wherein the fourth B-tree is only used for storing the engineering service data.
5. The method of claim 4,
the first B-tree, the second B-tree, the third B-tree, and the fourth B-tree are isolated from each other.
6. A data reorganization system of a big data platform is characterized by comprising:
the data acquisition unit is used for acquiring target acquisition data through a target acquisition port, wherein the target acquisition data is design acquisition data under the condition that the target acquisition port is a design acquisition port, the target acquisition data is cost acquisition data under the condition that the target acquisition port is a cost acquisition port, the target acquisition data is purchase acquisition data under the condition that the target acquisition port is a purchase acquisition port, and the target acquisition data is engineering acquisition data under the condition that the target acquisition port is an engineering acquisition port;
the data cleaning unit is used for cleaning the target data to obtain cleaning data;
and the data recombination unit is used for recombining the cleaning data into target service data and outputting the target service data through a target output port, wherein the target service data is design service data under the condition that the target output port is a design output port, the target service data is cost service data under the condition that the target output port is a cost output port, the target acquisition data is purchase acquisition data under the condition that the target output port is a purchase acquisition port, and the target output data is engineering service data under the condition that the target output port is an engineering output port.
7. The system of claim 6,
the design acquisition port, the cost acquisition port, the purchase acquisition port and the engineering acquisition port are isolated from each other;
the design output port, the cost output port, the purchase output port and the engineering output port are isolated from each other.
8. The system according to claim 6 or 7, characterized in that it further comprises:
a data query unit, configured to receive a query request input by a user, where the query request carries an identity of the user, and the query request is used to query the target service data;
the identity authentication unit is used for determining that the user has the authority to inquire the target service data; under the condition that the user is determined to have the authority of inquiring the target service data, acquiring the target service data from the target output port; and rejecting the query request under the condition that the user is determined not to have the authority of querying the target service data.
9. The system according to claim 6, characterized in that the system further comprises a data storage unit, in particular for,
when the target output port is the design output port, pushing the target service data into a first B-tree, wherein the first B-tree is only used for storing the design service data;
when the target output port is the cost output port, the target service data is pressed into a second B-tree, wherein the second B-tree is only used for storing the cost service data;
when the target output port is the purchase output port, the target business data is pressed into a third B tree, wherein the third B tree is only used for storing the purchase business data;
and when the target output port is the engineering output port, the target service data is pressed into a fourth B-tree, wherein the fourth B-tree is only used for storing the engineering service data.
10. The system of claim 9,
the first B-tree, the second B-tree, the third B-tree, and the fourth B-tree are isolated from each other.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910948061.3A CN110750568A (en) | 2019-09-29 | 2019-09-29 | Data recombination method and system for big data platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910948061.3A CN110750568A (en) | 2019-09-29 | 2019-09-29 | Data recombination method and system for big data platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110750568A true CN110750568A (en) | 2020-02-04 |
Family
ID=69277643
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910948061.3A Pending CN110750568A (en) | 2019-09-29 | 2019-09-29 | Data recombination method and system for big data platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110750568A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180268015A1 (en) * | 2015-09-02 | 2018-09-20 | Sasha Sugaberry | Method and apparatus for locating errors in documents via database queries, similarity-based information retrieval and modeling the errors for error resolution |
| CN108665206A (en) * | 2018-04-18 | 2018-10-16 | 山西同城商务信息股份有限公司 | A kind of wisdom logistics cloud platform system |
| CN108737325A (en) * | 2017-04-13 | 2018-11-02 | 华为技术有限公司 | A kind of multi-tenant data partition method, apparatus and system |
| CN109587142A (en) * | 2018-12-10 | 2019-04-05 | 北京华虹集成电路设计有限责任公司 | A kind of the data safety AM access module and equipment of service-oriented stream |
| CN110008226A (en) * | 2019-04-03 | 2019-07-12 | 平安信托有限责任公司 | Construction method, device, computer equipment and the storage medium of data model |
-
2019
- 2019-09-29 CN CN201910948061.3A patent/CN110750568A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180268015A1 (en) * | 2015-09-02 | 2018-09-20 | Sasha Sugaberry | Method and apparatus for locating errors in documents via database queries, similarity-based information retrieval and modeling the errors for error resolution |
| CN108737325A (en) * | 2017-04-13 | 2018-11-02 | 华为技术有限公司 | A kind of multi-tenant data partition method, apparatus and system |
| CN108665206A (en) * | 2018-04-18 | 2018-10-16 | 山西同城商务信息股份有限公司 | A kind of wisdom logistics cloud platform system |
| CN109587142A (en) * | 2018-12-10 | 2019-04-05 | 北京华虹集成电路设计有限责任公司 | A kind of the data safety AM access module and equipment of service-oriented stream |
| CN110008226A (en) * | 2019-04-03 | 2019-07-12 | 平安信托有限责任公司 | Construction method, device, computer equipment and the storage medium of data model |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10948526B2 (en) | Non-parametric statistical behavioral identification ecosystem for electricity fraud detection | |
| CN111782943B (en) | Information recommendation method, device, equipment and medium based on historical data record | |
| CN111046237B (en) | User behavior data processing method and device, electronic equipment and readable medium | |
| WO2020037917A1 (en) | User behavior data recommendation method, server and computer readable medium | |
| CN109816321A (en) | A kind of service management, device, equipment and computer readable storage medium | |
| CN113836131B (en) | Big data cleaning method and device, computer equipment and storage medium | |
| WO2021175021A1 (en) | Product push method and apparatus, computer device, and storage medium | |
| US9275125B1 (en) | System for organizing data from a plurality of users to create individual user profiles | |
| CN110766486A (en) | Method and device for determining item category | |
| CN113010542B (en) | Service data processing method, device, computer equipment and storage medium | |
| CN111752944A (en) | Data allocation method and device, computer equipment and storage medium | |
| CN113392150A (en) | Data table display method, device, equipment and medium based on service domain | |
| CN111339098A (en) | Authority management method, data query method and device | |
| CN114556317A (en) | Document tracking through version hash link graphs | |
| CN112328658A (en) | User profile data processing method, device, equipment and storage medium | |
| KR101614890B1 (en) | Method of creating multi tenancy history, server performing the same and storage media storing the same | |
| CN114240344A (en) | Enterprise personnel data processing method and device, computer equipment and storage medium | |
| CN117390011A (en) | Report data processing method, device, computer equipment and storage medium | |
| CN110750568A (en) | Data recombination method and system for big data platform | |
| CN115859273A (en) | Method, device and equipment for detecting abnormal access of database and storage medium | |
| CN114491259A (en) | Data recommendation method and device, terminal and storage medium | |
| CN113128595A (en) | Method, apparatus, device and computer readable medium for integrating customer information | |
| CN111143328A (en) | Agile business intelligent data construction method, system, equipment and storage medium | |
| CN108241750B (en) | Method and device for optimizing connecting lines in network diagram, storage medium and terminal | |
| CN109828983A (en) | PG data base processing method, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200204 |
|
| RJ01 | Rejection of invention patent application after publication |