CN113076305A - Data processing method and device, electronic equipment and storage medium - Google Patents
Data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN113076305A CN113076305A CN202110426542.5A CN202110426542A CN113076305A CN 113076305 A CN113076305 A CN 113076305A CN 202110426542 A CN202110426542 A CN 202110426542A CN 113076305 A CN113076305 A CN 113076305A
- Authority
- CN
- China
- Prior art keywords
- data
- service source
- source data
- target service
- metadata
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 22
- 238000007405 data analysis Methods 0.000 claims abstract description 48
- 238000000034 method Methods 0.000 claims abstract description 31
- 238000012545 processing Methods 0.000 claims abstract description 12
- 238000004458 analytical method Methods 0.000 claims description 16
- 239000008280 blood Substances 0.000 claims description 12
- 210000004369 blood Anatomy 0.000 claims description 12
- 238000005516 engineering process Methods 0.000 claims description 12
- 238000010276 construction Methods 0.000 claims description 9
- 238000005259 measurement Methods 0.000 claims description 3
- 230000004044 response Effects 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 15
- 230000006870 function Effects 0.000 description 9
- 238000004590 computer program Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000009471 action Effects 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000007418 data mining Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000004888 barrier function Effects 0.000 description 1
- 210000004204 blood vessel Anatomy 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/211—Schema design and management
- G06F16/212—Schema design and management with details for data modelling support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
- G06Q50/06—Electricity, gas or water supply
Abstract
The invention relates to a data processing method, a data processing device, an electronic device and a storage medium, wherein the method comprises the following steps: collecting service source data associated with a preset data analysis scene in a set mode to obtain target service source data; constructing a data identifier based on the target service source data and the preset data analysis scene; and performing associated storage on the data identifier and the target service source data, wherein the data identifier is used for a user to acquire the target service source data. By the technical scheme of the embodiment of the invention, the purpose of performing statistical storage on the service source data according to the set dimension is realized, so that the service source data is convenient for a user to search and use, and the difficulty of using the data by the user is reduced.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method and apparatus, an electronic device, and a storage medium.
Background
With the further deepening of the comprehensive digital transformation work of the power grid company, the demand for providing unified data service capability for data applications such as large-screen monitoring, thematic analysis, ad hoc reporting, detailed lists, data mining and self-service inquiry is increasing day by day.
However, the power grid data has the characteristics of large data volume, diversified service systems, diversified data types, rich application scenes and the like, and with the urgent promotion of the value demand of the service departments for mining the data assets, the rapid and accurate acquisition of the target data assets becomes a technical difficulty which restricts the data analysis personnel of each service department from performing self-service analysis and data mining, and is also a barrier which restricts the data driving service, process and operation decision and drives the state change of the production, operation and service of enterprises.
Therefore, how to accurately and clearly provide the data assets of each service system to the data analyst is a key to efficiently utilize the data assets.
Disclosure of Invention
In order to solve the above technical problems or at least partially solve the above technical problems, the present invention provides a data processing method, an apparatus, an electronic device, and a storage medium, which achieve the purpose of performing statistical storage on service source data according to a set dimension, thereby facilitating a user to search and use the service source data, and reducing the difficulty of the user in using the service source data.
In a first aspect, an embodiment of the present invention provides a data processing method, where the method includes:
collecting service source data associated with a preset data analysis scene in a set mode to obtain target service source data;
constructing a data identifier based on the target service source data and the preset data analysis scene;
and performing associated storage on the data identifier and the target service source data, wherein the data identifier is used for a user to acquire the target service source data.
In a second aspect, an embodiment of the present invention further provides a data processing apparatus, where the apparatus includes:
the collection module is used for collecting the service source data associated with the preset data analysis scene in a set mode to obtain target service source data;
the construction module is used for constructing a data identifier based on the target service source data and the preset data analysis scene;
and the storage module is used for storing the data identifier and the target service source data in a correlation manner, wherein the data identifier is used for the user to acquire the target service source data.
In a third aspect, an embodiment of the present invention further provides an apparatus, where the apparatus includes:
one or more processors;
a storage device for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method according to any one of the embodiments of the present invention.
In a fourth aspect, the embodiments of the present invention further provide a storage medium containing computer-executable instructions, which when executed by a computer processor, are used for executing the data processing method according to any one of the embodiments of the present invention.
Compared with the prior art, the technical scheme provided by the embodiment of the invention has the following advantages:
acquiring target service source data by collecting service source data associated with a preset data analysis scene in a set mode; constructing a data identifier based on target service source data and a preset data analysis scene; the data identification and the target service source data are stored in an associated mode, wherein the data identification is used for enabling a user to obtain the target service source data, the purpose of performing statistical storage on the service source data according to set dimensions (such as a data analysis scene) is achieved, the user can conveniently search and use the data, and the difficulty of using the data by the user is reduced.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating a relationship between blood vessels according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order that the above objects, features and advantages of the present invention may be more clearly understood, a solution of the present invention will be further described below. It should be noted that the embodiments of the present invention and features of the embodiments may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways than those described herein; it is to be understood that the embodiments described in this specification are only some embodiments of the invention, and not all embodiments.
Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present invention are shown in the drawings, it should be understood that the present invention may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and the embodiments of the present invention are illustrative only and are not intended to limit the scope of the present invention.
It should be understood that the various steps recited in the method embodiments of the present invention may be performed in a different order and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the invention is not limited in this respect.
The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.
It should be noted that the terms "first", "second", and the like in the present invention are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.
It is noted that references to "a", "an", and "the" modifications in the present invention are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that reference to "one or more" unless the context clearly dictates otherwise.
Fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present invention. The method can be suitable for processing the big data assets, so that the content and meaning of each data asset are visualized, a user can easily and intuitively know the content of the data assets, when the user needs to use the data, the user can easily select the data which the user wants from the visualized data assets, and the complexity of the user in using the data is reduced. The data processing method may be performed by a data processing apparatus, which may be implemented in the form of software and/or hardware.
As shown in fig. 1, the data processing method provided in this embodiment includes the following steps:
and 110, collecting the service source data associated with the preset data analysis scene in a set mode to obtain target service source data.
The preset data analysis scenario may be, for example, a power outage analysis scenario, a user satisfaction analysis scenario, or other specific scenarios. In one embodiment, the service source data associated with the power outage analysis scenario includes, for example, the time of the power outage, the power outage area, or the data (e.g., voltage, current, etc.) of the various lines within the power outage area. In another embodiment, the service source data associated with the user satisfaction analysis scenario includes, for example, an average outage duration over a period of time or an average number of blackouts over a period of time.
In one embodiment, collecting service source data associated with a preset data analysis scenario in a set manner to obtain target service source data includes: determining a data field according to a preset data analysis scene; and obtaining target business source data through database SQL search or data cube search based on the data fields. Specifically, one or more data sources are associated based on the database SQL or the data cube, a plurality of fields that may be used by a user in a preset data analysis scenario are selected, operation or index derivation is performed (for example, an average value is calculated, and the derived fields are source data corresponding to the "average value"), a data set that is used when the user performs data analysis in the preset data analysis scenario is generated or a data table is generated and stored in the database, the generated data set or data table may be called a data asset card, and the essence of data in the data asset card is target service source data that is used when the user performs data analysis in the preset data analysis scenario.
And 120, constructing a data identifier based on the target service source data and the preset data analysis scene.
The data identification can be information used for representing data content or data meaning, and the data identification can enable a user to easily and intuitively know the content and meaning of the data, so that the difficulty of the user in using the data is reduced.
In one embodiment, the constructing the data identifier based on the target service source data and the preset data analysis scenario includes: respectively constructing primary metadata and/or secondary metadata based on the target service source data and a preset data analysis scene; wherein the data identification packet comprises primary metadata and/or secondary metadata.
In one embodiment, the constructing of the primary metadata based on the target service source data and the preset data analysis scenario includes at least one of:
constructing a service domain identifier according to a service domain to which a preset data analysis scene belongs;
constructing a business object identifier according to a business object aimed at by a preset data analysis scene;
constructing a time identifier according to the acquisition time of the target service source data;
correspondingly, the primary metadata includes at least one of a service domain identifier, a service object identifier and a time identifier. Illustratively, the service domain identification includes at least one of: a marketing domain, a supplies domain, or a people domain; the business object identification comprises at least one of the following: a power outage analysis or a user satisfaction analysis. The primary metadata may also include creation time, release time, update time of the data asset card (i.e., data set or data table), description information of the data asset card (e.g., "average blackout time per month in a certain area and average blackout time after manual review, which may be used as primary reference data for customer complaints, blackout analysis, etc.), creator information, the system comprises manager information, a data standard responsibility department, a data quality responsibility department, a primary data theme and a secondary data theme (for example, the primary data theme and the secondary data theme are divided according to the classification of departments and business items), key indexes (business-related key analysis indexes, for example, in an automatic meter reading analysis business, the key analysis indexes are automatic meter reading rates), and other data asset cards related to the key indexes (which can be other data asset cards recommended by using an association rule algorithm and simultaneously viewed by a user).
In one embodiment, the data processing method further comprises: the data identification is stored in the form of a data asset card. For example, referring to the schematic diagram of a data asset card shown in table 1 below, the primary metadata includes a data asset card name "automatic meter reading condition", a service domain identifier "marketing domain", a service object "meter reading management", a creation time "2020-1-1", a release time "2020-2-3", an update time "2020-2-20", description information "automatic meter reading condition of the card summarizes automatic meter reading rate, and compares the difference between important indexes of automatic meter reading and manual meter reading", a creator "AAA", a manager "BBB", a data standard responsibility department "market reward part", a data quality department "information center", a primary data topic "market reward part", a secondary data topic "business management", a key index "automatic meter reading rate", a related data card 1 "meter reading rate statistics"), The relevant data asset card 2 "issue rate statistics", the relevant data asset card 3 "meter reading section maintenance", the number of fields "10", the number of records "1000", and the size "15K".
In one embodiment, constructing secondary metadata based on the target service source data and a preset data analysis scenario includes at least one of:
determining metadata of each field of the target service source data; the metadata of each field is determined as secondary metadata. Illustratively, the metadata for each field includes at least one of: business metadata, first technology metadata (data partitioned from a database perspective), and second technology metadata (data partitioned from a data analysis perspective). The service metadata includes at least one of: field names, business rules and codes; the first technical metadata includes at least one of: data type, measurement type, data unit, value range, data length, data precision, whether the key is a main key, whether the key can be empty and whether the key is a foreign key; the second technical metadata includes at least one of: valid records, nominal values, nominal number of values, maximum, minimum, mean, and standard deviation.
Table 1: schematic diagram of data asset card
And step 130, performing associated storage on the data identifier and the target service source data, wherein the data identifier is used for a user to acquire the target service source data.
In one embodiment, the associating and storing the data identifier and the target service source data comprises: and taking the table for storing the target service source data as a main table, and respectively associating a first data table for storing primary metadata and a second data table for storing secondary metadata. For example, the table for storing the target service source data is taken as a main table, the first data table is connected to the left, and then the second data table is connected to the left, so that the comprehensive semantization and visualization of the data asset card can be realized
According to the technical scheme of the embodiment, the data related to the specific data analysis scene is arranged together to form the data asset card, the data asset card and the target business source data contained in the data asset card are semantically processed, for example, the name, the responsibility department, the data description, the sharing range, the related business items, the number of fields, the field name, the business meaning of each field, the type, the length and the precision of the data of the target business source data included in the data asset card are displayed, so that the content of the data asset is clear, a user can simply, clearly and comprehensively know the overall view of the data asset card, the complexity of the user in using the data is reduced, and the efficiency of the user in using the data is improved.
In one embodiment, the data processing method further comprises:
and establishing a blood relationship graph among data tables where the target service source data are located based on the common fields, and displaying the blood relationship graph. By displaying the association relationship among all the related tables under the asset card in the form of a graph, a user can be made to know the content of the data asset card and the relationship among the data tables very clearly.
For example, refer to a schematic diagram of a blood relationship diagram as shown in fig. 2, where the data table in which the target service source data is located includes: a supplier data table (the data table includes fields having a supplier number, a supplier company name, a supplier address, etc.), a customer data table (the data table includes fields having a customer number, a company name, etc.), a product catalog data table (the data table includes fields having a product catalog number, a product catalog name, a product catalog description, etc.), a product data table (the data table includes fields having a product number, a product name, a supplier number, a product catalog number, etc.), an order data table (the data table includes fields having an order number, a customer number, an employee number, etc.), an order detail data table (the data table includes fields having an order number, a product number, etc.), and an employee data table (the data table includes fields having an age, an employee number, etc.). Because the supplier data table and the product data table both include the field "supplier number", that is, "supplier number" is a common field of the supplier data table and the product data table, there is an association between the supplier data table and the product data table, and a blood relationship diagram between the supplier data table and the product data table is established. Similarly, because the product catalog data table and the product data table both include the field "product catalog number", that is, "product catalog number" is a common field of the product catalog data table and the product data table, there is an association between the product catalog data table and the product data table, and a blood relationship chart between the product catalog data table and the product data table is established. Because the field "product number" is included in both the order detail data table and the product data table, that is, "product number" is a common field of the order detail data table and the product data table, there is an association between the order detail data table and the product data table, and a blood relationship diagram between the order detail data table and the product data table is established. Because the order detail data table and the order data table both include the field "order number", that is, "order number" is a common field of the order detail data table and the order data table, there is an association between the order detail data table and the order data table, and a blood relationship graph between the order detail data table and the order data table is established. Because the fields "customer number" are included in both the order data table and the customer data table, i.e., "customer number" is a common field of the order data table and the customer data table, there is an association between the order data table and the customer data table, and a blood relationship graph between the order data table and the customer data table is established. Because the fields "employee number" are included in both the order data table and the employee data table, i.e., "employee number" is a common field of the order data table and the employee data table, there is an association between the order data table and the employee data table, and a blooding relationship graph between the order data table and the employee data table is established.
The method comprises the steps that a primary metadata, a secondary metadata and a consanguineous relation graph of a data asset card are created, and the content, the purpose, the management characteristics, the business meanings of the data asset card, the business meanings of fields and the data characteristics are comprehensively displayed; the data asset card is associated with the primary metadata, the primary secondary metadata and the field metadata and is displayed to a user, so that the overall appearance of the data asset card is comprehensively displayed, and the user can know the meaning and the content of the source data in the data asset card very clearly.
In one embodiment, the data processing method further comprises:
determining a matching first data table or second data table based on the received search keyword in response to a search instruction of a user; determining a matched main table based on the matched first data table or the matched second data table; and providing the target service source data stored in the main table to the user.
In one embodiment, the data processing method further comprises: and recommending the service source data to the user based on the historical browsing record of the user and/or the current browsing information.
The data asset card generated by the technical scheme has stronger service pertinence and can better meet the requirements of users, and the service meaning and technical characteristics of the data are displayed in an all-around manner, so that the threshold of using the data by the users is favorably reduced.
Fig. 3 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention, where the apparatus includes: a collection module 310, a construction module 320, and a storage module 330.
The collection module 310 is configured to collect service source data associated with a preset data analysis scenario in a set manner, and obtain target service source data; a construction module 320, configured to analyze a scene construction data identifier based on the target service source data and the preset data; the storage module 330 is configured to perform association storage on the data identifier and the target service source data, where the data identifier is used for a user to obtain the target service source data.
On the basis of the above technical solution, the collecting module 310 includes:
the first determining unit is used for determining a data field according to the preset data analysis scene; and the searching unit is used for obtaining the target service source data through database SQL search or data cube search based on the data field.
On the basis of the above technical solutions, the building module 320 includes:
the construction unit is used for respectively constructing primary metadata and/or secondary metadata based on the target service source data and the preset data analysis scene; wherein the data identification comprises the primary metadata and/or the secondary metadata.
On the basis of the above technical solutions, the construction unit is specifically configured to at least one of:
constructing a service domain identifier according to the service domain to which the preset data analysis scene belongs;
constructing a business object identifier according to the business object aimed at by the preset data analysis scene;
constructing a time identifier according to the acquisition time of the target service source data;
correspondingly, the primary metadata includes at least one of the service domain identifier, the service object identifier and the time identifier.
On the basis of the above technical solutions, the service domain identifier includes at least one of the following: a marketing domain, a supplies domain, or a people domain;
the business object identification comprises at least one of the following: a power outage analysis or a user satisfaction analysis.
On the basis of the above technical solutions, the construction unit is specifically configured to:
determining metadata of each field of the target service source data;
determining metadata for each field as the secondary metadata.
On the basis of the above technical solutions, the metadata of each field includes at least one of the following: business metadata, first technology metadata, and second technology metadata.
On the basis of the above technical solutions, the service metadata includes at least one of the following: field names, business rules and codes;
the first technology metadata includes at least one of: data type, measurement type, data unit, value range, data length, data precision, whether the key is a main key, whether the key can be empty and whether the key is a foreign key;
the second technology metadata includes at least one of: valid records, nominal values, nominal number of values, maximum, minimum, mean, and standard deviation.
On the basis of the above technical solutions, the apparatus further includes: and the establishing module is used for establishing a blood relationship graph among data tables where the target service source data are located and displaying the blood relationship graph.
On the basis of the above technical solutions, the storage module 330 is further configured to store the data identifier in the form of a data asset card.
On the basis of the above technical solutions, the storage module 330 is further specifically configured to: and respectively associating a first data table for storing the primary metadata and a second data table for storing the secondary metadata by using the table for storing the target service source data as a main table.
On the basis of the above technical solutions, the apparatus further includes:
a first determining module, configured to determine, in response to a search instruction of a user, the first data table or the second data table that matches based on a received search keyword;
a second determining module for determining the matched primary table based on the matched first data table or the matched second data table;
and the return module is used for providing the target service source data stored in the main table to the user.
On the basis of the above technical solutions, the apparatus further includes:
and the recommending module is used for recommending the service source data to the user based on the historical browsing record of the user and/or the current browsing information.
According to the technical scheme of the embodiment of the invention, the data related to the specific data analysis scene is arranged together according to the specific data analysis scene to form the data asset card, the data asset card and the target service source data contained in the data asset card are semantically processed, such as the name, responsibility department, data description, sharing range, related service items, the number of fields, the field name, the service meaning of each field, the type, length and precision of the data and the like of the target service source data contained in the data asset card are displayed, so that the content of the data asset is clear, a user can simply, clearly and comprehensively know the full view of the data asset card, the complexity of the data used by the user is reduced, and the efficiency of the user for using the data is improved.
The data processing device provided by the embodiment of the invention can execute the data processing method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
It should be noted that, the units and modules included in the apparatus are merely divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the embodiment of the invention.
Referring now to fig. 4, a schematic diagram of an electronic device (e.g., the terminal device or server of fig. 4) 400 suitable for implementing embodiments of the present invention is shown. The terminal device in the embodiments of the present invention may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 4, electronic device 400 may include a processing device (e.g., central processing unit, graphics processor, etc.) 401 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)402 or a program loaded from a storage device 408 into a Random Access Memory (RAM) 403. In the RAM403, various programs and data necessary for the operation of the electronic apparatus 400 are also stored. The processing device 401, the ROM 402, and the RAM403 are connected to each other via a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.
Generally, the following devices may be connected to the I/O interface 405: input devices 406 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 407 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 408 including, for example, tape, hard disk, etc.; and a communication device 409. The communication means 409 may allow the electronic device 400 to communicate wirelessly or by wire with other devices to exchange data. While fig. 4 illustrates an electronic device 400 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.
In particular, according to an embodiment of the present invention, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, an embodiment of the invention includes a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication device 409, or from the storage device 408, or from the ROM 402. The computer program performs the above-described functions defined in the methods of embodiments of the invention when executed by the processing apparatus 401.
The terminal provided by the embodiment of the present invention and the data processing method provided by the above embodiment belong to the same inventive concept, and technical details that are not described in detail in the embodiment of the present invention may be referred to the above embodiment, and the embodiment of the present invention has the same beneficial effects as the above embodiment.
The embodiment of the invention also provides a computer storage medium, on which a computer program is stored, and the program is executed by a processor to realize the data processing method provided by the embodiment.
It should be noted that the computer readable medium of the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to:
collecting service source data associated with a preset data analysis scene in a set mode to obtain target service source data;
constructing a data identifier based on the target service source data and the preset data analysis scene;
and performing associated storage on the data identifier and the target service source data, wherein the data identifier is used for a user to acquire the target service source data.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including but not limited to an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present invention may be implemented by software or hardware. Where the name of a cell does not in some cases constitute a limitation on the cell itself, for example, an editable content display cell may also be described as an "editing cell".
The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.
In the context of the present invention, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The foregoing description is only exemplary of the preferred embodiments of the invention and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents is encompassed without departing from the spirit of the disclosure. For example, the above features and (but not limited to) features having similar functions disclosed in the present invention are mutually replaced to form the technical solution.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present invention, which enable those skilled in the art to understand or practice the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (16)
1. A data processing method, comprising:
collecting service source data associated with a preset data analysis scene in a set mode to obtain target service source data;
constructing a data identifier based on the target service source data and the preset data analysis scene;
and performing associated storage on the data identifier and the target service source data, wherein the data identifier is used for a user to acquire the target service source data.
2. The method according to claim 1, wherein the collecting the service source data associated with the preset data analysis scenario in a set manner to obtain the target service source data comprises:
determining a data field according to the preset data analysis scene;
and obtaining the target business source data through database SQL search or data cube search based on the data field.
3. The method of claim 1, wherein the analyzing the scene construction data identifier based on the target service source data and the preset data comprises:
respectively constructing primary metadata and/or secondary metadata based on the target service source data and the preset data analysis scene;
wherein the data identification comprises the primary metadata and/or the secondary metadata.
4. The method of claim 3, wherein the constructing primary metadata based on the target traffic source data and the preset data analysis scenario comprises at least one of:
constructing a service domain identifier according to the service domain to which the preset data analysis scene belongs;
constructing a business object identifier according to the business object aimed at by the preset data analysis scene;
constructing a time identifier according to the acquisition time of the target service source data;
correspondingly, the primary metadata includes at least one of the service domain identifier, the service object identifier and the time identifier.
5. The method of claim 4, wherein the service domain identifier comprises at least one of: a marketing domain, a supplies domain, or a people domain;
the business object identification comprises at least one of the following: a power outage analysis or a user satisfaction analysis.
6. The method of claim 3, wherein the constructing secondary metadata based on the target service source data and the preset data analysis scenario comprises:
determining metadata of each field of the target service source data;
determining metadata for each field as the secondary metadata.
7. The method of claim 6, wherein the metadata for each field comprises at least one of: business metadata, first technology metadata, and second technology metadata.
8. The method of claim 7, wherein the service metadata comprises at least one of: field names, business rules and codes;
the first technology metadata includes at least one of: data type, measurement type, data unit, value range, data length, data precision, whether the key is a main key, whether the key can be empty and whether the key is a foreign key;
the second technology metadata includes at least one of: valid records, nominal values, nominal number of values, maximum, minimum, mean, and standard deviation.
9. The method according to any one of claims 1-8, further comprising:
and establishing a blood relationship graph among data tables in which the target service source data are located based on the common fields, and displaying the blood relationship graph.
10. The method according to any one of claims 1-8, further comprising:
and storing the data identification in the form of a data asset card.
11. The method according to any one of claims 3-8, wherein the storing the data identifier in association with the target service source data comprises:
and respectively associating a first data table for storing the primary metadata and a second data table for storing the secondary metadata by using the table for storing the target service source data as a main table.
12. The method of claim 11, further comprising:
determining the first data table or the second data table matched based on the received search key word in response to a search instruction of a user;
determining the matched primary table based on the matched first data table or the matched second data table;
and providing the target service source data stored in the main table to a user.
13. The method of claim 12, further comprising:
and recommending the service source data to the user based on the historical browsing record of the user and/or the current browsing information.
14. A data processing apparatus, comprising:
the collection module is used for collecting the service source data associated with the preset data analysis scene in a set mode to obtain target service source data;
the construction module is used for constructing a data identifier based on the target service source data and the preset data analysis scene;
and the storage module is used for storing the data identifier and the target service source data in a correlation manner, wherein the data identifier is used for the user to acquire the target service source data.
15. An electronic device, characterized in that the electronic device comprises:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement a data processing method as claimed in any one of claims 1-13.
16. A storage medium containing computer-executable instructions for performing the data processing method of any one of claims 1-13 when executed by a computer processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110426542.5A CN113076305A (en) | 2021-04-20 | 2021-04-20 | Data processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110426542.5A CN113076305A (en) | 2021-04-20 | 2021-04-20 | Data processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113076305A true CN113076305A (en) | 2021-07-06 |
Family
ID=76618194
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110426542.5A Pending CN113076305A (en) | 2021-04-20 | 2021-04-20 | Data processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113076305A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113986932A (en) * | 2021-12-28 | 2022-01-28 | 恒生电子股份有限公司 | Data processing method and device, computer equipment and readable storage medium |
| CN115328569A (en) * | 2022-07-18 | 2022-11-11 | 易保网络技术(上海)有限公司 | Method, system, electronic device and computer readable storage medium for processing data conflict |
| CN117151496A (en) * | 2023-11-01 | 2023-12-01 | 广东电网有限责任公司 | Enterprise architecture alignment method, device, equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110347343A (en) * | 2019-07-16 | 2019-10-18 | 珠海格力电器股份有限公司 | Data managing method and device |
| CN110765321A (en) * | 2019-10-28 | 2020-02-07 | 北京明略软件系统有限公司 | Data storage path generation method and device and readable storage medium |
| CN110909010A (en) * | 2019-11-25 | 2020-03-24 | 杭州晨鹰军泰科技有限公司 | Data intelligent analysis configuration management method, device, equipment and storage medium |
| CN112463954A (en) * | 2020-11-11 | 2021-03-09 | 远光软件股份有限公司 | Visual multidimensional data display system and method based on semantic recognition |
| CN112559524A (en) * | 2020-12-14 | 2021-03-26 | 中国建设银行股份有限公司 | Index database establishing method and device and storage medium |
-
2021
- 2021-04-20 CN CN202110426542.5A patent/CN113076305A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110347343A (en) * | 2019-07-16 | 2019-10-18 | 珠海格力电器股份有限公司 | Data managing method and device |
| CN110765321A (en) * | 2019-10-28 | 2020-02-07 | 北京明略软件系统有限公司 | Data storage path generation method and device and readable storage medium |
| CN110909010A (en) * | 2019-11-25 | 2020-03-24 | 杭州晨鹰军泰科技有限公司 | Data intelligent analysis configuration management method, device, equipment and storage medium |
| CN112463954A (en) * | 2020-11-11 | 2021-03-09 | 远光软件股份有限公司 | Visual multidimensional data display system and method based on semantic recognition |
| CN112559524A (en) * | 2020-12-14 | 2021-03-26 | 中国建设银行股份有限公司 | Index database establishing method and device and storage medium |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113986932A (en) * | 2021-12-28 | 2022-01-28 | 恒生电子股份有限公司 | Data processing method and device, computer equipment and readable storage medium |
| CN115328569A (en) * | 2022-07-18 | 2022-11-11 | 易保网络技术(上海)有限公司 | Method, system, electronic device and computer readable storage medium for processing data conflict |
| CN115328569B (en) * | 2022-07-18 | 2024-03-15 | 易保网络技术(上海)有限公司 | Method, system, electronic device and computer readable storage medium for processing data conflict |
| CN117151496A (en) * | 2023-11-01 | 2023-12-01 | 广东电网有限责任公司 | Enterprise architecture alignment method, device, equipment and storage medium |
| CN117151496B (en) * | 2023-11-01 | 2024-03-15 | 广东电网有限责任公司 | Enterprise architecture alignment method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113076305A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN107679211B (en) | Method and device for pushing information | |
| US11310324B2 (en) | System and method for determining relevance of social content | |
| US20120331391A1 (en) | User interface for managing questions and answers across multiple social media data sources | |
| US9043413B2 (en) | System and method for extracting, collecting, enriching and ranking of email objects | |
| US20190034816A1 (en) | Methods and system for associating locations with annotations | |
| CN104462113A (en) | Search method and device and electronic equipment | |
| CN111125344B (en) | Related word recommendation method and device | |
| CN112528595A (en) | Document processing method and device and electronic equipment | |
| CN105430071A (en) | Method and device for pushing information | |
| CN111966866A (en) | Data asset management method and device | |
| CN114092056A (en) | Project management method, device, electronic equipment, storage medium and product | |
| KR20140000408A (en) | System and method for searching professionals | |
| CN110852720A (en) | Document processing method, device, equipment and storage medium | |
| CN112100216A (en) | Creative keyword processing method and device | |
| CN104240107A (en) | Community data screening system and method thereof | |
| CN110633411A (en) | Method and device for screening house resources, electronic equipment and storage medium | |
| Kontogianni et al. | Smart tourism through social network user modeling: a literature review | |
| KR20200086057A (en) | A system and method for bill monitoring | |
| CN114202389A (en) | User evaluation control method and device, electronic equipment and storage medium | |
| CN114741595A (en) | Information pushing method and device | |
| CN111382365B (en) | Method and device for outputting information | |
| CN115168652A (en) | Visual display method, device, medium and equipment for information assets | |
| CN111783440A (en) | Intention recognition method and device, readable medium and electronic equipment | |
| CN111898027A (en) | Method, device, electronic equipment and computer readable medium for determining feature dimension |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |