CN111831636A - Data processing method, device, computer system and readable storage medium - Google Patents
Data processing method, device, computer system and readable storage medium Download PDFInfo
- Publication number
- CN111831636A CN111831636A CN202010740433.6A CN202010740433A CN111831636A CN 111831636 A CN111831636 A CN 111831636A CN 202010740433 A CN202010740433 A CN 202010740433A CN 111831636 A CN111831636 A CN 111831636A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- analysis
- query request
- basic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/332—Query formulation
- G06F16/3325—Reformulation based on results of preceding query
- G06F16/3326—Reformulation based on results of preceding query using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages
- G06F16/3328—Reformulation based on results of preceding query using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages using graphical result space presentation or visualisation
Abstract
The invention discloses a data processing method, a device, a computer system and a readable storage medium, which relate to the technical field of artificial intelligence data processing and comprise the following steps: acquiring data in real time, and preprocessing the acquired data to acquire basic data; determining at least one target object according to the basic data, analyzing the basic data based on the target object, and generating a data set with keyword identification; receiving a query request sent by a client, and matching a data set corresponding to the query request as target data based on the query request; generating a target report based on the target data, and sending the target report to a client; by generating the data sets under all dimensions in advance, the problems that the existing report can not display data of a certain type or a certain dimension, a user needs to search for relevant data independently and then perform manual analysis, the query speed is low, the operation is complex, and the working efficiency is low are solved.
Description
Technical Field
The present invention relates to the field of artificial intelligence data processing technologies, and in particular, to a data processing method, an apparatus, a computer system, and a readable storage medium.
Background
With the continuous development of information technology, particularly, social networks, electronic commerce and mobile communication bring the human society into the era of big data, the big data provides an application space for the large-scale and distributed computing power of cloud computing, and the problem that a traditional computer cannot solve is solved. In the big data era, data analysis refers to the analysis of data with huge scale, a data analysis component becomes a solid background for enterprise decision making, more and more enterprises depend on the big data analysis to realize value landing, and intelligent operation is realized in the aspects of production, market, internal management and the like.
The inventor found in the research that most of the data analysis results are visually presented in the form of reports, and a user can check the display results of all data through the reports, but when the user needs to specifically check a certain type of data or a certain dimension of data analysis results, the user needs to independently search related data and then manually analyze the data, so that the query speed is low, the operation is complicated, and the working efficiency is low.
Disclosure of Invention
The invention aims to provide a data processing method, a data processing device, a computer system and a readable storage medium, which are used for solving the problems that in the prior art, the final analysis result is visually presented in the forms of common reports and the like, but the final analysis result cannot be visually presented in a certain type or a certain dimension of data, a user needs to independently search and search related data and then manually analyze the related data, the query speed is low, the operation is complicated, and the working efficiency is low.
In order to achieve the above object, the present invention provides a data processing method, including:
acquiring data in real time, and preprocessing the acquired data to acquire basic data;
determining at least one target object according to the basic data, analyzing the basic data based on the target object, and generating a data set which is associated with the target object and has a keyword identification;
receiving a query request sent by a client, and matching a data set corresponding to the query request as target data based on the query request;
and generating a target report based on the target data, and sending the target report to a client.
Further, preprocessing the acquired data to obtain basic data, including:
marking the acquired data according to a data source to acquire first processing data;
and matching corresponding processing rules from a preset scheme database according to data sources, and cleaning the first processing data by adopting the processing rules to obtain basic data.
Further, analyzing the basic data based on the target object to generate a data set with a keyword identifier associated with the target object, including:
acquiring any target object, and matching an analysis model corresponding to the target object from a model database;
processing corresponding basic data according to the analysis model to obtain an analysis result of the target object;
and analyzing the analysis result to generate a data set which is associated with the target object and has a keyword identification.
Further, analyzing the analysis result to generate a data set with a keyword identifier associated with the target object, including:
acquiring at least one dimension data based on the target analysis object;
decomposing the analysis result based on the dimension data to generate a data set corresponding to each dimension data;
marking a corresponding data set by adopting the dimension data to obtain a data set which is associated with the target object and has a keyword identifier;
and uploading the basic data, the analysis result and the data set with the keyword identification to a block chain.
Further, acquiring a data set corresponding to the query request as target data based on the query request includes:
acquiring user permission and keywords based on the query request;
verifying whether the viewing permission corresponding to the keyword is matched with the user permission;
if so, acquiring a data set matched with the query request as target data according to the keyword;
and if not, obtaining an un-inquired result as target data.
Further, generating a target report based on the target data includes:
matching corresponding template data from a template database based on the data type of the target data;
and generating the target report according to the template data and the data set.
In order to achieve the above object, the present invention also provides a data processing apparatus comprising:
the acquisition module is used for acquiring data in real time and preprocessing the acquired data to acquire basic data;
the analysis module is used for analyzing the basic data to generate a plurality of data sets with keyword identifications;
the matching module is used for receiving a query request sent by a client and matching a data set corresponding to the query request as target data based on the query request;
and the generating module is used for generating a target report based on the target data and sending the target report to the client.
Further, the analysis module comprises:
an analysis object determination unit configured to acquire at least one target analysis object based on the basic data;
an analysis model acquisition unit configured to match an analysis model corresponding to each of the target analysis objects from a model database;
the processing unit is used for processing the basic data one by adopting the analysis model to obtain an analysis result;
and the analysis unit is used for analyzing the analysis result to generate a data set with the keyword identification.
To achieve the above object, the present invention further provides a computer system, which includes a plurality of computer devices, each computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processors of the plurality of computer devices collectively implement the steps of the data processing method described above when executing the computer program.
To achieve the above object, the present invention further provides a computer-readable storage medium including a plurality of storage media, each storage medium having a computer program stored thereon, the computer programs stored in the storage media collectively implementing the steps of the data processing method when being executed by a processor.
According to the data processing method, the data processing device, the computer system and the readable storage medium, the target object is determined after real-time data are collected and the obtained data are processed and analyzed, a plurality of data sets with the keyword identifications are generated in advance based on the analysis of the target object, and after a user sends an inquiry request, the corresponding data sets are matched according to the user request to directly generate a target report, so that the problems that in the prior art, the final analysis result is visually displayed in the forms of common reports and the like, but the data of a certain class or a certain dimension cannot be displayed in an image mode, the user needs to independently search for related data and then manually analyze the related data, the inquiry speed is low, the operation is complex, and the working efficiency is low are solved.
Drawings
FIG. 1 is a flowchart of a first embodiment of a data processing method according to the present invention;
fig. 2 is a specific flowchart of preprocessing the acquired data to obtain basic data according to a first embodiment of the data processing method of the present invention;
FIG. 3 is a flowchart illustrating a specific process of analyzing the basic data to generate a plurality of data sets with keyword identifiers according to a first embodiment of the data processing method of the present invention;
fig. 4 is a specific flowchart of analyzing the analysis result to generate a data set with a keyword identifier according to the first embodiment of the data processing method of the present invention;
fig. 5 is a specific flowchart of acquiring a data set corresponding to the query request as target data based on the query request in the first embodiment of the data processing method according to the present invention;
FIG. 6 is a flowchart illustrating a specific process of generating a target report based on the target data according to a first embodiment of the data processing method of the present invention;
FIG. 7 is a block diagram of program modules of a second embodiment of a data processing apparatus according to the present invention;
FIG. 8 is a block diagram of an analysis module in a second embodiment of the data processing apparatus according to the present invention;
fig. 9 is a schematic diagram of a hardware structure of a computer device in the third embodiment of the computer system according to the present invention.
Reference numerals:
5. data processing device 510, acquisition module 520, analysis module 530, and matching module
540. Generation module 521, analysis object determination unit 522, and analysis model acquisition unit
523. Processing unit 524, analysis unit 6, computer device 61, memory
62. Processor with a memory having a plurality of memory cells
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The data processing method, the data processing device, the computer system and the readable storage medium are suitable for the field of data processing of artificial intelligence technology, and provide a data processing method of an acquisition module, an analysis module, a matching module and a generation module. The invention acquires real-time data through an acquisition module, preprocesses the acquired data to acquire basic data, analyzes the basic data through an analysis module, firstly matches a proper analysis model to analyze the basic data to acquire an analysis result, then disassembles the analysis result based on each dimension data of the analysis model to generate a data set with a keyword identifier corresponding to each dimension data, after receiving a query request sent by a user, adopts a matching module to match the data set corresponding to the query request as target data based on the query request, finally utilizes a generation module to generate a target report based on the target data and sends the target report to a user end for visual display, generates a plurality of data sets by splitting the analysis result in advance, and packs the basic data and the analysis result, the method and the device have the advantages that when a user inquires corresponding analysis results according to needs, the packed analysis data can be directly acquired for displaying, the method and the device are different from the prior art that only final analysis results are displayed in the same group of reports, the problems that the data analysis results under a specific dimensionality can be obtained only by the user independently searching partial data and then performing manual analysis on the partial data in the existing reports are solved, the inquiry speed is low, the operation is complex, and the working efficiency is low.
Example one
Referring to fig. 1, a data processing method of the present embodiment includes:
s100: acquiring data in real time, and preprocessing the acquired data to acquire basic data;
in the above embodiments, the collected data includes, but is not limited to, databases (e.g., Oracle database), customer entries (e.g., billing data, etc.), and other external data.
Specifically, preprocessing the acquired data to obtain basic data, referring to fig. 2, includes:
s110: marking the acquired data according to a data source to acquire first processing data;
in the above embodiments, the data sources include, but are not limited to, the following: user behavior of the terminal (Web, App, H5), backend server Log (Log), and business data (DataBase).
S120: and matching corresponding processing rules from a preset scheme database according to data sources, and cleaning the first processing data by adopting the processing rules to obtain basic data.
In this embodiment, the data may be divided into data with scene tags according to the data source, for example: for different scene tags, one or more data cleansing rules may be configured for the big data platform data, the graph data, and other data, by way of example and not limitation, for the big data platform data, a python script is used to call hive to achieve cleansing, and for the image data, an OCR recognition technology is used to perform recognition and cleansing processing, and the like.
S200: determining at least one target object according to the basic data, analyzing the basic data based on the target object, generating a data set with keyword identification associated with the target object, and generating a plurality of data sets with keyword identification;
in the above embodiment, the target analysis object is a final presentation result of data analysis, and an object to be analyzed is determined in advance according to basic data, such as various risk data analysis and business data analysis. Determining a target object according to the basic data, wherein the target object can be preset, or a form can be provided in advance, a plurality of target objects are preset in the form, each target object is associated with a plurality of dimensional data, dimension classification is performed on the basic data, and then the target object is determined according to the dimension corresponding to the basic data.
Specifically, analyzing the basic data based on the target object to generate a data set with a keyword identifier associated with the target object, please refer to fig. 3, which includes:
s210: acquiring any target object, and matching an analysis model corresponding to the target object from a model database;
in the above embodiment, a plurality of analysis models are prestored in the model database, and new analysis models are continuously updated or added based on changes in business requirements, the analysis models are obtained according to logic training of various businesses, and corresponding analysis models are selected according to target analysis objects to process data, that is, an analysis model is determined to map input features to predicted output, the analysis models may be a risk control model, a profit prediction model, a data classification model, and the like, and may be applied to scenes such as anti-fraud, post-loan monitoring, group case search, and the like, by way of example and not limitation, for data of the above large data platform type, a python script is used to call spark to realize data modeling, and for data of the above image type, an OCR recognition technology is used, and then, graph database modeling is used to perform analysis based on graph relationship modeling, and the like.
S220: processing corresponding basic data according to the analysis model to obtain an analysis result of the target object;
by way of example and not limitation, regarding the borrowing risk analysis as an example, when the amount of money borrowed by the user is small, the basic information of the user is acquired and analyzed to obtain the loan risk degree of the user, and whether the borrowing request of the user passes is judged based on the risk degree, when the amount of money borrowed by the user is large, the multi-dimensional information such as the basic information, the business information, the liability ratio and the like of the user is acquired and analyzed to judge whether the borrowing request of the user passes is judged, more specifically, the final risk result is obtained by scoring or weighting based on the multi-dimensional information such as the basic information, the business information, the liability ratio and the like, and the risk result is the analysis result, which may be presented as a chart or presented as a value set.
S230: and analyzing the analysis result to generate a data set which is associated with the target object and has a keyword identification.
Specifically, parsing the analysis result to generate a data set with a keyword identifier, please refer to fig. 4, which includes:
s231: acquiring at least one dimension data based on the target analysis object;
in this scheme, the analysis models and the dimension indexes acting on the various types of analysis models are obtained one by one based on the target analysis object, that is, each dimension index in the analysis models is obtained as dimension data, as an example, a certain target analysis object is certain APP user data, and the corresponding analysis result includes, but is not limited to, the following dimension indexes: the method comprises the following steps of operation timeliness analysis, APP online operation analysis, real-time data burying analysis, finest granularity ledger and the like, wherein the operation timeliness analysis is used for analyzing timeliness of each data scheduling process; the APP online operation analysis is to collect and analyze operation data, the embedded data analysis refers to a relevant technology and an implementation process thereof for capturing, processing and sending specific user behaviors or events, such as the number of times of clicking a certain button of a user, the duration of browsing a certain content and the like, the embedded point obtains user log data, and through the processing, analysis and modeling of the data, the preference and the demand of the user can be mined, the effect and the future trend of a product can be judged, and various system indexes can be monitored; the finest granularity ledger refers to subdividing and analyzing data directly recorded by an operator from the ledger records in the operation process; the operation timeliness analysis, the APP online operation analysis, the real-time buried point data analysis and the finest-granularity ledger are all dimension data corresponding to the target analysis object (namely APP user data).
S232: decomposing the analysis result based on the dimension data to generate a data set corresponding to each dimension data;
in the above embodiment, the analysis result is decomposed based on any dimension data, mainly by obtaining the analysis result and the basic data corresponding to the dimension data, and removing the analysis data irrelevant to the dimension data, that is, dividing the analysis result and the basic data into a plurality of data sets corresponding to the dimension indexes; for example, a certain analysis result includes analysis data M, N, O, P with multiple dimensions, where the data M is visiting time, visiting area, visiting page, current staying page, etc.; data N visitor browsing times, total in-station browsing times and the like; the data O, P is other APP operation management data, and the data set corresponding to the dimension data buried data analysis is { data M, data N }, and the data set corresponding to the dimension data online operation analysis is { data O, data P }
S233: and marking the corresponding data set by adopting the dimension data to obtain the data set with the keyword identification.
It should be noted that the keywords are obtained based on the dimensional data, each piece of dimensional data corresponds to a keyword, the data set with the keyword identifier is the data set corresponding to each piece of dimensional data, as in the above example, "buried data analysis" in S242, the corresponding data set is { data M, data N }, and a buried data analysis model is established based on data M, Q to predict the future trend of the product, so that a data set containing data M, N defined as the data set with the buried data analysis identifier is obtained.
As an example for facilitating understanding of the foregoing steps S241 to S243, taking the simple loan risk analysis scenario described in the foregoing step S230 as an example, the loan risk analysis includes dimensional data such as basic information, business information, liability ratio, and the like, each of the dimensional data corresponds to a data set, the data sets are marked by the corresponding dimensional data to obtain a plurality of data sets with keyword identifiers (corresponding to keywords such as the basic information, the business information, the liability ratio information, and the like), an analysis result displayed in a graph form is obtained by using an analysis model, the analysis result includes sub-tags (i.e., keyword tags) corresponding to a plurality of dimensional data, when a user queries analysis data corresponding to a certain dimensional data, the corresponding data set is obtained based on the sub-tag clicked by the user and displayed without displaying the analysis data corresponding to each dimensional index in the analysis result, the user can search the corresponding dimension index.
In the above scheme, the manner of generating the data set corresponding to each dimension data in advance is equivalent to packing the analysis data and the basic data corresponding to each dimension data in the analysis result, so that when a user queries the analysis result under the corresponding dimension according to the requirement, several packed data can be directly obtained for display, and the method is different from the method in the prior art in which all the analysis data are displayed in the same group of reports, and improves the query efficiency of the user.
In this embodiment, the basic data, the analysis result, and the data set with the keyword identifier may also be uploaded to a block chain, and the user equipment may download the basic data, the analysis result, and the data set with the keyword identifier from the block chain, so as to verify whether the data is tampered. The blockchain referred to in this example is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm, and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
S300: receiving a query request sent by a client, and matching a data set corresponding to the query request as target data based on the query request;
acquiring a data set corresponding to the query request as target data based on the query request, referring to fig. 5, including:
s310: acquiring user permission and keywords based on the query request;
s320: verifying whether the viewing authority corresponding to the query data is matched with the user authority;
specifically, the authority is set for the analysis result and the data set with the keyword identification so as to improve the data security, and the result display corresponding to the query data can be checked only when the user authority is higher than the data authority matched with the query data.
S330: if the data set is matched with the query request, acquiring a keyword according to the query request, and acquiring a data set matched with the query request as target data based on the keyword;
s340: and if not, obtaining an un-inquired result as target data.
Through the manner of setting the authority in the S31-S34, the data can be automatically divided into different layers according to the architecture, different users can check the data with different dimensions, and different work interfaces and various customized reports are formed through authority control.
In the above embodiment, the user right and the keyword may be obtained based on the query request, when the keyword is obtained through the query request, part of the keyword may be consistent with the keyword, or the keyword may be completely consistent with the keyword, and when a situation that both the keywords are partially matched with the query request occurs, the analysis result which is more consistent with the keyword is preferentially displayed, so that a situation that a target report cannot be obtained due to less information input by part of users is reduced.
S400: and generating a target report based on the target data, and sending the target report to a client.
Specifically, the generating of the target report based on the target data, referring to fig. 6, includes:
s410: matching corresponding template data from a template database based on the data type of the target data;
specifically, the template types include, but are not limited to, the above types including, but not limited to, forms, line graphs, pie charts, and the like.
S420: and generating the target report according to the template data and the data set.
In the scheme, a template database is preset, and template data matched with a plurality of data set types in the template database is preset. After the corresponding data set is acquired based on user requirements, the corresponding data set is matched with the template data to generate a target report, the implementation that a user sends a query request to the generation of the target report is shortened, the second-level display effect is achieved, the query time is further effectively shortened, and the working efficiency of the user is improved.
According to the scheme, the data are collected and analyzed, then the obtained analysis result is disassembled, a plurality of data sets with the keyword identification are generated in advance, after a user sends a query request, the corresponding data sets are matched according to the user request to directly generate a target report, the time of user query is greatly shortened, and the problems that in the prior art, the final analysis result is visually presented in the forms of common reports and the like, but the condition presentation of certain type or certain dimension data cannot be carried out, the user needs to independently search and search related data and then manually analyze the data, the query speed is low, the operation is complex, and the working efficiency is low are solved.
Meanwhile, the display of the data set corresponding to each dimension data is controlled in a mode of setting the authority, the analysis result and the basic data can be autonomously divided into different levels according to the architecture by using the authority control, and users in different levels can check the data in different levels to form different working interfaces and various customized reports.
Example two:
referring to fig. 7, a data processing apparatus 5 of the present embodiment includes:
the acquisition module 510 is configured to acquire data in real time, and preprocess the acquired data to acquire basic data;
the collected data includes, but is not limited to, databases (e.g., Oracle databases), customer entries (e.g., billing data, etc.), and other external data.
An analysis module 520, configured to analyze the basic data to generate a plurality of data sets with keyword identifiers;
referring to fig. 8, the analysis module 520 further includes the following:
an analysis object determination unit 521 for acquiring at least one target analysis object based on the basic data;
in the scheme, the target analysis object is a final presentation result of data analysis, and the object to be analyzed is judged in advance according to basic data, such as various risk data analysis, operation data analysis and the like.
An analysis model obtaining unit 522, configured to match an analysis model corresponding to each target analysis object from a model database;
specifically, the analysis model may be a risk control model, a profit prediction model, a data classification model, and the like, and is applied to data analysis obtained in scenes such as anti-fraud, post-loan monitoring, group case search, and the like.
A processing unit 523, configured to process the basic data one by using the analysis model to obtain an analysis result;
and an analyzing unit 524, configured to analyze the analysis result to generate a data set with a keyword identifier.
In the embodiment, the analysis result is decomposed based on any dimension data, and mainly the analysis result and the basic data corresponding to the dimension data are obtained, and the analysis data irrelevant to the dimension data are removed, that is, the analysis result and the basic data are divided into data sets corresponding to a plurality of dimension indexes and marked, so that a data set with the keyword identification can be generated.
It should be noted that, in order to further ensure the privacy and security of the acquired data, basic data, analysis results, keywords, and data sets, the acquired data, basic data, analysis results, keywords, and data sets may be stored in a node of a block chain. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
A matching module 530, configured to receive a query request sent by a client, and match, based on the query request, a data set corresponding to the query request as target data;
and the generating module 540 is configured to generate a target report based on the target data, and send the target report to the client.
According to the technical scheme, real-time data are collected through a collection module, the collected data are preprocessed to obtain basic data, the basic data are analyzed through an analysis module, data sets with keyword identifications corresponding to dimensional data are generated, after a query request sent by a user is received, a matching module is adopted to match the data sets corresponding to the query request based on the query request to serve as target data, and finally a generation module is used for generating a target report based on the target data and sending the target report to a user side for visual display.
When generating a data set with keyword identification, firstly, an analysis object determining unit is adopted to obtain a target analysis object, a proper analysis model is matched according to the target analysis object and the analysis model obtaining unit, basic data is analyzed, an analysis result is obtained for conventional display, finally, the analysis result is analyzed according to dimension data by utilizing an analysis unit, a data set corresponding to each dimension data is obtained, a user request is received subsequently, the data set under the corresponding dimension is matched according to the user request for display, the data under each dimension is classified and packaged by the method, the time of user query is greatly shortened, the problem that the final analysis result is visually displayed in the forms of common reports and the like in the prior art is solved, but the data of a certain class or a certain dimension can not be presented with images, and a user needs to independently search for related data and then manually analyze the data is solved, the query speed is slow, the operation is complex, and the working efficiency is low.
Example three:
in order to achieve the above object, the present invention further provides a computer system, which includes a plurality of computer devices 6, components of the data processing apparatus 1 according to the second embodiment can be distributed in different computer devices, and the computer devices can be smartphones, tablet computers, notebook computers, desktop computers, rack servers, blade servers, tower servers, or rack servers (including independent servers or a server cluster formed by a plurality of servers) which execute programs, and the like. The computer device of the embodiment at least includes but is not limited to: a memory 61, a processor 62, which may be communicatively coupled to each other via a system bus, as shown in fig. 9. It should be noted that fig. 9 only shows a computer device with components, but it should be understood that not all of the shown components are required to be implemented, and more or fewer components may be implemented instead.
In the present embodiment, the memory 61 (i.e., a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the memory 61 may be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. In other embodiments, the memory 61 may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like, provided on the computer device. Of course, the memory 61 may also include both internal and external storage devices of the computer device. In the present embodiment, the program code of the data processing apparatus of the memory 61, and the like. Further, the memory 51 may also be used to temporarily store various types of data that have been output or are to be output.
Processor 62 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 62 is typically used to control the overall operation of the computer device. In this embodiment, the processor 62 is configured to execute the program code stored in the memory 61 or process data, for example, execute a knowledge graph constructing apparatus, so as to implement the data processing method according to the first embodiment.
Example four:
to achieve the above objects, the present invention also provides a computer-readable storage system including a plurality of storage media, such as a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, which when executed by a processor 62, implements corresponding functions. The computer readable storage medium of this embodiment is used for storing a knowledge graph construction apparatus, and when executed by the processor 62, implements the data processing method of the first embodiment.
In one embodiment, the computer-readable storage medium includes a storage data area storing data created according to use of a blockchain node and a storage program area storing a computer program; wherein the computer program realizes the data processing method of any of the embodiments when executed by the processor 62.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A data processing method, comprising:
acquiring data in real time, and preprocessing the acquired data to acquire basic data;
determining at least one target object according to the basic data, analyzing the basic data based on the target object, and generating a data set which is associated with the target object and has a keyword identification;
receiving a query request sent by a client, and matching a data set corresponding to the query request as target data based on the query request;
and generating a target report based on the target data, and sending the target report to a client.
2. The data processing method according to claim 1, wherein preprocessing the acquired data to obtain basic data comprises:
marking the acquired data according to a data source to acquire first processing data;
and matching corresponding processing rules from a preset scheme database according to data sources, and cleaning the first processing data by adopting the processing rules to obtain basic data.
3. The data processing method of claim 1, wherein analyzing the base data based on the target object to generate a data set with keyword identifiers associated with the target object comprises:
acquiring any target object, and matching an analysis model corresponding to the target object from a model database;
processing corresponding basic data according to the analysis model to obtain an analysis result of the target object;
and analyzing the analysis result to generate a data set which is associated with the target object and has a keyword identification.
4. The data processing method of claim 3, wherein parsing the analysis result to generate a data set with keyword identifiers associated with the target object comprises:
acquiring at least one dimension data based on the target analysis object;
decomposing the analysis result based on the dimension data to generate a data set corresponding to each dimension data;
marking a corresponding data set by adopting the dimension data to obtain a data set which is associated with the target object and has a keyword identifier;
and uploading the basic data, the analysis result and the data set with the keyword identification to a block chain.
5. The data processing method according to claim 1, wherein acquiring a data set corresponding to the query request as target data based on the query request comprises:
acquiring user permission and keywords based on the query request;
verifying whether the viewing permission corresponding to the keyword is matched with the user permission;
if so, acquiring a data set matched with the query request as target data according to the keyword;
and if not, obtaining an un-inquired result as target data.
6. The data processing method of claim 1, wherein generating a target report based on the target data comprises:
matching corresponding template data from a template database based on the data type of the target data;
and generating the target report according to the template data and the data set.
7. A data processing apparatus, characterized by comprising:
the acquisition module is used for acquiring data in real time and preprocessing the acquired data to acquire basic data;
the analysis module is used for analyzing the basic data to generate a plurality of data sets with keyword identifications;
the matching module is used for receiving a query request sent by a client and matching a data set corresponding to the query request as target data based on the query request;
and the generating module is used for generating a target report based on the target data and sending the target report to the client.
8. A data processing apparatus as claimed in claim 7, wherein the analysis module comprises:
an analysis object determination unit configured to acquire at least one target analysis object based on the basic data;
an analysis model acquisition unit configured to match an analysis model corresponding to each of the target analysis objects from a model database;
the processing unit is used for processing the basic data one by adopting the analysis model to obtain an analysis result;
and the analysis unit is used for analyzing the analysis result to generate a data set with the keyword identification.
9. A computer system comprising a plurality of computer devices, each computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processors of the plurality of computer devices when executing the computer program collectively implement the steps of a data processing method as claimed in any one of claims 1 to 6.
10. A computer-readable storage medium comprising a plurality of storage media, each storage medium having a computer program stored thereon, wherein the computer programs stored in the plurality of storage media, when executed by a processor, collectively implement the steps of a data processing method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010740433.6A CN111831636A (en) | 2020-07-28 | 2020-07-28 | Data processing method, device, computer system and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010740433.6A CN111831636A (en) | 2020-07-28 | 2020-07-28 | Data processing method, device, computer system and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111831636A true CN111831636A (en) | 2020-10-27 |
Family
ID=72921024
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010740433.6A Pending CN111831636A (en) | 2020-07-28 | 2020-07-28 | Data processing method, device, computer system and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111831636A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112436963A (en) * | 2020-11-12 | 2021-03-02 | 中国联合网络通信集团有限公司 | Data analysis method and device |
| CN112527848A (en) * | 2020-12-22 | 2021-03-19 | 苏州科达科技股份有限公司 | Multi-data-source-based report data query method, device, system and storage medium |
| CN112597745A (en) * | 2020-12-23 | 2021-04-02 | 深圳赛安特技术服务有限公司 | Intelligent chart generation method and device, computer system and readable storage medium |
| CN112817951A (en) * | 2021-01-19 | 2021-05-18 | 立梦(上海)企业管理咨询有限公司 | Data analysis method for new media marketing |
| CN112818007A (en) * | 2021-02-03 | 2021-05-18 | 中科驭数（北京）科技有限公司 | Data processing method and device and readable storage medium |
| CN112926984A (en) * | 2021-04-13 | 2021-06-08 | 郭栋 | Information prediction method based on block chain safety big data and block chain service system |
| CN116151694A (en) * | 2023-04-19 | 2023-05-23 | 中国建筑科学研究院有限公司 | Data management system, method and computing device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107908606A (en) * | 2017-10-31 | 2018-04-13 | 上海壹账通金融科技有限公司 | Method and system based on different aforementioned sources automatic report generation |
| US10366079B1 (en) * | 2015-01-26 | 2019-07-30 | Microstrategy Incorporated | Enterprise connectivity |
| CN110334122A (en) * | 2019-07-11 | 2019-10-15 | 江苏曲速教育科技有限公司 | The query analysis method and system of educational data |
| CN110688541A (en) * | 2019-10-08 | 2020-01-14 | 中国建设银行股份有限公司 | Report data query method and device, storage medium and electronic equipment |
-
2020
- 2020-07-28 CN CN202010740433.6A patent/CN111831636A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10366079B1 (en) * | 2015-01-26 | 2019-07-30 | Microstrategy Incorporated | Enterprise connectivity |
| CN107908606A (en) * | 2017-10-31 | 2018-04-13 | 上海壹账通金融科技有限公司 | Method and system based on different aforementioned sources automatic report generation |
| CN110334122A (en) * | 2019-07-11 | 2019-10-15 | 江苏曲速教育科技有限公司 | The query analysis method and system of educational data |
| CN110688541A (en) * | 2019-10-08 | 2020-01-14 | 中国建设银行股份有限公司 | Report data query method and device, storage medium and electronic equipment |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112436963A (en) * | 2020-11-12 | 2021-03-02 | 中国联合网络通信集团有限公司 | Data analysis method and device |
| CN112527848A (en) * | 2020-12-22 | 2021-03-19 | 苏州科达科技股份有限公司 | Multi-data-source-based report data query method, device, system and storage medium |
| CN112527848B (en) * | 2020-12-22 | 2023-05-12 | 苏州科达科技股份有限公司 | Report data query method, device and system based on multiple data sources and storage medium |
| CN112597745A (en) * | 2020-12-23 | 2021-04-02 | 深圳赛安特技术服务有限公司 | Intelligent chart generation method and device, computer system and readable storage medium |
| CN112817951A (en) * | 2021-01-19 | 2021-05-18 | 立梦(上海)企业管理咨询有限公司 | Data analysis method for new media marketing |
| CN112818007A (en) * | 2021-02-03 | 2021-05-18 | 中科驭数（北京）科技有限公司 | Data processing method and device and readable storage medium |
| CN112818007B (en) * | 2021-02-03 | 2021-10-19 | 中科驭数（北京）科技有限公司 | Data processing method and device and readable storage medium |
| CN112926984A (en) * | 2021-04-13 | 2021-06-08 | 郭栋 | Information prediction method based on block chain safety big data and block chain service system |
| CN116151694A (en) * | 2023-04-19 | 2023-05-23 | 中国建筑科学研究院有限公司 | Data management system, method and computing device |
| CN116151694B (en) * | 2023-04-19 | 2023-06-20 | 中国建筑科学研究院有限公司 | Data management system, method and computing device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111831636A (en) | Data processing method, device, computer system and readable storage medium | |
| CN113610239B (en) | Feature processing method and feature processing system for machine learning | |
| US20190050435A1 (en) | Object data association index system and methods for the construction and applications thereof | |
| CN107818491A (en) | Electronic installation, Products Show method and storage medium based on user's Internet data | |
| CN112632405A (en) | Recommendation method, device, equipment and storage medium | |
| CN113836131A (en) | Big data cleaning method and device, computer equipment and storage medium | |
| CN110909195A (en) | Picture labeling method and device based on block chain, storage medium and server | |
| CN113946690A (en) | Potential customer mining method and device, electronic equipment and storage medium | |
| CN110543565A (en) | Auditing method, system and readable storage medium based on convolutional neural network model | |
| CN116362684A (en) | Library cluster-based book management method, library cluster-based book management device, library cluster-based book management equipment and storage medium | |
| CN116956326A (en) | Authority data processing method and device, computer equipment and storage medium | |
| CN116860311A (en) | Script analysis method, script analysis device, computer equipment and storage medium | |
| CN114841165B (en) | User data analysis and display method and device, electronic equipment and storage medium | |
| CN114817754B (en) | VR learning system | |
| CN115860835A (en) | Advertisement recommendation method, device and equipment based on artificial intelligence and storage medium | |
| CN115114519A (en) | Artificial intelligence based recommendation method and device, electronic equipment and storage medium | |
| CN114416806A (en) | Method and device for acquiring power safety knowledge data and computer equipment | |
| Motohashi et al. | Technological competitiveness of China's internet platformers: comparison of Google and Baidu by using patent text information | |
| CN114398562A (en) | Shop data management method, device, equipment and storage medium | |
| CN111460300A (en) | Network content pushing method and device and storage medium | |
| CN113724065B (en) | Auxiliary collecting method, device, equipment and storage medium based on flow guidance | |
| CN114510662B (en) | Multi-source data card display method and system and storage medium | |
| CN116450667A (en) | Data query method, device, computer equipment and storage medium | |
| CN113946751A (en) | Data acquisition and interpretation method, device, equipment and medium based on artificial intelligence | |
| CN117291693A (en) | Policy generation method, device, equipment and storage medium based on artificial intelligence |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |