CN117033467A - Data mining method and device based on pre-calculation and storage medium - Google Patents
Data mining method and device based on pre-calculation and storage medium Download PDFInfo
- Publication number
- CN117033467A CN117033467A CN202311057065.5A CN202311057065A CN117033467A CN 117033467 A CN117033467 A CN 117033467A CN 202311057065 A CN202311057065 A CN 202311057065A CN 117033467 A CN117033467 A CN 117033467A
- Authority
- CN
- China
- Prior art keywords
- service request
- data
- mining analysis
- mining
- intermediate result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004364 calculation method Methods 0.000 title claims abstract description 79
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000007418 data mining Methods 0.000 title claims abstract description 38
- 238000003860 storage Methods 0.000 title claims abstract description 25
- 238000004458 analytical method Methods 0.000 claims abstract description 103
- 238000005065 mining Methods 0.000 claims abstract description 102
- 238000004590 computer program Methods 0.000 claims description 15
- 230000004044 response Effects 0.000 claims description 7
- 238000000605 extraction Methods 0.000 claims description 6
- 238000004140 cleaning Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 15
- 230000006870 function Effects 0.000 description 9
- 238000012545 processing Methods 0.000 description 9
- 230000006872 improvement Effects 0.000 description 8
- 238000004422 calculation algorithm Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 206010000117 Abnormal behaviour Diseases 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 229920001296 polysiloxane Polymers 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 239000010979 ruby Substances 0.000 description 1
- 229910001750 ruby Inorganic materials 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24561—Intermediate data storage techniques for performance improvement
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Probability & Statistics with Applications (AREA)
- Mathematical Physics (AREA)
- Fuzzy Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The specification discloses a data mining method, a device and a storage medium based on pre-calculation, which are used for acquiring data to be mined, responding to a service request of a user, pre-calculating the data to be mined according to the service request, acquiring an intermediate result and storing the intermediate result; judging whether the mining analysis result corresponding to the service request exists in a stored mining analysis result set according to the service request, if not, carrying out mining analysis on the data to be mined according to the intermediate result when pre-calculation is completed, obtaining the mining analysis result corresponding to the service request, and displaying. According to the method, the mining analysis is carried out through the pre-calculated intermediate result, so that the time for mining and analyzing the data to be mined is reduced, and the efficiency of the data mining and analyzing is improved.
Description
Technical Field
The present disclosure relates to the field of computers, and in particular, to a data mining method, apparatus and storage medium based on pre-computation.
Background
With the development of internet technology and the change of demands of users, more and more data need to be processed by a computer. The computer can analyze the data to mine effective information in the data when processing the data, and can acquire the data mining analysis result in real time by adding computing resources, such as memory, when the data mining analysis is performed. But increases the computational resources as well as the cost of data maintenance and hardware. Moreover, when more data or the algorithm of data mining analysis is complex, the data mining analysis result cannot be obtained in real time only by adding computing resources, so that the data mining efficiency is low.
Based on this, the present specification provides a data mining method based on pre-calculation.
Disclosure of Invention
The present disclosure provides a data mining method, apparatus, storage medium and electronic device based on pre-calculation, so as to partially solve the above-mentioned problems in the prior art.
The technical scheme adopted in the specification is as follows:
the specification provides a data mining method based on pre-calculation, comprising the following steps:
acquiring data to be mined;
responding to a service request of a user, pre-calculating the data to be mined according to the service request, obtaining an intermediate result, and storing the intermediate result; judging whether a mining analysis result corresponding to the service request exists in a stored mining analysis result set according to the service request;
if not, when pre-calculation is completed, mining and analyzing the data to be mined according to the intermediate result, obtaining mining and analyzing results corresponding to the service request, and displaying the mining and analyzing results.
Optionally, pre-computing the data to be mined according to the service request to obtain an intermediate result, which specifically includes:
according to the service request, determining a pre-calculation model matched with the service request from a plurality of pre-calculation models which are pre-constructed;
inputting the data to be mined into the pre-calculation model to obtain an intermediate result output by the pre-calculation model, wherein the pre-calculation comprises data cleaning and feature extraction.
Optionally, pre-computing models are pre-built, specifically including:
for each service, acquiring data of the service;
according to the data of the service, pre-calculation model parameters are configured to construct a pre-calculation model.
Optionally, after obtaining the intermediate result, the method further comprises:
for each intermediate result, an index of the service request and the intermediate result is established and stored.
Optionally, according to the intermediate result, performing mining analysis on the data to be mined, which specifically includes:
determining an intermediate result of the service request according to the index;
and mining and analyzing the data to be mined according to the intermediate result.
Optionally, the method further comprises:
and storing the mining analysis result, and storing the corresponding relation between the service request and the mining analysis result.
Optionally, according to the service request, judging whether the mining analysis result corresponding to the service request exists in the stored mining analysis result set, which specifically includes:
judging whether the mining analysis result corresponding to the service request exists in the stored mining analysis result set according to the corresponding relation between the service request and the stored mining analysis result set.
The present specification provides a data mining apparatus based on pre-computation, comprising:
the data acquisition module is used for acquiring data to be mined;
the response module is used for responding to a service request of a user, pre-calculating the data to be mined according to the service request, obtaining an intermediate result and storing the intermediate result; judging whether a mining analysis result corresponding to the service request exists in a stored mining analysis result set according to the service request;
and the result acquisition module is used for carrying out mining analysis on the data to be mined according to the intermediate result when the pre-calculation is finished if not, so as to obtain mining analysis results corresponding to the service request and display the mining analysis results.
Optionally, the response module is specifically configured to determine, according to the service request, a pre-calculation model matched with the service request from a plurality of pre-calculation models constructed in advance; inputting the data to be mined into the pre-calculation model to obtain an intermediate result output by the pre-calculation model, wherein the pre-calculation comprises data cleaning and feature extraction.
Optionally, the apparatus further comprises:
the model construction module is used for acquiring data of each service according to each service; according to the data of the service, pre-calculation model parameters are configured to construct a pre-calculation model.
Optionally, the apparatus further comprises:
and the index establishing module is used for establishing an index of the service request and each intermediate result and storing the index.
Optionally, the result obtaining module is specifically configured to determine, according to the index, an intermediate result of the service request; and mining and analyzing the data to be mined according to the intermediate result.
Optionally, the apparatus further comprises:
and the storage module is used for storing the mining analysis result and storing the corresponding relation between the service request and the mining analysis result.
Optionally, the result obtaining module is specifically configured to determine, according to a correspondence between the service request and a stored mining analysis result set, whether a mining analysis result corresponding to the service request exists in the stored mining analysis result set.
The present specification provides a computer readable storage medium storing a computer program which when executed by a processor implements the pre-computation based data mining method described above.
The present specification provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the pre-calculation based data mining method described above when executing the program.
The above-mentioned at least one technical scheme that this specification adopted can reach following beneficial effect:
according to the pre-calculation-based data mining method provided by the specification, the method is used for pre-calculating the data to be mined, judging whether a result corresponding to the current service request exists in the stored mining analysis results, and if not, performing mining analysis on the data to be mined by using the pre-calculated intermediate result so as to obtain the result corresponding to the current service request. The mining analysis is carried out through the pre-calculated intermediate result, so that the time for carrying out mining analysis on the data to be mined is reduced, and the efficiency of data mining analysis is improved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the specification, illustrate and explain the exemplary embodiments of the present specification and their description, are not intended to limit the specification unduly. In the drawings:
FIG. 1 is a schematic flow chart of a pre-calculation-based data mining method provided in the present specification;
FIG. 2 is a schematic diagram of a pre-calculation based data mining apparatus structure provided in the present specification;
fig. 3 is a schematic structural diagram of an electronic device corresponding to fig. 1 provided in the present specification.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the present specification more apparent, the technical solutions of the present specification will be clearly and completely described below with reference to specific embodiments of the present specification and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present specification. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are intended to be within the scope of the present disclosure.
The following describes in detail the technical solutions provided by the embodiments of the present specification with reference to the accompanying drawings.
Fig. 1 is a schematic flow chart of a pre-calculation-based data mining method provided in the present specification, including the following steps:
s100: and acquiring data to be mined.
The timeliness of the mining analysis result is emphasized by the real-time data mining analysis, namely, a computing upgrading thought is generally adopted, namely, computer resources are increased, and the data mining analysis with low response time is realized by reasonably allocating the computing resources such as a CPU (central processing unit), a memory and the like. However, this causes the related costs of data maintenance and hardware to be increased, and in addition, only by virtue of calculation upgrade, when the data size is large or the algorithm is complex, there is still a problem of low response that analysis is difficult to support, resulting in low data mining efficiency. Accordingly, the present specification is directed to a pre-computed based data mining analysis method.
The execution body of the present specification may be a server for data mining analysis, or may be other electronic devices that may be used for data mining analysis. For convenience of explanation, the description uses a server as an execution subject, and the data mining method based on pre-calculation provided in the description is explained.
In order to perform mining analysis on data, a server may first obtain data to be mined, which may be data in a database.
S102: responding to a service request of a user, pre-calculating the data to be mined according to the service request, obtaining an intermediate result, and storing the intermediate result.
In one or more embodiments of the present description, traffic includes shortest paths, association rules, sub-graph matches, and the like. For different services, the data to be mined of the services are different, and then the corresponding pre-calculations are different. Thus, the server may determine from the service request, among a number of pre-calculated models that are pre-built, a pre-calculated model that the service request matches. The server may determine, from the identity of the service request, a pre-calculation model that matches its own service request, e.g. the identity of the service request is 1, and if the pre-calculation model identity is also 1, the pre-calculation model matches the service request. The present description is not limited in the manner in which the pre-computed model that matches the service request is determined.
And inputting the data to be mined into the pre-calculation model, obtaining an intermediate result output by the pre-calculation model, and storing the intermediate result. When the traffic is the shortest path, the intermediate result includes the shortest path between any two nodes in the network map. When the business is sub-graph matching, the intermediate result includes the data graph and sub-graph matching the business. The intermediate results differ for different services. The pre-calculation comprises data cleaning and feature extraction. Data cleansing includes processing missing values, deleting duplicate data, correcting format errors, removing noise, etc., to improve data quality. When the business is sub-graph matching, feature extraction includes extracting features such as vertex/edge embedding for nodes and edges.
It should be noted that, due to different services, the data to be mined is different, and the pre-calculation model is also different. For different types of traffic, different pre-calculation models need to be built. Wherein, different types of services refer to different services and/or different data to be mined. For example, in the existing path diagram, the diagram includes four nodes A, B, C, D, the sides of the diagram are paths between the nodes, the service 1 is determining the shortest path between a and B, the service 2 is determining the shortest path between C and D, and since the service 1 and the service 2 are both determining the shortest paths, and the data to be mined are both the path diagrams, the service 1 and the service 2 can use the same pre-calculation model.
In addition, since there is a difference in the pre-calculation model, the pre-calculation model may be constructed in advance in order to improve the efficiency of data mining.
Specifically, for each service, data of the service is acquired, and according to the data of the service, pre-calculation model parameters are configured to construct a pre-calculation model. That is, for each service, data in a database of the service is acquired, a mining algorithm for performing pre-calculation is selected according to the service and the data, and pre-calculation model parameters are configured to construct a pre-calculation model. When the service is the shortest path determination, any algorithm, such as a Breadth First Search (BFS), is selected from the algorithms supporting pre-calculation, and pre-calculation model parameters are configured according to the shortest path determination chart and the algorithm to construct a pre-calculation model.
S104: and judging whether the mining analysis result corresponding to the service request exists in the stored mining analysis result set according to the service request, if so, executing S106, and if not, executing S108.
After the mining analysis result corresponding to the service request is obtained, the server can store the mining analysis result and store the corresponding relation between the service request and the mining analysis result, so that the stored mining analysis result can be directly used when the same service request is executed subsequently. Therefore, when determining the mining analysis result, the server can judge whether the mining analysis result corresponding to the service request exists in the stored mining analysis result set according to the service request.
Specifically, according to the corresponding relation between the service request and the stored mining analysis result set, whether the mining analysis result corresponding to the service request exists in the stored mining analysis result set is judged. The present description does not limit the manner in which the correspondence between the service request and the mining analysis result is determined.
S106: and returning the mining analysis result corresponding to the service request to the user.
S108: and when the pre-calculation is completed, carrying out mining analysis on the data to be mined according to the intermediate result, obtaining mining analysis results corresponding to the service request, and displaying the mining analysis results.
If the mining analysis result corresponding to the service request does not exist in the stored mining analysis result set, whether the pre-calculation is completed or not can be judged, so that the mining analysis result is obtained by utilizing the intermediate result, and the mining data analysis efficiency is improved.
Specifically, when the pre-calculation is completed, the server performs mining analysis on the data to be mined according to the intermediate result, obtains mining analysis results corresponding to the service request, and displays the mining analysis results. After the intermediate results are obtained, an index of the service request and the intermediate results is established for each intermediate result, and the index is stored. And determining an intermediate result of the service request according to the index, and performing mining analysis on the data to be mined according to the intermediate result. And when the service is the shortest path, selecting the shortest path of the service from the intermediate results, namely the shortest path between any two nodes in the graph, according to the index to obtain the mining analysis result corresponding to the service request.
Based on the data mining method based on pre-calculation shown in fig. 1, pre-calculating the data to be mined, judging whether a result corresponding to the current service request exists in the stored mining analysis results, and if not, performing mining analysis on the data to be mined by using the pre-calculated intermediate result so as to acquire the result corresponding to the current service request. The mining analysis is carried out through the pre-calculated intermediate result, so that the time for carrying out mining analysis on the data to be mined is reduced, and the efficiency of data mining analysis is improved.
It should be noted that the shortest path, association rule and sub-graph matching can be applied in various fields, such as commodity recommendation, the shortest path can find the nearest neighbor of the user, the association rule finds commodity association, and the sub-graph matching detects the user behavior pattern. Abnormal transaction detection, namely identifying suspicious associated transactions by the shortest path, finding out high risk indexes by association rules, and finding out abnormal behavior pattern by sub-graph matching. And (3) logistics transportation, planning a logistics route by a shortest route, optimizing storage distribution by association rules, and analyzing a traffic network by sub-graph matching. Network security, shortest path recognition attack path, association rule discovery vulnerability association, sub-graph matching detection attack mode and the like. The corresponding data to be mined are data of actual service scenes, and are not described in detail in the specification.
For step S108, the example in S102 is taken along, if the server has already acquired the mining analysis result of the service 1, then, when executing the service 2, the mining analysis result of the service 2 may be acquired according to the intermediate result of the service 1. That is, for the similar service, the mining analysis result of the current service request can be obtained according to the intermediate result of the mining analysis result corresponding to the obtained service request.
The foregoing is a schematic flow diagram of a pre-calculation-based data mining method shown in fig. 1, and the present disclosure further provides a corresponding pre-calculation-based data mining apparatus, as shown in fig. 2.
Fig. 2 is a schematic diagram of a pre-calculation-based data mining apparatus provided in the present specification, including:
the data acquisition module 200 is used for acquiring data to be mined;
a response module 202, configured to respond to a service request of a user, pre-calculate the data to be mined according to the service request, obtain an intermediate result, and store the intermediate result; judging whether a mining analysis result corresponding to the service request exists in a stored mining analysis result set according to the service request;
and the result obtaining module 204 is configured to, if not, perform mining analysis on the data to be mined according to the intermediate result when the pre-calculation is completed, obtain a mining analysis result corresponding to the service request, and display the mining analysis result.
Optionally, the response module 202 is specifically configured to determine, according to the service request, a pre-calculation model that matches the service request from a plurality of pre-calculation models that are pre-constructed; inputting the data to be mined into the pre-calculation model to obtain an intermediate result output by the pre-calculation model, wherein the pre-calculation comprises data cleaning and feature extraction.
Optionally, the apparatus further comprises:
a model building module 206, configured to obtain, for each service, data of the service; according to the data of the service, pre-calculation model parameters are configured to construct a pre-calculation model.
Optionally, the apparatus further comprises:
an index establishing module 208, configured to establish, for each intermediate result, an index of the service request and the intermediate result, and store the index.
Optionally, the result obtaining module 204 is specifically configured to determine, according to the index, an intermediate result of the service request; and mining and analyzing the data to be mined according to the intermediate result.
Optionally, the apparatus further comprises:
and the storage module 210 is configured to store the mining analysis result, and store a correspondence between the service request and the mining analysis result.
Optionally, the result obtaining module 204 is specifically configured to determine, according to the correspondence between the service request and the stored mining analysis result set, whether the mining analysis result corresponding to the service request exists in the stored mining analysis result set.
The present specification also provides a computer readable storage medium storing a computer program operable to perform a pre-calculation based data mining method as provided in fig. 1 above.
The present specification also provides a schematic structural diagram of the electronic device shown in fig. 3, which corresponds to fig. 1. At the hardware level, as shown in fig. 3, the electronic device includes a processor, an internal bus, a network interface, a memory, and a nonvolatile storage, and may of course include hardware required by other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to implement the pre-calculation based data mining method described above with respect to fig. 1.
Of course, other implementations, such as logic devices or combinations of hardware and software, are not excluded from the present description, that is, the execution subject of the following processing flows is not limited to each logic unit, but may be hardware or logic devices.
In the 90 s of the 20 th century, improvements to one technology could clearly be distinguished as improvements in hardware (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software (improvements to the process flow). However, with the development of technology, many improvements of the current method flows can be regarded as direct improvements of hardware circuit structures. Designers almost always obtain corresponding hardware circuit structures by programming improved method flows into hardware circuits. Therefore, an improvement of a method flow cannot be said to be realized by a hardware entity module. For example, a programmable logic device (Programmable Logic Device, PLD) (e.g., field programmable gate array (Field Programmable Gate Array, FPGA)) is an integrated circuit whose logic function is determined by the programming of the device by a user. A designer programs to "integrate" a digital system onto a PLD without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, such programming is mostly implemented by using "logic compiler" software, which is similar to the software compiler used in program development and writing, and the original code before the compiling is also written in a specific programming language, which is called hardware description language (Hardware Description Language, HDL), but not just one of the hdds, but a plurality of kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), lava, lola, myHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog are currently most commonly used. It will also be apparent to those skilled in the art that a hardware circuit implementing the logic method flow can be readily obtained by merely slightly programming the method flow into an integrated circuit using several of the hardware description languages described above.
The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable logic controllers, and embedded microcontrollers, examples of which include, but are not limited to, the following microcontrollers: ARC 625D, atmel AT91SAM, microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic of the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller in a pure computer readable program code, it is well possible to implement the same functionality by logically programming the method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Such a controller may thus be regarded as a kind of hardware component, and means for performing various functions included therein may also be regarded as structures within the hardware component. Or even means for achieving the various functions may be regarded as either software modules implementing the methods or structures within hardware components.
The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functions of each element may be implemented in one or more software and/or hardware elements when implemented in the present specification.
It will be appreciated by those skilled in the art that embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the present specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present description can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
The present description is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the specification. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
It will be appreciated by those skilled in the art that embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the present specification may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present description can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
The description may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.
The foregoing is merely exemplary of the present disclosure and is not intended to limit the disclosure. Various modifications and alterations to this specification will become apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, or the like, which are within the spirit and principles of the present description, are intended to be included within the scope of the claims of the present description.
Claims (10)
1. A pre-calculation based data mining analysis method, the method comprising:
acquiring data to be mined;
responding to a service request of a user, pre-calculating the data to be mined according to the service request, obtaining an intermediate result, and storing the intermediate result; judging whether a mining analysis result corresponding to the service request exists in a stored mining analysis result set according to the service request;
if not, when pre-calculation is completed, mining and analyzing the data to be mined according to the intermediate result, obtaining mining and analyzing results corresponding to the service request, and displaying the mining and analyzing results.
2. The method of claim 1, wherein the pre-computing the data to be mined according to the service request to obtain an intermediate result specifically comprises:
according to the service request, determining a pre-calculation model matched with the service request from a plurality of pre-calculation models which are pre-constructed;
inputting the data to be mined into the pre-calculation model to obtain an intermediate result output by the pre-calculation model, wherein the pre-calculation comprises data cleaning and feature extraction.
3. The method according to claim 2, characterized in that the pre-calculation model is pre-built, in particular comprising:
for each service, acquiring data of the service;
according to the data of the service, pre-calculation model parameters are configured to construct a pre-calculation model.
4. The method of claim 1, wherein after obtaining the intermediate result, the method further comprises:
for each intermediate result, an index of the service request and the intermediate result is established and stored.
5. The method of claim 4, wherein the mining analysis is performed on the data to be mined according to the intermediate result, specifically comprising:
determining an intermediate result of the service request according to the index;
and mining and analyzing the data to be mined according to the intermediate result.
6. The method of claim 1, wherein the method further comprises:
and storing the mining analysis result, and storing the corresponding relation between the service request and the mining analysis result.
7. The method of claim 6, wherein determining whether there is a mining analysis result corresponding to the service request in the stored mining analysis result set according to the service request, specifically comprises:
judging whether the mining analysis result corresponding to the service request exists in the stored mining analysis result set according to the corresponding relation between the service request and the stored mining analysis result set.
8. A pre-calculation based data mining analysis apparatus, the apparatus comprising:
the data acquisition module is used for acquiring data to be mined;
the response module is used for responding to a service request of a user, pre-calculating the data to be mined according to the service request, obtaining an intermediate result and storing the intermediate result; judging whether a mining analysis result corresponding to the service request exists in a stored mining analysis result set according to the service request;
and the result acquisition module is used for carrying out mining analysis on the data to be mined according to the intermediate result when the pre-calculation is finished if not, so as to obtain mining analysis results corresponding to the service request and display the mining analysis results.
9. A computer-readable storage medium, characterized in that the storage medium stores a computer program which, when executed by a processor, implements the method of any of the preceding claims 1-7.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of the preceding claims 1-7 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311057065.5A CN117033467A (en) | 2023-08-18 | 2023-08-18 | Data mining method and device based on pre-calculation and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311057065.5A CN117033467A (en) | 2023-08-18 | 2023-08-18 | Data mining method and device based on pre-calculation and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117033467A true CN117033467A (en) | 2023-11-10 |
Family
ID=88644751
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311057065.5A Pending CN117033467A (en) | 2023-08-18 | 2023-08-18 | Data mining method and device based on pre-calculation and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117033467A (en) |
-
2023
- 2023-08-18 CN CN202311057065.5A patent/CN117033467A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8824337B1 (en) | Alternate directions in hierarchical road networks | |
| CN110262937B (en) | Identification method and device for index abnormality reasons | |
| CN110020427B (en) | Policy determination method and device | |
| CN110119860B (en) | Rubbish account detection method, device and equipment | |
| CN111080304A (en) | Credible relationship identification method, device and equipment | |
| CN116011815A (en) | Model training method and device, electronic equipment and storage medium | |
| CN116049761A (en) | Data processing method, device and equipment | |
| CN116757278B (en) | Training method and device of prediction model, storage medium and electronic equipment | |
| CN111694992A (en) | Data processing method and device | |
| CN116308738B (en) | Model training method, business wind control method and device | |
| CN113992429B (en) | Event processing method, device and equipment | |
| CN117033467A (en) | Data mining method and device based on pre-calculation and storage medium | |
| CN116501927A (en) | Graph data processing system, method, equipment and storage medium | |
| CN113902356B (en) | Regional flow data analysis method and device | |
| CN115795342B (en) | Method and device for classifying business scenes, storage medium and electronic equipment | |
| CN117171401B (en) | Query method and device for shortest path in graph data based on hierarchical pre-calculation | |
| CN107135137B (en) | Network group identification method and device | |
| CN115862675B (en) | Emotion recognition method, device, equipment and storage medium | |
| CN109325127A (en) | A kind of Risk Identification Method and device | |
| CN116341642B (en) | Data processing method and device, storage medium and electronic equipment | |
| WO2024114163A1 (en) | Resource transfer processing method and apparatus | |
| CN117909926A (en) | Risk identification method and device, storage medium and electronic equipment | |
| CN111461352B (en) | Model training method, service node identification device and electronic equipment | |
| CN116957587A (en) | Risk identification method, risk identification device, storage medium and electronic equipment | |
| CN116151620A (en) | Sub-graph matching method and device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |