CN116431692A - Data processing method, device, computer equipment and storage medium - Google Patents
Data processing method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN116431692A CN116431692A CN202310398288.1A CN202310398288A CN116431692A CN 116431692 A CN116431692 A CN 116431692A CN 202310398288 A CN202310398288 A CN 202310398288A CN 116431692 A CN116431692 A CN 116431692A
- Authority
- CN
- China
- Prior art keywords
- data
- instruction
- data acquisition
- target
- acquisition instruction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 30
- 238000000034 method Methods 0.000 claims abstract description 56
- 238000012216 screening Methods 0.000 claims abstract description 55
- 238000004590 computer program Methods 0.000 claims abstract description 24
- 238000006243 chemical reaction Methods 0.000 claims description 51
- 238000012545 processing Methods 0.000 claims description 27
- 238000010586 diagram Methods 0.000 description 11
- 230000003993 interaction Effects 0.000 description 7
- 238000013079 data visualisation Methods 0.000 description 6
- 238000001914 filtration Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000002452 interceptive effect Effects 0.000 description 4
- 238000004220 aggregation Methods 0.000 description 3
- 230000002776 aggregation Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 241001178520 Stomatepia mongo Species 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000004140 cleaning Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 229920001971 elastomer Polymers 0.000 description 1
- 239000000806 elastomer Substances 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present disclosure relates to a data processing method, apparatus, computer device, storage medium and computer program product. The method comprises the following steps: acquiring historical data acquisition instructions and data acquisition instructions, wherein the data acquisition instructions comprise keywords and data screening conditions; under the condition that a target historical data acquisition instruction which accords with a preset matching degree with the data acquisition instruction exists in the historical data acquisition instruction, acquiring a cache address corresponding to the target historical data acquisition instruction; and acquiring target data matched with the data acquisition instruction from the cache address according to the keyword and the data screening condition. By adopting the method, the data acquisition efficiency can be improved.
Description
Technical Field
The present disclosure relates to the field of big data technologies, and in particular, to a data processing method, apparatus, computer device, and storage medium.
Background
A database is a "repository" that organizes, stores, and manages data according to a data structure, and is a collection of large amounts of data that are stored in a computer for a long period of time, organized, sharable, and uniformly managed. With the development of the information age, in general, the amount of data to be stored and processed is enormous when data processing is performed.
In the traditional technology, a search engine is directly utilized to acquire required data from a database according to requirements; however, when the data is acquired by this method and the data amount in the database is large or the demand is complex, the data screening process takes a lot of time, and the data processing efficiency is low.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a data processing method, apparatus, computer device, storage medium, and computer program product that improve data acquisition efficiency.
In a first aspect, an embodiment of the present disclosure provides a data processing method. The method comprises the following steps:
acquiring historical data acquisition instructions and data acquisition instructions, wherein the data acquisition instructions comprise keywords and data screening conditions;
under the condition that a target historical data acquisition instruction which accords with a preset matching degree with the data acquisition instruction exists in the historical data acquisition instruction, acquiring a cache address corresponding to the target historical data acquisition instruction;
and acquiring target data matched with the data acquisition instruction from the cache address according to the keyword and the data screening condition.
In one embodiment, the historical data acquisition instruction includes a plurality of acquisition instructions, and when a target historical data acquisition instruction which accords with a preset matching degree with the data acquisition instruction exists in the historical data acquisition instruction, acquiring a cache address corresponding to the target historical data acquisition instruction includes:
Acquiring the matching degree between each acquisition instruction in the historical data acquisition instructions and the data acquisition instructions;
determining an acquisition instruction with the matching degree conforming to the preset matching degree as a target historical data acquisition instruction;
and obtaining a cache address corresponding to the target historical data obtaining instruction.
In one embodiment, the acquiring instruction for determining that the matching degree meets the preset matching degree is a target historical data acquiring instruction, and the method includes:
determining an acquisition instruction with the matching degree conforming to the preset matching degree as a candidate instruction;
and determining the candidate instruction with the highest matching degree as a target historical data acquisition instruction.
In one embodiment, after the acquiring the historical data acquiring instruction and the data acquiring instruction, the method further includes:
and under the condition that the historical data acquisition instruction does not have an acquisition instruction which accords with the preset matching degree with the data acquisition instruction, acquiring target data matched with the data acquisition instruction from an original database according to the keywords and the data screening conditions.
In one embodiment, after the target data matching the data acquisition instruction is acquired from the cache address, the method further includes:
Receiving a data conversion instruction, wherein the data conversion instruction comprises a data display type;
and converting the target data into data corresponding to the data display type to obtain converted target data.
In a second aspect, an embodiment of the present disclosure further provides a data processing method, where the method is applied to a client, and the method includes:
receiving a data acquisition instruction, wherein the data acquisition instruction comprises a keyword and a data screening condition;
the data acquisition instruction is sent to a server, target data matched with the data acquisition instruction and sent by the server are received, wherein the server is used for acquiring historical data acquisition instructions and data acquisition instructions, and when the historical data acquisition instructions exist in the historical data acquisition instructions and the data acquisition instructions meet the target historical data acquisition instructions with preset matching degree, a cache address corresponding to the target historical data acquisition instructions is acquired, and the target data is acquired from the cache address according to the keywords and the data screening conditions;
receiving a data conversion instruction, wherein the data conversion instruction comprises a data display type;
and sending the data conversion instruction to the server, receiving and displaying the converted target data sent by the server, wherein the server converts the target data into data corresponding to the data display type according to the data conversion instruction, and obtaining the converted target data.
In a third aspect, an embodiment of the present disclosure further provides a data processing apparatus. The device comprises:
the first acquisition module is used for acquiring historical data acquisition instructions and data acquisition instructions, wherein the data acquisition instructions comprise keywords and data screening conditions;
the second acquisition module is used for acquiring a cache address corresponding to the target historical data acquisition instruction under the condition that the target historical data acquisition instruction which accords with the preset matching degree exists in the historical data acquisition instruction;
and the third acquisition module is used for acquiring target data matched with the data acquisition instruction from the cache address according to the keyword and the data screening condition.
In one embodiment, the historical data acquisition instruction includes a plurality of acquisition instructions, and the second acquisition module includes:
the first acquisition sub-module is used for acquiring the matching degree between each acquisition instruction in the historical data acquisition instructions and the data acquisition instructions;
the first determining submodule is used for determining an acquisition instruction with the matching degree conforming to the preset matching degree as a target historical data acquisition instruction;
and the second acquisition sub-module is used for acquiring the cache address corresponding to the target historical data acquisition instruction.
In one embodiment, the first determining sub-module includes:
the first determining unit is used for determining an acquisition instruction with the matching degree conforming to the preset matching degree as a candidate instruction;
and the second determining unit is used for determining the candidate instruction with the highest matching degree as a target historical data acquisition instruction.
In one embodiment, after the first obtaining module, the method further includes:
and the third acquisition sub-module is used for acquiring target data matched with the data acquisition instruction from an original database according to the keywords and the data screening conditions under the condition that the acquisition instruction which accords with the preset matching degree with the data acquisition instruction does not exist in the historical data acquisition instruction.
In one embodiment, after the third obtaining module, the method further includes:
the receiving module is used for receiving a data conversion instruction, wherein the data conversion instruction comprises a data display type;
and the conversion module is used for converting the target data into data corresponding to the data display type to obtain converted target data.
In a fourth aspect, embodiments of the present disclosure further provide a data processing apparatus, where the apparatus is applied to a client, the apparatus includes:
The first receiving module is used for receiving a data acquisition instruction, wherein the data acquisition instruction comprises a keyword and a data screening condition;
the first sending module is used for sending the data acquisition instruction to a server and receiving target data matched with the data acquisition instruction and sent by the server, wherein the server is used for acquiring a historical data acquisition instruction and a data acquisition instruction, and acquiring a cache address corresponding to the target historical data acquisition instruction under the condition that the historical data acquisition instruction has the target historical data acquisition instruction which accords with a preset matching degree, and acquiring the target data from the cache address according to the keyword and the data screening condition;
the second receiving module is used for receiving a data conversion instruction, wherein the data conversion instruction comprises a data display type;
the second sending module is used for sending the data conversion instruction to the server, receiving and displaying the converted target data sent by the server, wherein the server converts the target data into data corresponding to the data display type according to the data conversion instruction, and obtaining the converted target data.
In a fifth aspect, embodiments of the present disclosure also provide a computer device. The computer device comprises a memory storing a computer program and a processor implementing the steps of the method of any of the embodiments of the present disclosure when the computer program is executed.
In a sixth aspect, embodiments of the present disclosure also provide a computer-readable storage medium. The computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the method of any of the embodiments of the present disclosure.
In a seventh aspect, the disclosed embodiments also provide a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of the method according to any of the embodiments of the present disclosure.
According to the embodiment of the disclosure, when data are acquired, a historical data acquisition instruction and a data acquisition instruction are acquired, under the condition that a target historical data acquisition instruction which accords with the preset matching degree exists in the historical data acquisition instruction, a cache address where the target historical data acquisition instruction is located is acquired, according to the keyword and the data screening condition in the data acquisition instruction, target data which are matched with the data acquisition instruction are acquired from the cache address, when the data are acquired, the data acquisition instruction and the historical data acquisition instruction can be matched, under the condition that the preset matching degree accords with, the target data are directly acquired from the corresponding cache address, and searching and screening of the keyword and the data screening condition are not required for a large amount of data in a database, so that the workload of data processing is greatly reduced, the data acquisition efficiency is improved, and the resource utilization rate is improved.
Drawings
FIG. 1 is a flow diagram of a data processing method in one embodiment;
FIG. 2 is a flow diagram of a data processing method in one embodiment;
FIG. 3 is a flow diagram of a data processing method in one embodiment;
FIG. 4 is a flow diagram of a data processing method in one embodiment;
FIG. 5 is a flow diagram of a data processing method in one embodiment;
FIG. 6 is a block diagram of a data processing apparatus in one embodiment;
FIG. 7 is a flow chart of a data processing method according to another embodiment;
fig. 8 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure more apparent, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the disclosed embodiments and are not intended to limit the disclosed embodiments.
In one embodiment, as shown in fig. 1, there is provided a data processing method, the method comprising:
step S110, acquiring historical data acquisition instructions and data acquisition instructions, wherein the data acquisition instructions comprise keywords and data screening conditions;
In the embodiment of the disclosure, a data acquisition instruction is acquired, wherein the data acquisition instruction comprises a keyword and a data screening condition. Specifically, the data acquisition instruction is configured to acquire the required data from the corresponding database or the cache address, and in this embodiment, the required data may be acquired by limiting the data screening conditions with keywords. Taking the method described in this embodiment as an example when applied to a data retrieval scenario, the data acquisition instruction may include a data retrieval instruction, the data retrieval instruction may include a retrieval formula, the retrieval formula may include a keyword and a data filtering condition, the data retrieval instruction may be sent by a user through a client, and the server acquires the data retrieval instruction, in one example, the user may send the data retrieval instruction through a preset search engine, and the search engine may include but is not limited to an elastomer search engine. In this embodiment, a history data obtaining instruction is further obtained, where the history data obtaining instruction may include a history data obtaining instruction corresponding to a time period from a preset history time to a current time period when the server executes the data processing method, the preset history time period may be determined according to an actual application scenario, and in a general case, the history data obtaining instruction and the data obtaining instruction have the same format, and the history data obtaining instruction includes at least one of a corresponding keyword and a data filtering condition. One or more of the historical data acquisition instructions may correspond. In one example, the data screening conditions may include, but are not limited to, query conditions, filter types, ordering methods, etc., which may include, but are not limited to, full text searches, scope searches, aggregate queries, etc., providing basic support for subsequent data visualization and interaction; the sorting method can include, but is not limited to, time sorting, frequency sorting, keyword weight sorting and the like, and the retrieved data can be arranged according to a specific order through sorting, so that ordered data support is provided for subsequent data visualization and interaction; filter types may include, but are not limited to, time filters, frequency filters, keyword filters, filtering the retrieved data to remove unnecessary data to reduce the amount of data and improve the efficiency of data visualization and interaction.
Step S120, under the condition that a target historical data acquisition instruction which accords with a preset matching degree exists in the historical data acquisition instruction, acquiring a cache address corresponding to the target historical data acquisition instruction;
in the embodiment of the disclosure, after the historical data acquisition instruction and the current data acquisition instruction are acquired, calculating the matching degree, and calculating the similarity according to the specific contents of the historical data acquisition instruction and the current data acquisition instruction, wherein the higher the similarity is, the higher the matching degree is. When the matching degree accords with the preset matching degree, the corresponding historical data acquisition instruction is determined to be a target historical data acquisition instruction, and the preset matching degree can be obtained according to actual application scene setting. In one example, the preset matching degree may be set to a matching degree corresponding to when the instructions are identical, that is, when the historical data acquisition instruction and the current data acquisition instruction are identical, determining that the historical data acquisition instruction is a target historical data acquisition instruction; in another example, the preset matching degree may be set to a matching degree corresponding to a higher similarity degree, and when the similarity degree between the historical data acquisition instruction and the current data acquisition instruction is higher, the historical data acquisition instruction is determined to be the target historical data acquisition instruction. In one possible implementation manner, when calculating the matching degree, the current data acquisition instruction includes all contents in the target historical data acquisition instruction, that is, the instruction is regarded as a set, the first set corresponding to the target historical data acquisition instruction is a subset of the second set corresponding to the current data acquisition instruction, and the more elements of the second set included in the first set, the higher the similarity of the two is, and the higher the matching degree is. After determining the target historical data acquisition instruction, acquiring a cache address corresponding to the target historical data acquisition instruction, wherein the historical data acquisition instruction is an instruction in a historical data processing process, the historical data acquisition instruction has corresponding historical target data, in this embodiment, the historical target data or a database address corresponding to the historical target data is stored in the corresponding cache address, and an association relationship exists between the cache address and the corresponding historical data acquisition instruction, and according to the historical data acquisition instruction, the cache address of the corresponding historical target data or the database address corresponding to the historical target data can be determined. The history data obtaining instruction may include a plurality of candidate history data obtaining instructions conforming to a preset matching degree, and in one example, the candidate history data obtaining instruction with the highest matching degree may be used as a target history obtaining instruction to obtain a corresponding cache address; in another example, the candidate history data fetch instruction may be directly taken as the target history fetch instruction, and a corresponding plurality of cache addresses may be fetched.
In one embodiment, the historical target data corresponding to the historical data acquisition instruction may be acquired from the original database. The target data corresponding to the current data acquisition instruction is also part or all of the original database. Wherein the raw database may include, but is not limited to, a mongo db database. When the original database is obtained, preprocessing is performed on the collected original data, including but not limited to data cleaning, data conversion, data normalization and the like, wherein the data cleaning refers to deleting incomplete or invalid data, the data conversion refers to converting the data into a uniform format, and the data normalization refers to converting the data into the same scale and range for comparison and analysis. And storing the preprocessed data into a database to obtain an original database. In some possible implementations, during storage, data may be sorted by different classifications and labels for subsequent data retrieval and ordering; in storing data, compression algorithms or the like may be used to reduce the occupation of storage space. In one example, the data in the original database is updated periodically, and the updating mode includes incremental updating and full updating, wherein the incremental updating only updates the latest data, and the full updating refers to updating all the data into the database after being collected and processed again. In another example, data backup and restoration are also required, the data is backed up to a cloud storage or a local storage, and backup and restoration tests are performed regularly to ensure the safety and reliability of the data.
And step S130, acquiring target data matched with the data acquisition instruction from the cache address according to the keyword and the data screening condition.
In the embodiment of the disclosure, according to the keyword and the data screening condition corresponding to the current data acquisition instruction, target data matched with the data acquisition instruction is acquired from the cache address, wherein the target data matched with the data acquisition instruction can be directly or indirectly acquired from the cache address, for example, when historical target data is stored in the cache address, the target data can be directly acquired from the historical target data according to the keyword and the data screening condition; when the database address corresponding to the historical target data is stored in the cache address, the target data can be obtained from the corresponding database address according to the keywords and the data screening conditions. In one example, when the target historical data acquisition instruction and the current data acquisition instruction are identical, the data in the cache address can be directly acquired as the target data, or the target data can be directly acquired from the database address cached in the cache address, and the screening by using the keywords and the data screening condition is not needed. In one example, after the target data is acquired, the target data or a database address corresponding to the target data may be cached in a cache address matched with the current data acquisition instruction for use in subsequent data processing.
According to the embodiment of the disclosure, when data are acquired, a historical data acquisition instruction and a data acquisition instruction are acquired, under the condition that a target historical data acquisition instruction which accords with the preset matching degree exists in the historical data acquisition instruction, a cache address where the target historical data acquisition instruction is located is acquired, according to the keyword and the data screening condition in the data acquisition instruction, target data which are matched with the data acquisition instruction are acquired from the cache address, when the data are acquired, the data acquisition instruction and the historical data acquisition instruction can be matched, under the condition that the preset matching degree accords with, the target data are directly acquired from the corresponding cache address, and searching and screening of the keyword and the data screening condition are not required for a large amount of data in a database, so that the workload of data processing is greatly reduced, the data acquisition efficiency is improved, and the resource utilization rate is improved.
In one embodiment, as shown in fig. 2, the historical data obtaining instruction includes a plurality of obtaining instructions, where when a target historical data obtaining instruction which accords with a preset matching degree with the data obtaining instruction exists in the historical data obtaining instruction, obtaining a cache address corresponding to the target historical data obtaining instruction includes:
Step S121, acquiring a matching degree between each acquisition instruction in the historical data acquisition instructions and the data acquisition instructions;
step S122, determining an acquisition instruction with the matching degree conforming to the preset matching degree as a target historical data acquisition instruction;
step S123, obtaining a cache address corresponding to the target historical data obtaining instruction.
In the embodiment of the disclosure, the historical data acquisition instruction includes a plurality of acquisition instructions, and after the historical data acquisition instruction and the data acquisition instruction are acquired, the matching degree between each acquisition instruction and the data acquisition instruction in the historical data acquisition instruction is determined, wherein the matching degree can be determined by calculating the similarity between the instructions, and the higher the similarity is, the higher the matching degree is. The acquisition instruction with the matching degree conforming to the preset matching degree is determined to be a target historical data acquisition instruction, the corresponding cache address is determined according to the target historical data acquisition instruction, the preset matching degree can be a proper matching degree which is obtained in advance according to the actual application scene setting, and when the matching degree between the instructions conforms to the preset matching degree, namely, the matching degree is larger than or equal to the preset matching degree, the matching degree is considered to be higher at the moment. The obtaining instructions with the matching degree conforming to the preset matching degree can include one or more, and in one example, all obtaining instructions conforming to the preset matching degree can be directly used as target historical data obtaining instructions; in another example, the acquisition instruction with the highest matching degree among the acquisition instructions with matching degrees matching with the preset matching degree may be used as the target historical data acquisition instruction.
According to the method and the device for obtaining the target historical data, when the historical data obtaining instructions comprise a plurality of obtaining instructions, the matching degree of each obtaining instruction and the current data obtaining instruction is calculated, and the target historical data obtaining instructions are determined, so that the obtained target historical data obtaining instructions are more accurate and reliable, the data searching and screening efficiency is improved, and meanwhile the accuracy of the subsequent target data obtaining is guaranteed; and the method is suitable for the situation of multiple acquisition instructions and is suitable for more application scenes.
In one embodiment, as shown in fig. 3, the acquiring instruction for determining that the matching degree meets the preset matching degree is a target historical data acquiring instruction, which includes:
step S1221, determining an acquisition instruction with a matching degree conforming to a preset matching degree as a candidate instruction;
in step S1222, the candidate instruction with the highest matching degree is determined as the target historical data acquisition instruction.
In the embodiment of the disclosure, when determining the target historical data acquisition instructions, calculating the matching degree between each acquisition instruction and the current data acquisition instruction, and determining the acquisition instruction with the matching degree conforming to the preset matching degree as the candidate instruction. The historical acquisition instructions comprise a plurality of acquisition instructions, and the acquisition instructions with the matching degree conforming to the preset matching degree can be one or more, namely the candidate instructions can be one or more. In this embodiment, a candidate instruction with the highest matching degree is determined as a target historical data acquisition instruction, and when the candidate instruction is one, the candidate instruction is the target historical data acquisition instruction; when a plurality of candidate instructions are provided, matching degrees corresponding to the plurality of candidate instructions are compared, and the candidate instruction with the highest matching degree is taken as a target historical data acquisition instruction.
According to the method and the device for obtaining the target historical data, when the target historical data obtaining instruction is determined, the candidate obtaining instruction with the highest matching degree is determined to be the target historical data obtaining instruction, the candidate instruction is further screened through the matching degree, the accuracy of the target historical data obtaining instruction is guaranteed, meanwhile, the data volume corresponding to the target historical data obtaining instruction is reduced, the data processing volume in the subsequent data query screening process is reduced, and the resource utilization rate and the data processing efficiency are improved.
In one embodiment, as shown in fig. 4, after the acquiring historical data acquiring instruction and the data acquiring instruction, further includes:
step S140, when there is no acquisition instruction conforming to the preset matching degree with the data acquisition instruction in the historical data acquisition instruction, acquiring target data matching with the data acquisition instruction from an original database according to the keyword and the data screening condition.
In this embodiment, after the historical data acquisition instruction and the data acquisition instruction are acquired, the matching degree between the historical data acquisition instruction and the data acquisition instruction is calculated, when the historical data acquisition instruction does not have the acquisition instruction which accords with the preset matching degree with the data acquisition instruction, the target data is directly acquired from the original database, and when the target data is acquired, the data in the original database is screened and filtered according to the keywords and the data screening conditions, so that the target data matched with the data acquisition instruction is obtained. In an example, the original database may be a database determined in advance according to an actual application scenario, for example, the original database described in the specific embodiment of step S120 is a target object corresponding to the historical data acquisition instruction and the current data acquisition instruction. In one example, after the target data is acquired, the target data or a database address corresponding to the target data may be cached in a cache address matched with the current data acquisition instruction for use in subsequent data processing.
According to the method and the device for acquiring the target data, when the historical data acquisition instruction is not matched with the data acquisition instruction, the target data can be acquired from the original database according to the keywords and the data screening conditions, so that the acquisition of the data in different application scenes is realized, the target data is acquired from the corresponding data storage area according to the matching degree between the instructions, and the method and the device are suitable for more application scenes.
In one embodiment, as shown in fig. 5, after the target data matching the data fetch instruction is fetched from the cache address, the method further includes:
step S150, receiving a data conversion instruction, wherein the data conversion instruction comprises a data display type;
step S160, converting the target data into data corresponding to the data display type, to obtain converted target data.
In the embodiment of the disclosure, after target data is acquired, a data conversion instruction is received, wherein the data conversion instruction includes a data display type. The data conversion instruction can include that a user sends the data through a terminal, the data display type can include but is not limited to different types of data chart types, further the data conversion instruction can further include data display parameters, and the data such as the size, the position and the like of the data chart display can be adjusted. Data diagram types may include, but are not limited to, bar diagrams, line diagrams, pie charts, and the like. Different data features and relationships may be exhibited by different chart types. And converting the target data into data corresponding to the data display type according to the data conversion instruction, and obtaining the converted data. In an example, the converted data is sent to a client, and the client displays the converted target data according to the data display type, wherein an interactive data display mode can be provided during display, a user can perform interactive operation on the data through modes of selection, filtering, sorting and the like, the change and trend of the data are known in real time, and the interactive operation mode of the user can include but is not limited to various modes such as mouse operation, gesture operation and the like. In one example, when converting, target data is first preprocessed to improve the quality and usability of the data to accommodate different chart presentations and interactions, the preprocessing may include, but is not limited to, data format conversion, data cleansing, data aggregation, and the like.
According to the embodiment of the disclosure, after the target data is obtained, the target data can be converted according to the data conversion instruction, so that the target data can be displayed according to requirements and actual application scenes, and the method and the device are applicable to more application scenes by combining data visualization and interaction.
Based on the same inventive concept, the embodiments of the present disclosure also provide a data processing apparatus for implementing the above-mentioned related data processing method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation of one or more embodiments of the data processing device provided below may refer to the limitation of the data processing method hereinabove, and will not be repeated herein.
In one embodiment, as shown in FIG. 6, there is provided a data processing apparatus 600 comprising:
a first obtaining module 610, configured to obtain a historical data obtaining instruction and a data obtaining instruction, where the data obtaining instruction includes a keyword and a data filtering condition;
a second obtaining module 620, configured to obtain, in the case where a target historical data obtaining instruction that meets a preset matching degree with the data obtaining instruction exists in the historical data obtaining instruction, a cache address corresponding to the target historical data obtaining instruction;
And a third obtaining module 630, configured to obtain, from the cache address, target data that matches the data obtaining instruction according to the keyword and the data filtering condition.
In one embodiment, the historical data acquisition instruction includes a plurality of acquisition instructions, and the second acquisition module includes:
the first acquisition sub-module is used for acquiring the matching degree between each acquisition instruction in the historical data acquisition instructions and the data acquisition instructions;
the first determining submodule is used for determining an acquisition instruction with the matching degree conforming to the preset matching degree as a target historical data acquisition instruction;
and the second acquisition sub-module is used for acquiring the cache address corresponding to the target historical data acquisition instruction.
In one embodiment, the first determining sub-module comprises:
the first determining unit is used for determining an acquisition instruction with the matching degree conforming to the preset matching degree as a candidate instruction;
and the second determining unit is used for determining the candidate instruction with the highest matching degree as a target historical data acquisition instruction.
In one embodiment, after the first obtaining module, the method further includes:
and the third acquisition sub-module is used for acquiring target data matched with the data acquisition instruction from an original database according to the keywords and the data screening conditions under the condition that the acquisition instruction which accords with the preset matching degree with the data acquisition instruction does not exist in the historical data acquisition instruction.
In one embodiment, after the third obtaining module, the method further includes:
the receiving module is used for receiving a data conversion instruction, wherein the data conversion instruction comprises a data display type;
and the conversion module is used for converting the target data into data corresponding to the data display type to obtain converted target data.
Based on the same inventive concept, the embodiments of the present disclosure further provide a data processing method, where the implementation of the solution to the problem provided by the method is similar to the implementation described in the foregoing method, so specific limitations in one or more embodiments of the data processing method provided below may be referred to the above limitations of the data processing method, which are not repeated herein. Referring to fig. 7, the method is applied to a client, and includes:
step S710, receiving a data acquisition instruction, wherein the data acquisition instruction comprises a keyword and a data screening condition;
step S720, sending the data acquisition instruction to a server, and receiving target data matched with the data acquisition instruction and sent by the server, where the server is configured to acquire a historical data acquisition instruction and a data acquisition instruction, and acquire a cache address corresponding to the target historical data acquisition instruction when there is a target historical data acquisition instruction in the historical data acquisition instruction, where the target historical data acquisition instruction meets a preset matching degree, and acquire the target data from the cache address according to the keyword and the data screening condition;
Step S730, receiving a data conversion instruction, where the data conversion instruction includes a data display type;
step S740, sending the data conversion instruction to the server, receiving the converted target data sent by the server, and displaying the converted target data, where the server converts the target data into data corresponding to the data display type according to the data conversion instruction, so as to obtain the converted target data.
In the embodiment of the disclosure, a client receives a data acquisition instruction and sends the data acquisition instruction to a server. The server receives a data acquisition instruction and acquires a historical data acquisition instruction, and under the condition that a target historical data acquisition instruction which accords with a preset matching degree exists in the historical data acquisition instruction, a cache address corresponding to the target historical data acquisition instruction is acquired, the target data is acquired from the cache address according to the keywords and the data screening condition, and the target data is sent to the client. The client receives the target data, receives a data conversion instruction, wherein the data conversion instruction comprises a display type, sends the data conversion instruction to the server, receives the data conversion instruction by the server, performs data conversion to obtain converted target data, sends the converted target data to the client, and receives and displays the converted target data by the client.
According to the embodiment of the disclosure, when data are acquired, a historical data acquisition instruction and a data acquisition instruction are acquired, under the condition that a target historical data acquisition instruction which accords with the preset matching degree exists in the historical data acquisition instruction, a cache address where the target historical data acquisition instruction is located is acquired, and according to the keyword and the data screening condition in the data acquisition instruction, target data which are matched with the data acquisition instruction are acquired from the cache address, so that when the data are acquired, the data acquisition instruction and the historical data acquisition instruction can be matched, under the condition that the preset matching degree accords with, the target data are directly acquired from the corresponding cache address, and the searching and screening of the keyword and the data screening condition are not required for a large amount of data in a database, so that the workload of data processing is greatly reduced, the data acquisition efficiency is improved, and the resource utilization rate is improved; after the target data is obtained, the target data can be converted according to the data conversion instruction, so that the target data can be displayed according to requirements and actual application scenes, and the target data is combined with data visualization and interaction, so that the method is suitable for more application scenes. The method and the device can rapidly process and analyze large-scale data sets, provide functions of real-time query, aggregation, visualization, custom development and the like, and finally present chart results meeting screening requirements.
In one embodiment, the historical data acquisition instruction includes a plurality of acquisition instructions, and when there is a target historical data acquisition instruction that matches the data acquisition instruction with a preset matching degree in the historical data acquisition instruction, acquiring a cache address corresponding to the target historical data acquisition instruction includes:
the server acquires the matching degree between each acquisition instruction in the historical data acquisition instructions and the data acquisition instructions;
the server determines an acquisition instruction with the matching degree conforming to the preset matching degree as a target historical data acquisition instruction;
and the server acquires a cache address corresponding to the target historical data acquisition instruction.
In one embodiment, the acquiring instruction for determining that the matching degree meets the preset matching degree is a target historical data acquiring instruction, and the method includes:
the server determines an acquisition instruction with the matching degree conforming to the preset matching degree as a candidate instruction;
and the server determines the candidate instruction with the highest matching degree as a target historical data acquisition instruction.
In one embodiment, after the acquiring the historical data acquiring instruction and the data acquiring instruction, further includes:
and under the condition that the historical data acquisition instruction does not have an acquisition instruction which accords with the preset matching degree with the data acquisition instruction, the server acquires target data matched with the data acquisition instruction from an original database according to the keywords and the data screening condition.
In one embodiment, a data processing method is provided by which a user can retrieve and sort data stored in a MongoDB database using an elastic search engine. Wherein the user can select different filters and ordering conditions as needed to obtain the desired data set. The data set obtained from the elastic search in the previous step is stored in the mongo db so that the data can be read more quickly at the next access. When the user selects the filter and the sorting condition with the matching degree meeting the preset matching degree, the data can be directly read from the cache. The acquired data is displayed to the user in the form of a chart, and the user can select different chart types and display modes according to the needs and perform interactive operations, such as zooming, translation, selection and the like. By storing the data returned by the elastic search in the MongoDB for caching, the data reading speed can be increased, thereby accelerating data visualization and interaction. The method can rapidly process and analyze large-scale data sets, provide functions of real-time query, aggregation, visualization, custom development and the like, and finally present chart results meeting screening requirements.
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least a portion of the steps in the figures may include steps or stages that are not necessarily performed at the same time, but may be performed at different times, nor does the order in which the steps or stages are performed necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the steps or stages in other steps.
Based on the same inventive concept, the embodiments of the present disclosure also provide a data processing apparatus for implementing the above-mentioned related data processing method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation of one or more embodiments of the data processing device provided below may refer to the limitation of the data processing method hereinabove, and will not be repeated herein.
In one embodiment, there is also provided a data processing apparatus, the apparatus being applied to a client, the apparatus comprising:
the first receiving module is used for receiving a data acquisition instruction, wherein the data acquisition instruction comprises a keyword and a data screening condition;
the first sending module is used for sending the data acquisition instruction to a server and receiving target data matched with the data acquisition instruction and sent by the server, wherein the server is used for acquiring a historical data acquisition instruction and a data acquisition instruction, and acquiring a cache address corresponding to the target historical data acquisition instruction under the condition that the historical data acquisition instruction has the target historical data acquisition instruction which accords with a preset matching degree, and acquiring the target data from the cache address according to the keyword and the data screening condition;
The second receiving module is used for receiving a data conversion instruction, wherein the data conversion instruction comprises a data display type;
the second sending module is used for sending the data conversion instruction to the server, receiving and displaying the converted target data sent by the server, wherein the server converts the target data into data corresponding to the data display type according to the data conversion instruction, and obtaining the converted target data.
Each of the modules in the above-described data processing apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a server, and the internal structure of which may be as shown in fig. 8. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer equipment is used for storing data related in the data processing process of data acquisition instructions, target data and the like. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data processing method.
It will be appreciated by those skilled in the art that the structure shown in fig. 8 is merely a block diagram of a portion of the structure associated with an embodiment of the present disclosure and is not limiting of the computer device to which an embodiment of the present disclosure may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In an embodiment, there is also provided a computer device comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, carries out the steps of the method embodiments described above.
In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.
It should be noted that, the user information (including, but not limited to, user equipment information, user personal information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) according to the embodiments of the present disclosure are information and data authorized by the user or sufficiently authorized by each party.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided by the present disclosure may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the embodiments provided by the present disclosure may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided in the present disclosure may be general-purpose processors, central processing units, graphic processors, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like, without being limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples merely represent a few implementations of the disclosed embodiments, which are described in more detail and are not to be construed as limiting the scope of the disclosed embodiments. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made to the disclosed embodiments without departing from the spirit of the disclosed embodiments. Accordingly, the scope of the disclosed embodiments should be determined from the following claims.
Claims (11)
1. A method of data processing, the method comprising:
acquiring historical data acquisition instructions and data acquisition instructions, wherein the data acquisition instructions comprise keywords and data screening conditions;
under the condition that a target historical data acquisition instruction which accords with a preset matching degree with the data acquisition instruction exists in the historical data acquisition instruction, acquiring a cache address corresponding to the target historical data acquisition instruction;
And acquiring target data matched with the data acquisition instruction from the cache address according to the keyword and the data screening condition.
2. The method according to claim 1, wherein the history data obtaining instruction includes a plurality of obtaining instructions, and when there is a target history data obtaining instruction that matches the data obtaining instruction with a preset matching degree in the history data obtaining instruction, obtaining a cache address corresponding to the target history data obtaining instruction includes:
acquiring the matching degree between each acquisition instruction in the historical data acquisition instructions and the data acquisition instructions;
determining an acquisition instruction with the matching degree conforming to the preset matching degree as a target historical data acquisition instruction;
and obtaining a cache address corresponding to the target historical data obtaining instruction.
3. The method according to claim 2, wherein the acquisition instruction for determining that the matching degree meets the preset matching degree is a target historical data acquisition instruction, including:
determining an acquisition instruction with the matching degree conforming to the preset matching degree as a candidate instruction;
and determining the candidate instruction with the highest matching degree as a target historical data acquisition instruction.
4. The method of claim 1, further comprising, after the acquiring historical data acquisition instructions and data acquisition instructions:
and under the condition that the historical data acquisition instruction does not have an acquisition instruction which accords with the preset matching degree with the data acquisition instruction, acquiring target data matched with the data acquisition instruction from an original database according to the keywords and the data screening conditions.
5. The method of claim 1, further comprising, after said fetching target data from said cache address that matches said data fetch instruction:
receiving a data conversion instruction, wherein the data conversion instruction comprises a data display type;
and converting the target data into data corresponding to the data display type to obtain converted target data.
6. A method of data processing, the method being applied to a client, the method comprising:
receiving a data acquisition instruction, wherein the data acquisition instruction comprises a keyword and a data screening condition;
the data acquisition instruction is sent to a server, target data matched with the data acquisition instruction and sent by the server are received, wherein the server is used for acquiring historical data acquisition instructions and data acquisition instructions, and when the historical data acquisition instructions exist in the historical data acquisition instructions and the data acquisition instructions meet the target historical data acquisition instructions with preset matching degree, a cache address corresponding to the target historical data acquisition instructions is acquired, and the target data is acquired from the cache address according to the keywords and the data screening conditions;
Receiving a data conversion instruction, wherein the data conversion instruction comprises a data display type;
and sending the data conversion instruction to the server, receiving and displaying the converted target data sent by the server, wherein the server converts the target data into data corresponding to the data display type according to the data conversion instruction, and obtaining the converted target data.
7. A data processing apparatus, the apparatus comprising:
the first acquisition module is used for acquiring historical data acquisition instructions and data acquisition instructions, wherein the data acquisition instructions comprise keywords and data screening conditions;
the second acquisition module is used for acquiring a cache address corresponding to the target historical data acquisition instruction under the condition that the target historical data acquisition instruction which accords with the preset matching degree exists in the historical data acquisition instruction;
and the third acquisition module is used for acquiring target data matched with the data acquisition instruction from the cache address according to the keyword and the data screening condition.
8. A data processing apparatus, the apparatus being applied to a client, the apparatus comprising:
The first receiving module is used for receiving a data acquisition instruction, wherein the data acquisition instruction comprises a keyword and a data screening condition;
the first sending module is used for sending the data acquisition instruction to a server and receiving target data matched with the data acquisition instruction and sent by the server, wherein the server is used for acquiring a historical data acquisition instruction and a data acquisition instruction, and acquiring a cache address corresponding to the target historical data acquisition instruction under the condition that the historical data acquisition instruction has the target historical data acquisition instruction which accords with a preset matching degree, and acquiring the target data from the cache address according to the keyword and the data screening condition;
the second receiving module is used for receiving a data conversion instruction, wherein the data conversion instruction comprises a data display type;
the second sending module is used for sending the data conversion instruction to the server, receiving and displaying the converted target data sent by the server, wherein the server converts the target data into data corresponding to the data display type according to the data conversion instruction, and obtaining the converted target data.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the data processing method of any one of claims 1 to 5 or the steps of the data processing method of claim 6 when the computer program is executed.
10. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the data processing method of any one of claims 1 to 5, or the steps of the data processing method of claim 6.
11. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the data processing method of any one of claims 1 to 5 or the steps of the data processing method of claim 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310398288.1A CN116431692A (en) | 2023-04-14 | 2023-04-14 | Data processing method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310398288.1A CN116431692A (en) | 2023-04-14 | 2023-04-14 | Data processing method, device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116431692A true CN116431692A (en) | 2023-07-14 |
Family
ID=87093981
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310398288.1A Pending CN116431692A (en) | 2023-04-14 | 2023-04-14 | Data processing method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116431692A (en) |
-
2023
- 2023-04-14 CN CN202310398288.1A patent/CN116431692A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11074560B2 (en) | Tracking processed machine data | |
| KR101732754B1 (en) | Content-based image search | |
| US9336314B2 (en) | Dynamic facet ordering for faceted search | |
| US9563820B2 (en) | Presentation and organization of content | |
| US20160328432A1 (en) | System and method for management of time series data sets | |
| US7353218B2 (en) | Methods and apparatus for clustering evolving data streams through online and offline components | |
| CN107861981B (en) | Data processing method and device | |
| US9846697B2 (en) | Indexing data updates associated with an electronic catalog system | |
| CN112527843B (en) | Data query method, device, terminal equipment and storage medium | |
| CN112988863A (en) | Elasticissearch-based efficient search engine method for heterogeneous multiple data sources | |
| US20230409588A1 (en) | System and method for subset searching and associated search operators | |
| CN111090705B (en) | Multidimensional data processing method, device and equipment and storage medium | |
| US10176202B1 (en) | Methods and systems for content-based image retrieval | |
| US11630854B2 (en) | Segmenting users with sparse data utilizing hash partitions | |
| WO2017095439A1 (en) | Incremental clustering of a data stream via an orthogonal transform based indexing | |
| WO2017095413A1 (en) | Incremental automatic update of ranked neighbor lists based on k-th nearest neighbors | |
| CN114817717A (en) | Search method, search device, computer equipment and storage medium | |
| CN107430633B (en) | System and method for data storage and computer readable medium | |
| WO2017065795A1 (en) | Incremental update of a neighbor graph via an orthogonal transform based indexing | |
| CN112115281A (en) | Data retrieval method, device and storage medium | |
| CN110399396B (en) | Efficient data processing | |
| CN116431692A (en) | Data processing method, device, computer equipment and storage medium | |
| CN116450890A (en) | Graph data processing method, device and system, electronic equipment and storage medium | |
| CA2703132A1 (en) | Methods and system for information storage enabling fast information retrieval | |
| CN111382342B (en) | Method, device and equipment for acquiring hot search words and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |