CN113407609A - External data using method, device and equipment - Google Patents
External data using method, device and equipment Download PDFInfo
- Publication number
- CN113407609A CN113407609A CN202110728706.XA CN202110728706A CN113407609A CN 113407609 A CN113407609 A CN 113407609A CN 202110728706 A CN202110728706 A CN 202110728706A CN 113407609 A CN113407609 A CN 113407609A
- Authority
- CN
- China
- Prior art keywords
- data
- field
- target
- external data
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 81
- 238000006243 chemical reaction Methods 0.000 claims abstract description 10
- 238000004590 computer program Methods 0.000 claims description 17
- 230000008569 process Effects 0.000 description 14
- 238000007405 data analysis Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 11
- 238000012545 processing Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 9
- 230000010354 integration Effects 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 5
- 238000007726 management method Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000011835 investigation Methods 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 101100136092 Drosophila melanogaster peng gene Proteins 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 239000013589 supplement Substances 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003032 molecular docking Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001502 supplementing effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides an external data using method, device and equipment. The method comprises the following steps: the server can acquire external data from the front-end system according to a preset frequency. The server can perform format conversion on the external data to generate message data in a preset format. And the server updates the target data according to the message data. And the server disassembles the field contents corresponding to each target field from the target data according to the target field set. The server can store the field content corresponding to each target field in the target field set in the storage unit. When the server obtains the query request sent by the user from the data retrieval platform, the server can obtain the field content corresponding to the field to be queried from the storage unit according to the field to be queried included in the query request. The method reduces the data volume of the data to be retrieved, and improves the retrieval efficiency and the use efficiency of the external data.
Description
Technical Field
The present application relates to the field of computers, and in particular, to a method, an apparatus, and a device for using external data.
Background
As IT era shifts to DT era, the value of data is more and more emphasized. In order to expand data dimensionality, external data is added in the data analysis process, and the method becomes an important data supplement link. For external data, how to effectively complete unified management of the external data, establish a perfect processing chain and complete and flexible data service becomes an important part in the data planning process.
External data is typically voluminous and unstructured. At present, the application of external data mainly comprises three steps of data access, data integration and data analysis. The data access comprises the step of obtaining a message of external data, and then analyzing the message to obtain the external data. And the data integration comprises the step of arranging the external data obtained by analysis into a standard format and storing the standard format in a database. The data analysis comprises reading external data from a database for analysis.
Although the application method of the external data solves the problem of storage of the external data, the application method of the external data still has the problem of low use efficiency of the external data.
Disclosure of Invention
The application provides an external data using method, device and equipment, which are used for solving the problem of low external data using efficiency.
In a first aspect, the present application provides an external data using method, including:
acquiring external data at a preset frequency, and determining target data according to the external data;
according to the target data and the target field set, field contents corresponding to each target field in the target field set are obtained through disassembling, and the target field set comprises field names of at least one field to be inquired;
and storing the field content corresponding to each target field in the target field set into a storage unit, so that the storage unit returns the corresponding field content when receiving a query request.
Optionally, the determining target data according to the external data specifically includes:
carrying out format conversion on the external data to generate message data in a preset format;
and updating the target data according to the message data.
Optionally, the updating the target data according to the message data specifically includes:
classifying the message data according to preset data categories to obtain different types of message data;
and updating the different types of target data according to the different types of message data.
Optionally, the method further comprises:
and storing the external data into a storage unit so that the storage unit returns corresponding external data when receiving the query request.
Optionally, when the target field is a subscription field, the method further includes:
generating subscription information according to the target field and the field content corresponding to the target field;
and sending the subscription information.
Optionally, the method further comprises:
acquiring a query record, wherein the query record comprises a field name of a field to be queried;
and updating the target field set according to the query record so as to increase the target fields in the target field set.
In a second aspect, the present application provides an external data usage device, comprising:
the acquisition module is used for acquiring external data at a preset frequency and determining target data according to the external data;
a disassembling module, configured to disassemble to obtain field content corresponding to each target field in a target field set according to the target data and the target field set, where the target field set includes a field name of at least one field to be queried;
and the first storage module is used for storing the field contents corresponding to each target field in the target field set into a storage unit so that the storage unit returns the corresponding field contents when receiving the query request.
Optionally, the obtaining module specifically includes:
the generating submodule is used for carrying out format conversion on the external data and generating message data in a preset format;
and the updating submodule is used for updating the target data according to the message data.
Optionally, the update sub-module is specifically configured to classify the packet data according to a preset data category to obtain different types of packet data; and updating the different types of target data according to the different types of message data.
Optionally, the apparatus further comprises:
and the second storage module is used for storing the external data into the storage unit so that the storage unit returns corresponding external data when receiving the query request.
Optionally, when the target field is a subscription field, the apparatus further includes:
the subscription module is used for generating subscription information according to the target field and the field content corresponding to the target field; and sending the subscription information.
Optionally, the apparatus further comprises:
the target field updating module is used for acquiring a query record, wherein the query record comprises a field name of a field to be queried; and updating the target field according to the query record so as to increase the field name in the target field.
In a third aspect, the present application provides a server, comprising: a memory and a processor;
the memory is used for storing program instructions; the processor is arranged to invoke program instructions in the memory to perform the external data usage method of the first aspect and any one of the possible designs of the first aspect.
In a fourth aspect, the present application provides a readable storage medium, in which execution instructions are stored, and when the execution instructions are executed by at least one processor of the server, the server executes the external data using method in any one of the possible designs of the first aspect and the first aspect.
In a fifth aspect, the present application provides a computer program product comprising a computer program that, when executed by a processor, implements the method of external data usage in the first aspect and any one of the possible designs of the first aspect.
According to the external data using method, external data are obtained from a front-end system according to a preset frequency; carrying out format conversion on the external data to generate message data in a preset format; updating target data according to the message data; according to the target field set, field contents corresponding to all target fields are disassembled from target data; storing the field content corresponding to each target field in the target field set into a storage unit; when the query request sent by the user is obtained from the data retrieval platform, the field content corresponding to the field to be queried is obtained from the storage unit according to the field to be queried included in the query request, so that the retrieval efficiency of the user on the field in the external data is improved, the use efficiency of the external data is improved, and the practicability effect of the external data is improved.
Drawings
In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic diagram of an external data utilization system according to an embodiment of the present application;
fig. 2 is a schematic diagram of snapshot generation according to an embodiment of the present application;
FIG. 3 is a flowchart of a method for using external data according to an embodiment of the present application;
FIG. 4 is a flow chart of another external data using method provided by an embodiment of the present application;
FIG. 5 is a flow chart of another external data using method according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of an external data utilization device according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of another external data utilization device according to an embodiment of the present application;
FIG. 8 is a schematic structural diagram of another external data utilization device according to an embodiment of the present application;
fig. 9 is a schematic hardware structure diagram of a server according to an embodiment of the present application.
Detailed Description
To make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be clearly and completely described below with reference to the drawings in the present application, and it is obvious that the described embodiments are some, but not all embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context. Also, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context indicates otherwise. The terms "or" and/or "as used herein are to be construed as inclusive or meaning any one or any combination. Thus, "A, B or C" or "A, B and/or C" means "any of the following: a; b; c; a and B; a and C; b and C; A. b and C ". An exception to this definition will occur only when a combination of elements, functions, steps or operations are inherently mutually exclusive in some way. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
As IT era shifts to DT era, the value of data is more and more emphasized. In order to expand data dimensionality, external data is added in the data analysis process, and the method becomes an important data supplement link. For external data, how to effectively complete unified management of the external data, establish a perfect processing chain and complete and flexible data service becomes an important part in the data planning process.
During the data analysis, external data is applied as auxiliary data in the data analysis. External data generally refers to data from third parties, not generated by the company itself. For example, the third party data of the banking system may include credit, industry and business, court, etc. The external data is mainly used for supplementing the internal data and increasing data dimensionality. For example, credit assessment data is used to assess the level of client loan, effective forewarning of the client using court data, etc.
Currently, external data is usually characterized by large volume and unstructured. The application of the external data mainly comprises three steps of data access, data integration and data analysis. The data access comprises the step of obtaining a message of external data, and then analyzing the message to obtain the external data. And the data integration comprises the step of arranging the external data obtained by analysis into a standard format and storing the standard format in a database. The data analysis comprises reading external data from a database for analysis. The above process realizes unified management of external data. However, in the above data analysis process, data needs to be read from the database first, and then data analysis is performed. The efficiency of the use of this data analysis process is limited by database storage and computational power. In the analysis process, data calculation is limited by SQL syntax, and data service is single.
Therefore, the problem of single data analysis service exists in the prior art. Moreover, how to establish a perfect processing chain and complete and flexible data service and realize the efficient, practical and flexible processing of external data becomes a technical problem to be solved urgently.
In order to solve the above problems, the present application provides an external data using method. In the application, the server uses sqoop to periodically extract external data stored in the database of the front-end system, so as to realize data access. The external data is third-party data acquired by the front-end system in real time. After the server acquires the external data, the server processes the message data by using two modes. In one mode, the server will perform data integration, processing, and structured mapping based on Spark to obtain the target data. In another mode, external message data is directly imported into the HBase. For the target data, after determining the target field, the server may disassemble the target data to obtain the field content corresponding to each field according to the target field. Wherein the target field may be determined by the server according to the type of the data service. For example, when the data service type is a data retrieval service, the target field may include a field that the user may query. For another example, when the data service type is a data subscription service, the target field may be a field selected by the user. Aiming at external data directly imported into HBase, the server provides K-V query of unstructured data, and flexible and effective data retrieval service is achieved. In addition, the server can analyze the external data through a machine learning technology, so that scheduling analysis service and abnormal alarm service are realized.
The Sqoop is an open source big data component and is mainly used for data transmission between hadoop and a traditional database.
The Spark is an open source big data calculation engine and supports interactive calculation and complex algorithms. When the Spark engine is used to complete various operations, the intermediate result can be saved in the memory. The data storage mode can improve the efficiency of iterative operation. Also, operations performed by the Spark engine may include SQL queries, text processing, machine learning, and the like.
Among them, Hbase is a columnar storage database, suitable for unstructured data storage.
Among them, Elastic Search is a Lucene-based Search engine, and provides data retrieval capability through Restful API interface.
The technical solution of the present application will be described in detail below with specific examples. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments.
Fig. 1 is a schematic diagram illustrating an external data usage system according to an embodiment of the present application. As shown in fig. 1, the external data using system of the present application is connected to a front-end system for acquiring external data from the front-end system. The front-end system comprises a database (Data Base, DB). After the front system acquires the external data, the external data is stored in the DB. The external data use system shown in the present application uses sqoop to periodically extract external data from the DB. The external data use System realizes processing of external data and service docking based on a Hadoop Distributed File System (HDFS). The HDFS acquires external data extracted by Sqoop. The external data is message data in a text format or a binary format. The server will then process the data according to the service type.
Firstly, aiming at field data related services, the HDFS completes integration, processing and structural mapping of data based on Spark to obtain a Json format snapshot. The server may use the subscription management function to obtain the subscription field for each user and determine that the subscription field is a target field. HDFS may obtain the field contents of the target field from the snapshot in Json format. The server may issue the field content to the corresponding user terminal in batch through a File Transfer Protocol (FTP). The server may also import the Json formatted snapshot into Elastic Search (ES) using Spark. The server may generate a corresponding index in the ES. The server may also obtain retrieval instructions from the data retrieval platform. The retrieve instruction is to retrieve field contents of one or more fields from a Json format snapshot of ES.
Secondly, aiming at the real-time query requirement, the server can directly lead external data into HBase through bulk load. The server may provide the user with a Key-Value query service for the external data. For example, when the Key is time, the server may query for historical data of a certain enterprise at the time according to the time.
The specific process of the HDFS completing data integration, processing and structured mapping based on Spark can be as shown in fig. 2. The data of the source layer is external data obtained by the HDFS from DB directly through sqoop. The format of the external data is a text format or a binary format. The external data may include credit data, business data, court data, and the like. The pedestrian credit investigation and the Penta credit investigation shown in FIG. 2 are two external data from different sources in the credit investigation data. The Sqoop periodically acquires these external data from the DB. In the example shown in fig. 2, the interval time for sqoop to acquire external data is T. After acquiring the data of the person credit and the Peng element credit acquired by sqoop from the DB, Spark compares the data of the person credit and the Peng element credit acquired before T day, and determines the newly added data. And the Sqark integrates newly added data of the pedestrian credit and the Pengcheng credit to obtain T-day increment data. And Sqark acquires the snapshot acquired before the T day, and determines the snapshot to be the snapshot of the T-1 day. And the Sqark merges the data of the T-day increment into the T-1-day snapshot to obtain the T-day snapshot. The data in the snapshot of the T day is snapshot data obtained after the data in the snapshot of the T-1 day is covered by the data of the increment of the T day. The T-day increment data can cover part or all of the data in the T-1-day snapshot. The snapshot of the day T may further include auxiliary information such as a data type, a warehousing time, a data version, and the like. The auxiliary information is added to the snapshot by the server when the T-day snapshot is generated.
In the present application, a server is used as an execution agent to execute the external data using method of the following embodiments. Specifically, the execution body may be a hardware device of the server, or a software application in the server, or a computer-readable storage medium on which the software application implementing the following embodiment is installed, or code of the software application implementing the following embodiment.
Fig. 3 shows a flowchart of an external data using method according to an embodiment of the present application. On the basis of the embodiments shown in fig. 1 and fig. 2, as shown in fig. 3, with a server as an execution subject, the method of the embodiment may include the following steps:
s101, acquiring external data at a preset frequency, and determining target data according to the external data.
In this embodiment, the server may obtain the external data from the front-end system according to the preset frequency. The preset frequency may be determined according to the actual data amount, and the preset frequency is the time interval T in fig. 2. For example, once a day, once a week, once a month, etc. The front-end system may store all external data. Or, the front-end system can delete the external data after uploading the external data to the server each time. Alternatively, the front-end system may delete the external data after uploading the external data to the server and retaining the external data in the front-end system for a certain time. The external data acquired by the server may include all external data in the front-end system. Alternatively, the external data acquired by the server may include the external data acquired by the head-end system in the time interval T.
And after the server acquires the external data, determining target data according to the external data. The server may specifically determine the target data according to the external data, and the determining may include:
In this step, the server may perform format conversion on the external data to unify formats of the external data. The format of the external data may include a text format and a binary format. The preset format may be a Json format. The server can obtain data content from external data according to the preset Json format. And the server stores the acquired data content into the message data in the Json format. For example, the server may obtain a company name from external data and store the company name in the message data in the Json format.
For different kinds of external data, the preset format may include different contents. For example, when the external data is credit data, the message data in the preset format may include fields such as name and certificate type. When the external data is business information, the message data in the preset format may include fields such as a company name and a unified social credit code.
When the external data only includes the T-day new addition, the generated message data in the preset format may include the field contents of some or all of the fields. The field content appearing in the message data is the field content updated within the T day. Otherwise, the field content appearing in the message data may be the field content updated within the T day, and may also be the field content updated before the T day.
And 2, updating the target data according to the message data.
In this step, the server updates the target data according to the message data. The target data before updating is the snapshot of day T-1 in fig. 2. The target data before the update is the target data generated last time. The format of the message data is the same as the format of the target data before the update. And when the message data is newly added for T days, the server uses the field in the message data to cover the field in the target data before updating. Otherwise, the server uses the field of the message data different from the target data before updating to cover the field in the target data before updating. And further obtaining updated target data.
Different contents may be included in the preset format thereof due to different kinds of external data. And aiming at different preset formats, the generated message data and the target data are different. Therefore, the server can update the target data according to different types of external data, message data and target data by using the following specific steps:
and 2.1, classifying the message data according to the preset data category to obtain different types of message data.
In this step, the server may classify the external data according to a preset data category. The server may determine a preset format of the external data according to the preset data category. The server can generate the corresponding message data according to the preset format and the external data.
And 2.2, updating different types of target data according to different types of message data.
In this step, the server may determine the data type of the target data corresponding to the data type of the message data. Further, the server determines the corresponding target data before updating according to the data type. And the server updates the target data before updating by using the message data to obtain the updated target data.
S102, according to the target data and the target field set, field contents corresponding to each target field in the target field set are obtained through disassembling, and the target field set comprises the field name of at least one field to be inquired.
In this embodiment, the server may determine the target field set according to the retrieval content of the user. That is, each target field in the set of target fields is a field that the user may retrieve. And the server disassembles the field contents corresponding to each target field from the target data according to the target field set. The format of the target field may be a Json format. The field content corresponding to the disassembled target field may be a value of data with a name consistent with the target field in the Json format.
S103, storing the field content corresponding to each target field in the target field set into the storage unit, so that the storage unit returns the corresponding field content when receiving the query request.
In this embodiment, the server may store, in the storage unit, field contents corresponding to each target field in the target field set. The storage unit may be the storage unit of the data retrieval platform shown in fig. 1. When the server obtains the query request sent by the user from the data retrieval platform, the server can obtain the field content corresponding to the field to be queried from the storage unit according to the field to be queried included in the query request. The field content corresponding to the field to be inquired is fed back to the user terminal through the data retrieval platform.
According to the external data using method, the server can acquire the external data from the front-end system according to the preset frequency. The server can perform format conversion on the external data to generate message data in a preset format. And the server updates the target data according to the message data. And the server disassembles the field contents corresponding to each target field from the target data according to the target field set. The server can store the field content corresponding to each target field in the target field set in the storage unit. When the server obtains the query request sent by the user from the data retrieval platform, the server can obtain the field content corresponding to the field to be queried from the storage unit according to the field to be queried included in the query request. According to the method and the device, the target field in the external data is stored in the storage unit, so that the data volume of the data to be retrieved is reduced, and the retrieval efficiency and the use efficiency of the external data are improved.
Fig. 4 is a flowchart illustrating another external data using method according to an embodiment of the present application. On the basis of the embodiments shown in fig. 1 to fig. 3, as shown in fig. 4, with a server as an execution subject, the method of the embodiment may include the following steps:
s201, acquiring external data at a preset frequency, and determining target data according to the external data.
S202, according to the target data and the target field set, field contents corresponding to all target fields in the target field set are obtained through disassembling, and the target field set comprises field names of at least one field to be inquired.
S203, storing the field content corresponding to each target field in the target field set into the storage unit, so that the storage unit returns the corresponding field content when receiving the query request.
Steps S201 to S203 are similar to steps S101 to S103 in the embodiment of fig. 2, and are not described again in this embodiment.
S204, obtaining a query record, wherein the query record comprises the field name of the field to be queried.
In this embodiment, the server may obtain the query record of the user on the data retrieval platform. The query record comprises a query request and a field to be queried in the query request.
And S205, updating the target field set according to the query record so as to increase the target fields in the target field set.
In this embodiment, the server may perform statistics on the field to be queried in the query record. When a field that is not present in the target field set appears in the query record, the server may add the field as a new target field in the target field set.
According to the external data using method, the server can acquire the external data from the front-end system according to the preset frequency. The server can perform format conversion on the external data to generate message data in a preset format. And the server updates the target data according to the message data. And the server disassembles the field contents corresponding to each target field from the target data according to the target field set. The server can store the field content corresponding to each target field in the target field set in the storage unit. The server can obtain the query records of the user on the data retrieval platform. The server may count the fields to be queried in the query record. The server updates the target field set according to the query record to add the target fields in the target field set. In the application, the field content stored in the storage unit is updated by updating the target field set, so that the practicability of data in the storage unit is ensured, and on the basis of ensuring the retrieval success rate of a user, the external data is stored in the storage unit as little as possible, so that the database retrieval efficiency is improved.
Fig. 5 is a flowchart illustrating a further external data using method according to an embodiment of the present application. On the basis of the embodiments shown in fig. 1 to 4, as shown in fig. 5, with a server as an execution subject, the method of this embodiment may include the following steps:
s301, acquiring external data at a preset frequency, and determining target data according to the external data.
Step S301 is similar to the step S101 in the embodiment of fig. 2, and this embodiment is not described herein again.
S302, storing the external data into the storage unit so that the storage unit returns corresponding external data when receiving the query request.
In this embodiment, the server may directly store the external data in the storage unit. The storage unit may be a storage unit corresponding to the message query service shown in fig. 1. After the external data is directly stored in the storage unit, the server can directly feed back the corresponding external data to the user terminal by acquiring the query request. For example, when the query request is used to query for business information, a company name may be included in the query request. The server may acquire all external data related to the company name from the storage unit according to the company name. All external data includes the data of the company at each time point. The server feeds back the external data to the user terminal.
And S303, according to the target data and the subscription field set, disassembling to obtain field contents corresponding to each subscription field in the subscription field set.
In this embodiment, the target field set in the server may be a subscription field set. The subscription fields included in the set of subscription fields may be determined according to user selection. The user can select the fields which the user wants to view at the subscription platform, and a subscription field set is generated. And the server disassembles the field content corresponding to each subscription field in the subscription field set from the target data according to the subscription field set.
S304, generating subscription information according to the field content corresponding to each subscription field in the subscription field set.
In this embodiment, the server may integrate the field contents corresponding to each subscription field in the subscription field set into an output text. The output text is the subscription information. The subscription information is sent to the user terminal for the user to view the field content of his subscription.
S305, subscription information is sent.
In this embodiment, the server may send the subscription information to the user terminal through an FTP protocol. The data format of the subscription field and the field content in the subscription information may be a text format.
According to the external data using method, the server can acquire the external data at the preset frequency and determine the target data according to the external data. The server may directly store the external data in the storage unit. The server can directly feed back the corresponding external data to the user terminal by acquiring the query request. And the server disassembles the field content corresponding to each subscription field in the subscription field set according to the target data and the subscription field set. The server may generate the subscription information according to the field content corresponding to each subscription field in the subscription field set. The server may transmit the subscription information to the user terminal through the FTP protocol. In the application, the server applies the external data to the data services of multiple differences through the multiple query modes, the problem of single external data service is solved, and the practicability and the use efficiency of the external data are improved.
Fig. 6 is a schematic structural diagram of an external data using apparatus according to an embodiment of the present application, and as shown in fig. 6, an external data using apparatus 10 according to the present embodiment is used to implement an operation corresponding to a server in any one of the method embodiments described above, where the external data using apparatus 10 according to the present embodiment includes:
and the acquisition module 11 is configured to acquire external data at a preset frequency and determine target data according to the external data.
And a disassembling module 12, configured to disassemble to obtain field content corresponding to each target field in the target field set according to the target data and the target field set, where the target field set includes a field name of at least one field to be queried.
The first saving module 13 is configured to save field contents corresponding to each target field in the target field set into the storage unit, so that the storage unit returns corresponding field contents when receiving the query request.
The external data using apparatus 10 provided in the embodiment of the present application can execute the above method embodiment, and specific implementation principles and technical effects thereof can be referred to the above method embodiment, which is not described herein again.
Fig. 7 is a schematic structural diagram of another external data using device according to an embodiment of the present application, and based on the embodiment shown in fig. 6, as shown in fig. 7, an external data using device 10 according to this embodiment is used to implement operations corresponding to a server in any one of the method embodiments described above, where an obtaining module 11 according to this embodiment includes:
the generating sub-module 111 is configured to perform format conversion on the external data to generate message data in a preset format.
And an updating submodule 112, configured to update the target data according to the message data.
In an example, the updating sub-module 112 is specifically configured to classify the message data according to a preset data category to obtain different types of message data. And updating different types of target data according to different types of message data.
The external data using apparatus 10 provided in the embodiment of the present application can execute the above method embodiment, and specific implementation principles and technical effects thereof can be referred to the above method embodiment, which is not described herein again.
Fig. 8 is a schematic structural diagram of another external data using device according to an embodiment of the present application, and based on the embodiments shown in fig. 6 and 7, as shown in fig. 8, an external data using device 10 according to this embodiment is used to implement operations corresponding to a server in any of the method embodiments described above, where the external data using device 10 according to this embodiment includes:
and the second saving module 14 is configured to save the external data into the storage unit, so that the storage unit returns corresponding external data when receiving the query request.
And the subscription module 15 is configured to generate subscription information according to the target field and the field content corresponding to the target field. And sending subscription information.
And the target field updating module 16 is configured to obtain a query record, where the query record includes a field name of a field to be queried. The target field set is updated according to the query record to add the target fields in the target field set.
The external data using apparatus 10 provided in the embodiment of the present application can execute the above method embodiment, and specific implementation principles and technical effects thereof can be referred to the above method embodiment, which is not described herein again.
Fig. 9 shows a hardware structure diagram of a server according to an embodiment of the present application. As shown in fig. 9, the server 20 is configured to implement the operation corresponding to the server in any of the above method embodiments, where the server 20 of this embodiment may include: memory 21, processor 22 and communication interface 24.
A memory 21 for storing a computer program. The Memory 21 may include a Random Access Memory (RAM), a Non-Volatile Memory (NVM), at least one disk Memory, a usb disk, a removable hard disk, a read-only Memory, a magnetic disk or an optical disk.
A processor 22 for executing the computer program stored in the memory to implement the external data using method in the above-described embodiments. Reference may be made in particular to the description relating to the method embodiments described above. The Processor 22 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present invention may be embodied directly in a hardware processor, or in a combination of the hardware and software modules within the processor.
Alternatively, the memory 21 may be separate or integrated with the processor 22.
When memory 21 is a separate device from processor 22, server 20 may also include bus 23. The bus 23 is used to connect the memory 21 and the processor 22. The bus 23 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, the buses in the figures of the present application are not limited to only one bus or one type of bus.
The communication interface 24 may be connected to the processor 21 via a bus 23. The communication interface 24 is used to acquire external data and transmit the external data to the processor to implement the above-described external data usage method. The communication interface 24 is also used for acquiring the query request sent by the user terminal and feeding back the query result to the user terminal.
The server provided in this embodiment may be used to execute the external data using method, and the implementation manner and the technical effect are similar, which are not described herein again.
The present application also provides a computer-readable storage medium, in which a computer program is stored, and the computer program is used for implementing the methods provided by the above-mentioned various embodiments when being executed by a processor.
The computer-readable storage medium may be a computer storage medium or a communication medium. Communication media includes any medium that facilitates transfer of a computer program from one place to another. Computer storage media may be any available media that can be accessed by a general purpose or special purpose computer. For example, a computer readable storage medium is coupled to the processor such that the processor can read information from, and write information to, the computer readable storage medium. Of course, the computer readable storage medium may also be integral to the processor. The processor and the computer-readable storage medium may reside in an Application Specific Integrated Circuit (ASIC). Additionally, the ASIC may reside in user equipment. Of course, the processor and the computer-readable storage medium may also reside as discrete components in a communication device.
In particular, the computer-readable storage medium may be implemented by any type of volatile or non-volatile Memory device or combination thereof, such as Static Random-Access Memory (SRAM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk, or optical disk. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.
The present application also provides a computer program product comprising a computer program stored in a computer readable storage medium. The computer program can be read by at least one processor of the device from a computer-readable storage medium, and execution of the computer program by the at least one processor causes the device to implement the methods provided by the various embodiments described above.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of modules is merely a division of logical functions, and an actual implementation may have another division, for example, a plurality of modules may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.
Wherein the modules may be physically separated, e.g. mounted at different locations of one device, or mounted on different devices, or distributed over multiple network elements, or distributed over multiple processors. The modules may also be integrated, for example, in the same device, or in a set of codes. The respective modules may exist in the form of hardware, or may also exist in the form of software, or may also be implemented in the form of software plus hardware. The method and the device can select part or all of the modules according to actual needs to achieve the purpose of the scheme of the embodiment.
When the respective modules are implemented as integrated modules in the form of software functional modules, they may be stored in a computer-readable storage medium. The software functional module is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device) or a processor to execute some steps of the methods according to the embodiments of the present application.
It should be understood that, although the respective steps in the flowcharts in the above-described embodiments are sequentially shown as indicated by arrows, the steps are not necessarily performed sequentially as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages, which may be performed at the same time or at different times. The execution sequence may be sequential, or may be alternated or rotated with other steps or at least a portion of the sub-steps or stages of other steps.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same. Although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: it is also possible to modify the solutions described in the previous embodiments or to substitute some or all of them with equivalents. And the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.
Claims (10)
1. A method for using external data, the method comprising:
acquiring external data at a preset frequency, and determining target data according to the external data;
according to the target data and the target field set, field contents corresponding to each target field in the target field set are obtained through disassembling, and the target field set comprises field names of at least one field to be inquired;
and storing the field content corresponding to each target field in the target field set into a storage unit, so that the storage unit returns the corresponding field content when receiving a query request.
2. The external data using method according to claim 1, wherein the determining target data according to the external data specifically includes:
carrying out format conversion on the external data to generate message data in a preset format;
and updating the target data according to the message data.
3. The external data using method according to claim 2, wherein the updating the target data according to the message data specifically includes:
classifying the message data according to preset data categories to obtain different types of message data;
and updating the different types of target data according to the different types of message data.
4. The external data using method according to any one of claims 1 to 3, further comprising:
and storing the external data into a storage unit so that the storage unit returns corresponding external data when receiving the query request.
5. The external data usage method according to any one of claims 1 to 3, wherein when the target field is a subscription field, the method further includes:
generating subscription information according to the target field and the field content corresponding to the target field;
and sending the subscription information.
6. The external data using method according to any one of claims 1 to 3, further comprising:
acquiring a query record, wherein the query record comprises a field name of a field to be queried;
and updating the target field set according to the query record so as to increase the target fields in the target field set.
7. An external data usage apparatus, the apparatus comprising:
the acquisition module is used for acquiring external data at a preset frequency and determining target data according to the external data;
a disassembling module, configured to disassemble to obtain field content corresponding to each target field in a target field set according to the target data and the target field set, where the target field set includes a field name of at least one field to be queried;
and the storage module is used for storing the field contents corresponding to each target field in the target field set into a storage unit so that the storage unit returns the corresponding field contents when receiving the query request.
8. A server, characterized in that the server comprises: a memory, a processor;
the memory is used for storing a computer program; the processor is configured to implement the external data usage method according to the computer program stored in the memory, according to any one of claims 1 to 6.
9. A computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, is configured to implement the external data usage method according to any one of claims 1 to 6.
10. A computer program product, characterized in that the computer program product comprises a computer program which, when being executed by a processor, implements the external data usage method of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110728706.XA CN113407609A (en) | 2021-06-29 | 2021-06-29 | External data using method, device and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110728706.XA CN113407609A (en) | 2021-06-29 | 2021-06-29 | External data using method, device and equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113407609A true CN113407609A (en) | 2021-09-17 |
Family
ID=77680278
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110728706.XA Pending CN113407609A (en) | 2021-06-29 | 2021-06-29 | External data using method, device and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113407609A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105279281A (en) * | 2015-11-17 | 2016-01-27 | 天泽信息产业股份有限公司 | Internet-of-things data access method |
| CN110278231A (en) * | 2018-03-16 | 2019-09-24 | 中移(苏州)软件技术有限公司 | A kind of data subscription distribution method and system |
| CN110457346A (en) * | 2019-07-05 | 2019-11-15 | 中国平安财产保险股份有限公司 | Data query method, apparatus and computer readable storage medium |
| CN110888839A (en) * | 2019-11-29 | 2020-03-17 | 厦门安胜网络科技有限公司 | Data storage and data search method and device |
| CN111176882A (en) * | 2019-12-23 | 2020-05-19 | 平安信托有限责任公司 | Specific data processing method, specific data processing device, computer equipment and storage medium |
| CN112800061A (en) * | 2021-01-29 | 2021-05-14 | 北京锐安科技有限公司 | Data storage method, device, server and storage medium |
-
2021
- 2021-06-29 CN CN202110728706.XA patent/CN113407609A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105279281A (en) * | 2015-11-17 | 2016-01-27 | 天泽信息产业股份有限公司 | Internet-of-things data access method |
| CN110278231A (en) * | 2018-03-16 | 2019-09-24 | 中移(苏州)软件技术有限公司 | A kind of data subscription distribution method and system |
| CN110457346A (en) * | 2019-07-05 | 2019-11-15 | 中国平安财产保险股份有限公司 | Data query method, apparatus and computer readable storage medium |
| CN110888839A (en) * | 2019-11-29 | 2020-03-17 | 厦门安胜网络科技有限公司 | Data storage and data search method and device |
| CN111176882A (en) * | 2019-12-23 | 2020-05-19 | 平安信托有限责任公司 | Specific data processing method, specific data processing device, computer equipment and storage medium |
| CN112800061A (en) * | 2021-01-29 | 2021-05-14 | 北京锐安科技有限公司 | Data storage method, device, server and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 孔淑红: ""MBA管理信息系统 精华读本"", 31 December 2002, 合肥：安徽人民出版社, pages: 388 - 390 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8799230B2 (en) | Method and system for centralized issue tracking | |
| CN110851465B (en) | Data query method and system | |
| CN112434015B (en) | Data storage method and device, electronic equipment and medium | |
| CN112035531B (en) | Sensitive data processing method, device, equipment and medium | |
| CN112364021B (en) | Service data processing method, device and storage medium | |
| US11620284B2 (en) | Backend data aggregation system and method | |
| CN117151045A (en) | Report processing method and device based on block chain and computer equipment | |
| CN114565443B (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN113407609A (en) | External data using method, device and equipment | |
| CN112464049B (en) | Method, device and equipment for downloading number detail list | |
| CN114817297A (en) | Method and device for processing data | |
| US11860883B2 (en) | System and method for implementing data usage analysis for database systems | |
| CN114116908A (en) | Data management method and device and electronic equipment | |
| CN109033271B (en) | Data insertion method and device based on column storage, server and storage medium | |
| CN111611056A (en) | Data processing method and device, computer equipment and storage medium | |
| CN111131393A (en) | User activity data statistical method, electronic device and storage medium | |
| CN114490095B (en) | Request result determination method and device, storage medium and electronic device | |
| CN116048468A (en) | Method, device, electronic equipment and medium for loading cache data and processing data | |
| CN113987322A (en) | Index data query method and device, computer equipment and computer program product | |
| CN116860541A (en) | Service data acquisition method, device, computer equipment and storage medium | |
| CN116820326A (en) | Data processing method, device, electronic equipment and storage medium | |
| CN115422199A (en) | Processing method and device of multidimensional statistical data and computer equipment | |
| CN114860807A (en) | Data query method, device, equipment and storage medium of block chain | |
| CN116415914A (en) | Service data processing method, device, computer equipment and storage medium | |
| CN117271606A (en) | Financial data processing method, apparatus, device, storage medium, and program product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |