CN111339171B - Data query method, device and equipment - Google Patents
Data query method, device and equipment Download PDFInfo
- Publication number
- CN111339171B CN111339171B CN202010082801.2A CN202010082801A CN111339171B CN 111339171 B CN111339171 B CN 111339171B CN 202010082801 A CN202010082801 A CN 202010082801A CN 111339171 B CN111339171 B CN 111339171B
- Authority
- CN
- China
- Prior art keywords
- data
- attribute field
- changed
- main
- data source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Mathematical Physics (AREA)
- Fuzzy Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data query method, a data query device and data query equipment, relates to the technical field of data processing, and can timely update common statistical data and data behind pages, improve service stability and accelerate data query. The method comprises the following steps: setting the association identification of each attribute field in the main data and the data source; extracting a change attribute field in response to detecting that a change of the attribute field occurs in the main data; associating the changed data corresponding to the changed attribute field in the associated data source to the main data based on the association identifier of the changed attribute field and the data source; receiving a query request of changed data, and screening out the changed data corresponding to a target attribute field from the changed data associated to the main data according to the target attribute field carried by the query request.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a method, an apparatus, and a device for querying data.
Background
With the expansion of business scale, enterprises are also rich in various business scenes at the front end, so that the development of the front end of the system becomes more and more complex. Generally, in order to implement various service scenarios, an enterprise often needs to interact among a plurality of service systems in an operation process, a large amount of service data is generated in each service scenario, and for the continuously generated data, a query requirement for the service data is often met.
At present, in the process of inquiring business data, business data users and operators carry out customized development according to requirements provided by business personnel, including business data export, inquiry display, search bars and the like.
In the process of implementing the invention, the inventor finds that the related art has at least the following problems:
the business development process is accompanied with business change, corresponding business data can be changed, and for the changed business data, the data query process needs to depend on background development, so that the change of the business data generated each time needs the background development for maintenance.
Disclosure of Invention
In view of this, the present application provides a method, an apparatus, and a device for data query, and mainly aims to solve the problem that the speed of data query is slow because common statistical data and data after paging cannot be updated in time in the process of querying data at present.
According to an aspect of the present application, there is provided a method of data query, the method comprising:
setting the association identification of each attribute field in the main data and the data source;
extracting a change attribute field in response to detecting that a change of the attribute field occurs in the main data;
associating the changed data corresponding to the changed attribute field in the associated data source to the main data based on the association identifier of the changed attribute field and the data source;
receiving a query request of changed data, and screening out the changed data corresponding to a target attribute field from the changed data associated to the main data according to the target attribute field carried by the query request.
Further, the setting of the association identifier between each attribute field in the main data and the data source specifically includes:
acquiring a data access interface defined for each attribute field in the main data in advance;
and setting the association identifier of each attribute field in the main data and the data source by analyzing the calling parameter of the data access interface.
Further, the acquiring the data access interface defined in advance for each attribute field in the main data specifically includes:
acquiring interface data types supported by each attribute field in the main data;
and defining a data access interface for each attribute field in the main data in advance according to the interface data type supported by each attribute field.
Further, the setting of the association identifier between each attribute field in the main data and the data source by analyzing the call parameter of the data access interface specifically includes:
determining the address identifier of the calling interface by analyzing the calling parameter of the data access interface;
and setting the address identifier of the calling interface as the associated identifier of each attribute field in the main data and the data source.
Further, the storing of the master data in the distributed full-text search engine and the creating of the dynamic index mapping, and associating the changed data corresponding to the changed attribute field in the associated data source to the master data based on the associated identifier of the changed attribute field and the data source specifically include:
defining the association identifier of the attribute changing field and the data source into a dynamic index mapping according to the dynamic index mapping created by the main data in the distributed full-text search engine;
and mapping and connecting the associated data source of the changed attribute field by using the dynamic index, and associating the changed data corresponding to the changed attribute field in the associated data source to the main data.
Further, the associating data source that is connected to the changed attribute field by using the dynamic index map and associates the changed data corresponding to the changed attribute field in the associating data source with the master data specifically includes:
connecting the associated data source of the changed attribute field by using the associated identifier of the changed attribute field provided by the dynamic index mapping;
and inquiring the changed data corresponding to the changed attribute field from the associated data source, and associating the changed data corresponding to the changed attribute field in the associated data source to the main data.
Further, the screening, according to the target attribute field carried by the query request, of the changed data associated with the main data, changed data corresponding to the target attribute field specifically includes:
taking the target attribute field carried by the query request as a matching field, and traversing and matching the changed data associated to the main data;
and acquiring the changed data which is matched with the target attribute field, and screening the changed data corresponding to the target attribute field.
According to another aspect of the present application, there is provided an apparatus for data query, the apparatus comprising:
the setting unit is used for setting the association identifier of each attribute field in the main data and the data source;
an extracting unit configured to extract a change attribute field in response to detection of occurrence of a change in the attribute field in the main data;
the association unit is used for associating the changed data corresponding to the changed attribute field in the associated data source to the main data based on the association identifier of the changed attribute field and the data source;
and the screening unit is used for receiving a query request of the changed data and screening the changed data corresponding to the target attribute field from the changed data associated to the main data according to the target attribute field carried by the query request.
Further, the setting unit includes:
the acquisition module is used for acquiring a data access interface which is defined for each attribute field in the main data in advance;
and the setting module is used for setting the association identifier of each attribute field and the data source in the main data by analyzing the calling parameter of the data access interface.
Further, the obtaining module comprises:
the acquisition submodule is used for acquiring the interface data types supported by each attribute field in the main data;
and the definition submodule is used for defining a data access interface for each attribute field in the main data in advance according to the interface data type supported by each attribute field.
Further, the setting module includes:
the determining submodule is used for determining the address identifier of the calling interface by analyzing the calling parameter of the data access interface;
and the setting submodule is used for setting the address identifier of the calling interface as the associated identifier of each attribute field and the data source in the main data.
Further, the main data is stored in a distributed full-text search engine, and a dynamic index map is created, and the association unit includes:
the definition module is used for defining the association identifier of the change attribute field and the data source into a dynamic index mapping according to the dynamic index mapping created by the main data in the distributed full-text search engine;
and the association module is used for mapping and connecting the associated data source of the changed attribute field by using the dynamic index and associating the changed data corresponding to the changed attribute field in the associated data source to the main data.
Further, the association module includes:
the connection sub-module is used for connecting the associated data source of the changed attribute field by using the associated identification of the changed attribute field provided by the dynamic index mapping;
and the query submodule is used for querying the changed data corresponding to the changed attribute field from the associated data source and associating the changed data corresponding to the changed attribute field in the associated data source to the main data.
Further, the screening unit includes:
the matching module is used for traversing and matching the changed data associated to the main data by taking the target attribute field carried by the query request as a matching field;
and the screening module is used for acquiring the changed data which are matched consistently and screening the changed data corresponding to the target attribute field.
According to yet another aspect of the present application, a storage medium is provided, on which a computer program is stored, which program, when being executed by a processor, carries out the above-mentioned method of data querying.
According to still another aspect of the present application, there is provided a data query entity apparatus, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, where the processor implements the data query method when executing the computer program.
By means of the technical scheme, compared with the conventional method for performing customized data query through the requirements provided by the service, the method, the device and the equipment for data query provided by the application have the advantages that in the conventional method, in general, each attribute field in the main data corresponds to different data acquisition sources, the application dynamically configures the acquisition form of each attribute field in advance by setting the association identifier of each attribute field in the main data and the data source, and associates each attribute field with the acquired data source by using the association identifier. The method can extract the change attribute field in response to the detection of the change of the attribute field in the main data, and then associate the change data corresponding to the change attribute field in the associated data source to the main data based on the associated identification of the change attribute field and the data source, so that the change data can be obtained and inquired simply without wasting effort of a background after the data is changed, common statistical data and data behind pages can be updated in time, the data inquiry speed is improved, and background development resources are saved.
The foregoing description is only an overview of the technical solutions of the present application, and the present application can be implemented according to the content of the description in order to make the technical means of the present application more clearly understood, and the following detailed description of the present application is given in order to make the above and other objects, features, and advantages of the present application more clearly understandable.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
fig. 1 is a schematic flowchart illustrating a method for querying data according to an embodiment of the present application;
FIG. 2 is a flow chart illustrating another method for querying data provided by an embodiment of the present application;
FIG. 3 is a schematic structural diagram illustrating an apparatus for querying data according to an embodiment of the present application;
fig. 4 shows a schematic structural diagram of another data querying device provided in the embodiment of the present application.
Detailed Description
The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments. It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.
At present, the existing data query mode is customized development through the requirements provided by the service, the change of each service data needs background development and maintenance, and as the service increases, if the background development is not timely, common statistical data and data behind pages cannot be updated timely, even the service is unstable, so that the data query speed is slow, and the data query efficiency is influenced. In order to solve the problem, the present embodiment provides a method for querying data, as shown in fig. 1, the method includes:
101. and setting the association identification of each attribute field in the main data and the data source.
The main data refers to high-value data which needs to be shared among various systems in the whole service range, such as an operation/transaction application system and an analysis type system. The attribute field may be a field in different service contexts, for example, a field in a service context of an interactive design may be a field related to user behavior, and a field in a service context of a location may be a field related to location information.
In general, a user may set field attributes on a control screen, such as display, must input, hiding, and optional input, of a field according to his/her business requirements, for example, setting "hide" for sensitive or unused information, "must input" for information that must be maintained, and "display" for information that cannot be modified. In particular, in the process of adding the attribute field to the main data, different obtaining ways of the attribute field are not used, for example, the attribute field related to the order information is from a cache database, and needs to be queried and obtained from the cache database, the attribute field related to the weather is from, and needs to be obtained by using an http protocol, considering the enormous number of fields and the change of subsequent fields, the configuration of the attribute field is not constant, in order to facilitate querying the attribute field in the main data, association identifiers of each attribute field in the main data and a data source can be set, where the association identifiers can indicate an association relationship between the attribute field and the data source, and an obtaining manner applied to the data source, for example, if the attribute field has an association relationship with the cache database, the data can be obtained by using an sql query manner for the cache database, if the attribute field and the application service have an incidence relation, the data can be acquired by using a mode called by the soa distributed framework aiming at the application service, and the data can be acquired by using a mode of carrying out value assignment on the embedded expression aiming at the executable script.
It will be appreciated that the various attribute fields need to be maintained frequently after the main data is recorded, and that such maintenance may include, but is not limited to, periodic cleaning, updating, logical verification of the data, etc.
The execution main body of the embodiment of the invention can be a device or equipment for data query, and can be configured at a user side, a data management platform is arranged at the user side to set the associated identification of each attribute field in the main data and the data source, and then the associated identification is utilized to associate the changed data into the main data.
Specifically, source data dispersed in each business system can be extracted and stored into a database of main data according to a main data model by using a data integration tool configured in a data management platform, and the integration of the dispersed data and the isomorphism of heterogeneous data sources are completed in the extraction process. In the extraction process, logical conversion or calculation can be performed on the source data, such as making some value mapping, deriving some new attribute fields, and the like. The data extraction process can use ETL tools Information, ODI, button, etc., and can also write a storage process by using an interface request mode.
102. In response to detecting a change in the attribute field in the primary data, a change attribute field is extracted.
In an actual application scenario, as business changes, table design in main data also needs to be changed correspondingly, useless attribute fields in a conventional data table need to be deleted, new attribute fields are derived, and the attribute fields in the main data are changed frequently.
For the embodiment of the present invention, a monitoring mechanism may be specifically set for the main data to detect whether the attribute field in the main data is changed, and a data synchronization component with data consistency may also be used to detect whether the attribute field in the main data is changed, which is not limited herein.
103. And associating the changed data corresponding to the changed attribute field in the associated data source to the main data based on the association identifier of the changed attribute field and the data source.
For the embodiment of the invention, the associated data source for acquiring the changed data corresponding to the changed attribute field can be determined by using the associated identifier of the changed attribute field and the data source, and when the attribute field is changed, the associated data source can be directly associated to the associated data source to acquire the changed data corresponding to the changed attribute field from the associated data source.
For example, the attribute field is changed to add the attribute values of the "name" field and the "name" field to the order data, and the associated data source of the order data is the order database in the cache, so that the attribute values of the "name" field and the "name" field can be added directly from the order database in the cache.
In order to facilitate the association processing of the data sources of the attribute fields, the related connection information of the data sources can be summarized in a form of creating a data source connection pool, so that when the system is initialized, the data source connection pool is responsible for distributing, managing and releasing the related connection information of the data sources for the attribute fields, connection does not need to be acquired, the related connection information can be directly associated to the data sources, and the data processing efficiency is improved.
For the embodiment of the present invention, the association identifier of the change attribute field and the data source may include connection information of the data source, which may be a name of the data source, interface information for invoking the data source, a database identifier corresponding to the data source, and the like, where the association identifier may be a combination of at least one of a plurality of connection information, and it is understood that in order to facilitate association of the change data, the interface information for invoking the data source is generally used as the association identifier.
104. Receiving a query request of changed data, and screening out the changed data corresponding to a target attribute field from the changed data associated to the main data according to the target attribute field carried by the query request.
The query request of the changed data is to query the attribute field changed in the main data, and for the changed attribute field, the target attribute field is an attribute field having an association relationship with the changed attribute field, for example, the changed attribute field is a date of birth, month of birth, date of birth, age, and the like.
In general, background customized maintenance is required for the changed attribute field, and the changed attribute field is intercepted from a corresponding data source, but in the embodiment of the invention, a mapping relation can be formed by configuring the associated identifier associated with the data source for the changed attribute field in advance, the changed data corresponding to the changed attribute field is associated to the main data, and further, when a query request of the changed data is received, the changed data corresponding to the target attribute field can be directly screened from the changed data in the main data in association without depending on a database.
Compared with the conventional method for performing customized data query according to the requirements provided by the service, the method for performing data query according to the embodiment of the present invention generally has the advantages that each attribute field in the main data corresponds to different data acquisition sources. The method can extract the change attribute field in response to the detection of the change of the attribute field in the main data, and then associate the change data corresponding to the change attribute field in the associated data source to the main data based on the associated identification of the change attribute field and the data source, so that the change data can be obtained and inquired simply without wasting effort of a background after the data is changed, common statistical data and data behind pages can be updated in time, the data inquiry speed is improved, and background development resources are saved.
Further, as a refinement and an extension of the specific implementation of the foregoing embodiment, in order to fully describe the specific implementation process of the present embodiment, the present embodiment provides another data query method, as shown in fig. 2, where the method includes:
201. and acquiring a data access interface which is defined for each attribute field in the main data in advance.
Typically, the attribute fields in the master data come from different data sources, and may come from data in various system databases or files, for example, a local database, a business file stored locally. Sometimes the main data needed may not be directly accessed to the database or to the file, but the data already processed by the third party, such as: weather conditions, stock information. The data is not directly opened to the outside, but the holder provides some defined data access interfaces for the outside. By utilizing the data access interface, the main data can be associated with the third-party data source, and then the data can be requested from the third-party data source.
When the interface is used for requesting data, the interface return data can be of multiple data types, and the multiple data types suitable for the interface return data are not defined in the main data. For the embodiment of the present invention, a data access interface may be defined in advance for each attribute field in the main data according to the interface data type supported by each attribute field by specifically acquiring the interface data type supported by each attribute field in the main data.
It can be understood that, here, the data access interfaces defined by the attribute fields in the main data may form a field interface list, and through the field interface list, a data acquisition request may be sent to a third party associated with the corresponding attribute field, and the receiving request interface returns the attribute value of the corresponding field.
202. And setting the association identifier of each attribute field in the main data and the data source by analyzing the calling parameter of the data access interface.
Generally, before calling a data access interface, an interface description document provided by a website or an interface provider needs to be found, and a control in an interface calling process, a calling parameter for acquiring data, and the like are recorded in the interface description document, where the calling parameter may include, but is not limited to, an interface identifier, a request address, a request parameter name, a request parameter type, and the like. For example, a request is made to connect to a weather data access interface, and the call parameters may include a city name, an interface address, and the like.
For the embodiment of the present invention, the address identifier of the calling interface may be specifically determined by analyzing the calling parameter of the data access interface, the address identifier of the calling interface may implement access to the data source, and the address identifier of the calling interface is further set as the association identifier of each attribute field in the main data and the data source. The association identifier of each attribute field and the data source in the main data is equivalent to a connection channel between the attribute field and the data source, where the attribute field may be connected with the file data source, such as excle and access, or the attribute field may be connected with the service data source, such as oracle and ODBC.
In general, each attribute field in the master data is from different data sources, when a new attribute field needs to be added to the master data, it is often necessary to deploy a data access component connected to the data source with great effort, and after a general data access component is designed, differences and requirements of projects change continuously, it is difficult to ensure that no new attribute field is additionally added, and especially after a project is run, the extra operation and maintenance workload brought by adding one attribute field far exceeds the workload for adjusting the code. According to the embodiment of the invention, the association identification of the data source mapped by the attribute field is set aiming at different attribute fields, so that the association relation between the main data and each data source is established, and the attribute field in the main data is conveniently processed.
203. In response to detecting a change in the attribute field in the primary data, a change attribute field is extracted.
For the embodiment of the invention, because the data volume in the main data is huge, the communication between the bottom databases is needed, in order to ensure the data consistency, the binlog log file exported by the data synchronization component can be used for detecting whether the attribute field in the main data is changed, the data synchronization component can be DRC, the real-time synchronization of heterogeneous databases is supported, the data record changes and subscribes the service, the binlog log records all the operation logs of adding, deleting and changing the main data, the execution time of the operation also comprises the execution time of the operation, namely the data operation executed by the server, and a new binlog log can be generated every time the server is restarted or the server is operated.
In general, a data synchronization component monitors that a main server generates a binlog log file in an application process, the binlog log generated by the main server records a modified form of main data of each row, and then the same data is modified by other slave servers, so that changed data in the main data is monitored by using a log file derived from the binlog, and a change attribute field is extracted.
204. And defining the association identifier of the change attribute field and the data source into a dynamic index mapping according to the dynamic index mapping created by the main data in the distributed full-text search engine.
With the continuous increase of the business data volume, the relational database has weak capability of supporting full-text retrieval or fuzzy query, the speed is low in the field query process, and the distributed full-text search engine can provide more convenient functions of data storage, query, display and the like. The embodiment of the invention has one node by starting the distributed full-text search engine on the server, stores the main data in the distributed full-text search engine, and also has a plurality of nodes on the same server by starting a plurality of distributed search engine processes, the main data stored in the distributed full-text search engine can be spread on the plurality of nodes, and the nodes can be rapidly communicated.
It can be understood that, in order to facilitate automatic identification of the attribute fields stored in the distributed full-text search engine, the distributed full-text search engine may set an automatic mapping mechanism for the concerned write content, may implement mapping of the types of the attribute fields, may also implement mapping of the data sources of the attribute fields, and implements the mapping relationship between the attribute fields and the types and data sources in the distributed full-text search engine by creating dynamic index mapping in the distributed full-text search engine and using the automatic mapping mechanism.
For the embodiment of the present invention, a plurality of index fragments may be created in the distributed full-text search engine, and specifically, in the process of creating a dynamic index map in the distributed full-text search engine, a unique mapping category may be set for a change attribute field in one index fragment, so as to avoid a situation where a plurality of mapping categories have a field with the same name, the unique mapping category of the change field in the index fragment may be defined, and an association identifier of the change field in the index fragment may be specified, for example, when the association identifier is a request address, the request address may be specified in the same format.
205. And mapping and connecting the associated data source of the changed attribute field by using the dynamic index, and associating the changed data corresponding to the changed attribute field in the associated data source to the main data.
Because each index fragment can automatically generate an index id during creation, the index id can be directly mapped to a data source of an attribute field by using dynamic index mapping, when an attribute field is changed, a distributed full-text search engine can find out the index id mapped by changing data by using a correlation identifier, automatically correlate the changed data corresponding to the attribute field in the data source to the mapped index id, and further add the changed data to main data corresponding to the index id.
It can be understood that, in order to better implement mapping of the change attribute field, a field dynamic mapping rule may be set, and the change data corresponding to the change attribute field is associated into the main data by using the field dynamic mapping rule. For example, the changed data in the associated data source may not be in the date format of the attribute field in the index, and time detection may be provided to automatically detect whether the changed data is in the date format when the changed data is inserted into the main data, and if not, automatically convert the changed data into the date format. Of course, the time format can be set by self-definition, the time detection is forbidden, and the like.
For the embodiment of the present invention, the associated data source of the changed attribute field may be connected by using the associated identifier of the changed attribute field provided by the dynamic index mapping, the changed data corresponding to the changed attribute field may be further queried from the associated data source, and the changed data corresponding to the changed attribute field in the associated data source may be associated to the main data.
206. And receiving a query request for changing data, and traversing and matching the changed data associated to the main data by taking a target attribute field carried by the query request as a matching field.
Since the main data is stored in each index fragment in the distributed full-text search engine, when a query request for changing data is received, a node receiving the request forwards the query request to a group of index fragments containing the main data, and an available index fragment can be selected by using a polling mechanism and the query request is forwarded. Specifically, in the process of selecting an available index fragment by using a polling mechanism, a query instruction can be generated to traverse the changed data associated with the main data according to a target attribute field in the query request as a matching field, and then an index id associated with the changed data is selected.
It should be noted that, in order to improve the accuracy of data query, a tokenizer may be specified for each query request, target attribute field, and each index shard, and the tokenizer is used to perform a tokenization process on the attribute field when each index shard stores main data. Similarly, the word segmentation processing is carried out on the target attribute field carried by the query request, so that the matching precision of the target attribute field and the changed data is improved.
207. And acquiring the changed data which is matched with the target attribute field, and screening the changed data corresponding to the target attribute field.
For the embodiment of the invention, the description of the changed data which is matched consistently meets the query condition, and specifically, the changed data corresponding to the target attribute field can be screened from the main data in the corresponding index id by using the index id associated with the changed data, so that the queried changed data is displayed.
Because the target attribute field is also the change attribute field, the change data corresponding to the target attribute field is inevitably in the change data of the associated data source, once the query request of the change data occurs, the data required to be queried is inevitably in the change data associated to the main data, and the change data corresponding to the target attribute field is further screened out, so that after the data is changed, a background does not need to take labor to obtain the change data, the change data can be directly associated to the data source by using the associated identifier, and the data query efficiency is improved.
Further, as a specific implementation of the method in fig. 1 and fig. 2, an embodiment of the present application provides a device for querying data, as shown in fig. 3, the device includes: a setting unit 31, an extracting unit 32, an associating unit 33, and a screening unit 34.
A setting unit 31, configured to set association identifiers of each attribute field in the main data and the data source;
an extracting unit 32, operable to extract a change attribute field in response to detecting that a change of the attribute field has occurred in the main data;
the associating unit 33 may be configured to associate, based on the association identifier of the change attribute field and the data source, the change data corresponding to the change attribute field in the associated data source into the master data;
the screening unit 34 may be configured to receive an inquiry request for changed data, and screen, according to a target attribute field carried by the inquiry request, changed data corresponding to the target attribute field from changed data associated with the main data.
Compared with the conventional method for performing customized data query according to the requirements provided by the service, the data query device provided by the embodiment of the invention has the advantages that in the conventional method, each attribute field in the main data corresponds to different data acquisition sources, the acquisition form of each attribute field is dynamically configured in advance by setting the association identifier of each attribute field in the main data and the data source, and each attribute field is associated with the acquired data source by using the association identifier. The method can extract the change attribute field in response to the detection of the change of the attribute field in the main data, and then associate the change data corresponding to the change attribute field in the associated data source to the main data based on the associated identification of the change attribute field and the data source, so that the change data can be obtained and inquired simply without wasting effort of a background after the data is changed, common statistical data and data behind pages can be updated in time, the data inquiry speed is improved, and background development resources are saved.
In a specific application scenario, as shown in fig. 4, the setting unit 31 includes:
the obtaining module 311 may be configured to obtain a data access interface defined in advance for each attribute field in the main data;
the setting module 312 may be configured to set association identifiers of each attribute field in the main data and the data source by analyzing the call parameter of the data access interface.
In a specific application scenario, as shown in fig. 4, the obtaining module 311 includes:
the obtaining sub-module 3111 may be configured to obtain interface data types supported by each attribute field in the main data;
the defining sub-module 3112 may be configured to define a data access interface for each attribute field in the main data in advance according to the interface data type supported by each attribute field.
In a specific application scenario, as shown in fig. 4, the setting module 312 includes:
the determining submodule 3121 may be configured to determine an address identifier of the call interface by analyzing a call parameter of the data access interface;
the setting sub-module 3122 may be configured to set the address identifier of the call interface to an association identifier of each attribute field in the main data and the data source.
In a specific application scenario, as shown in fig. 4, the main data is stored in a distributed full-text search engine, and a dynamic index map is created, where the associating unit 33 includes:
the defining module 331 may be configured to define, according to a dynamic index map created by main data in a distributed full-text search engine, the association identifier between the change attribute field and the data source in the dynamic index map;
the associating module 332 may be configured to map, by using the dynamic index, an associated data source connected to the changed attribute field, and associate changed data in the associated data source corresponding to the changed attribute field with the master data.
In a specific application scenario, as shown in fig. 4, the association module 332 includes:
the connection sub-module 3321 may be configured to connect the associated data source of the change attribute field with the associated identifier of the change attribute field provided by the dynamic index map;
the query sub-module 3322 may be configured to query the associated data source for the changed data corresponding to the changed attribute field, and associate the changed data corresponding to the changed attribute field in the associated data source with the main data.
In a specific application scenario, as shown in fig. 4, the screening unit 34 includes:
a matching module 341, configured to use the target attribute field carried by the query request as a matching field, and traverse and match the changed data associated with the main data;
the screening module 342 may be configured to obtain the changed data that is consistent with the matching, and screen out the changed data corresponding to the target attribute field.
It should be noted that other corresponding descriptions of the functional units related to the data query apparatus provided in this embodiment may refer to the corresponding descriptions in fig. 1 and fig. 2, and are not described herein again.
Based on the methods shown in fig. 1 and fig. 2, correspondingly, the present application further provides a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method for querying the data shown in fig. 1 and fig. 2 is implemented.
Based on such understanding, the technical solution of the present application may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.), and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the implementation scenarios of the present application.
Based on the method shown in fig. 1 and fig. 2 and the virtual device embodiment shown in fig. 3 and fig. 4, in order to achieve the above object, an embodiment of the present application further provides an entity device for data query, which may specifically be a computer, a smart phone, a tablet computer, a smart watch, a server, or a network device, and the entity device includes a storage medium and a processor; a storage medium for storing a computer program; a processor for executing a computer program to implement the above-described method of data query as shown in fig. 1 and 2.
Optionally, the entity device may further include a user interface, a network interface, a camera, a Radio Frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, and the like. The user interface may include a Display screen (Display), an input unit such as a keypad (Keyboard), etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface), etc.
Those skilled in the art will appreciate that the structure of the entity device for store search information processing provided by the present embodiment is not limited to the entity device, and may include more or less components, or some components in combination, or different component arrangements.
The storage medium may further include an operating system and a network communication module. The operating system is a program for managing hardware and software resources of the actual device for store search information processing, and supports the operation of the information processing program and other software and/or programs. The network communication module is used for realizing communication among components in the storage medium and communication with other hardware and software in the information processing entity device.
Through the above description of the embodiments, those skilled in the art will clearly understand that the present application can be implemented by software plus a necessary general hardware platform, and can also be implemented by hardware. Through the technical scheme, compared with the existing mode, the change data corresponding to the field to be detected which is preferentially changed is stored by utilizing the cache, the monitoring on the partial data can be realized by setting the effective monitoring time of the cache without depending on the query operation of a large number of databases, the change data can be monitored in time, the delay is reduced, and the timeliness of data consistency detection is improved.
Those skilled in the art will appreciate that the figures are merely schematic representations of one preferred implementation scenario and that the blocks or flow diagrams in the figures are not necessarily required to practice the present application. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario may be distributed in the devices in the implementation scenario according to the description of the implementation scenario, or may be located in one or more devices different from the present implementation scenario with corresponding changes. The modules of the implementation scenario may be combined into one module, or may be further split into a plurality of sub-modules.
The above application serial numbers are for description purposes only and do not represent the superiority or inferiority of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of the present application, but the present application is not limited thereto, and any variations that can be made by those skilled in the art are intended to fall within the scope of the present application.
Claims (16)
1. A method of data querying, comprising:
setting association identification of each attribute field and a data source in main data based on an acquisition path of the attribute field, wherein the main data is data needing to be shared among systems in the whole service range;
extracting a change attribute field in response to detecting that a change of the attribute field occurs in the main data;
associating the changed data corresponding to the changed attribute field in the associated data source to the main data based on the association identifier of the changed attribute field and the data source;
receiving a query request of changed data, and screening out the changed data corresponding to a target attribute field from the changed data associated to the main data according to the target attribute field carried by the query request.
2. The method according to claim 1, wherein the setting of the association identifier between each attribute field in the main data and the data source specifically includes:
acquiring a data access interface defined for each attribute field in the main data in advance;
and setting the association identifier of each attribute field in the main data and the data source by analyzing the calling parameter of the data access interface.
3. The method according to claim 2, wherein the obtaining of the data access interface defined in advance for each attribute field in the master data specifically includes:
acquiring interface data types supported by each attribute field in the main data;
and defining a data access interface for each attribute field in the main data in advance according to the interface data type supported by each attribute field.
4. The method according to claim 2, wherein the setting of the association identifier between each attribute field in the main data and the data source by analyzing the call parameter of the data access interface specifically comprises:
determining the address identifier of the calling interface by analyzing the calling parameter of the data access interface;
and setting the address identifier of the calling interface as the associated identifier of each attribute field in the main data and the data source.
5. The method according to any one of claims 1 to 4, wherein the master data is stored in a distributed full-text search engine, a dynamic index map is created, and associating the changed data corresponding to the changed attribute field in the associated data source to the master data based on the association identifier of the changed attribute field and the data source specifically includes:
defining the association identifier of the attribute changing field and the data source into a dynamic index mapping according to the dynamic index mapping created by the main data in the distributed full-text search engine;
and mapping and connecting the associated data source of the changed attribute field by using the dynamic index, and associating the changed data corresponding to the changed attribute field in the associated data source to the main data.
6. The method according to claim 5, wherein the associating data source that connects the changed attribute field by using the dynamic index map associates the changed data corresponding to the changed attribute field in the associating data source with the master data, specifically including:
connecting the associated data source of the changed attribute field by using the associated identifier of the changed attribute field provided by the dynamic index mapping;
and inquiring the changed data corresponding to the changed attribute field from the associated data source, and associating the changed data corresponding to the changed attribute field in the associated data source to the main data.
7. The method according to claim 1, wherein the step of screening, according to the target attribute field carried by the query request, changed data corresponding to the target attribute field from changed data associated with the main data includes:
taking the target attribute field carried by the query request as a matching field, and traversing and matching the changed data associated to the main data;
and acquiring the changed data which is matched with the target attribute field, and screening the changed data corresponding to the target attribute field.
8. An apparatus for querying data, comprising:
the setting unit is used for setting the association identifier of each attribute field and a data source in the main data based on the acquisition path of the attribute field, wherein the main data is data which needs to be shared among systems in the whole service range;
an extracting unit configured to extract a change attribute field in response to detection of occurrence of a change in the attribute field in the main data;
the association unit is used for associating the changed data corresponding to the changed attribute field in the associated data source to the main data based on the association identifier of the changed attribute field and the data source;
and the screening unit is used for receiving a query request of the changed data and screening the changed data corresponding to the target attribute field from the changed data associated to the main data according to the target attribute field carried by the query request.
9. The apparatus according to claim 8, wherein the setting unit includes:
the acquisition module is used for acquiring a data access interface which is defined for each attribute field in the main data in advance;
and the setting module is used for setting the association identifier of each attribute field and the data source in the main data by analyzing the calling parameter of the data access interface.
10. The apparatus of claim 9, wherein the obtaining module comprises:
the acquisition submodule is used for acquiring the interface data types supported by each attribute field in the main data;
and the definition submodule is used for defining a data access interface for each attribute field in the main data in advance according to the interface data type supported by each attribute field.
11. The apparatus of claim 9, wherein the setup module comprises:
the determining submodule is used for determining the address identifier of the calling interface by analyzing the calling parameter of the data access interface;
and the setting submodule is used for setting the address identifier of the calling interface as the associated identifier of each attribute field and the data source in the main data.
12. The apparatus according to any one of claims 9-11, wherein the main data is stored in a distributed full-text search engine and created with a dynamic index map, and the associating unit includes:
the definition module is used for defining the association identifier of the change attribute field and the data source into a dynamic index mapping according to the dynamic index mapping created by the main data in the distributed full-text search engine;
and the association module is used for mapping and connecting the associated data source of the changed attribute field by using the dynamic index and associating the changed data corresponding to the changed attribute field in the associated data source to the main data.
13. The apparatus of claim 12, wherein the associating module comprises:
the connection sub-module is used for connecting the associated data source of the changed attribute field by using the associated identification of the changed attribute field provided by the dynamic index mapping;
and the query submodule is used for querying the changed data corresponding to the changed attribute field from the associated data source and associating the changed data corresponding to the changed attribute field in the associated data source to the main data.
14. The apparatus of claim 8, wherein the screening unit comprises:
the matching module is used for traversing and matching the changed data associated to the main data by taking the target attribute field carried by the query request as a matching field;
and the screening module is used for acquiring the changed data which are matched consistently and screening the changed data corresponding to the target attribute field.
15. A storage medium on which a computer program is stored, which program, when being executed by a processor, carries out the method of data querying of one of claims 1 to 7.
16. An apparatus for data query, comprising a storage medium, a processor and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the method for data query according to any one of claims 1 to 7 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010082801.2A CN111339171B (en) | 2020-02-07 | 2020-02-07 | Data query method, device and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010082801.2A CN111339171B (en) | 2020-02-07 | 2020-02-07 | Data query method, device and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111339171A CN111339171A (en) | 2020-06-26 |
| CN111339171B true CN111339171B (en) | 2021-05-04 |
Family
ID=71181489
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010082801.2A Active CN111339171B (en) | 2020-02-07 | 2020-02-07 | Data query method, device and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111339171B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112165465B (en) * | 2020-09-16 | 2023-04-07 | 杭州安恒信息技术股份有限公司 | Data processing method and device for outgoing third-party platform and electronic device |
| CN112765215A (en) * | 2021-01-19 | 2021-05-07 | 上海幻电信息科技有限公司 | Data query method and device |
| CN112764744B (en) * | 2021-01-25 | 2023-04-14 | 芝麻云信（武汉）科技有限公司 | Interface service design method for ERP system |
| CN113590746A (en) * | 2021-07-01 | 2021-11-02 | 中国国家铁路集团有限公司 | Method, device, equipment and medium for comprehensively retrieving information of data warehouse |
| CN113936833B (en) * | 2021-09-06 | 2023-06-13 | 中核核电运行管理有限公司 | Automatic control method and device for permanent change state of nuclear power plant |
| CN114138899A (en) * | 2021-11-22 | 2022-03-04 | 支付宝(杭州)信息技术有限公司 | Block chain-based data stream transfer method and device |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7917545B2 (en) * | 2004-09-30 | 2011-03-29 | Microsoft Corporation | Web-based data form |
| CN110019211A (en) * | 2017-11-27 | 2019-07-16 | 北京京东尚科信息技术有限公司 | The methods, devices and systems of association index |
| CN108090154A (en) * | 2017-12-08 | 2018-05-29 | 广州市申迪计算机系统有限公司 | A kind of isomerous multi-source data fusion querying method and device |
| CN108829727B (en) * | 2018-05-09 | 2021-06-25 | 未鲲(上海)科技服务有限公司 | Data storage method and device, computer equipment and storage medium |
-
2020
- 2020-02-07 CN CN202010082801.2A patent/CN111339171B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN111339171A (en) | 2020-06-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111339171B (en) | Data query method, device and equipment | |
| CN108228817B (en) | Data processing method, device and system | |
| US10275355B2 (en) | Method and apparatus for cleaning files in a mobile terminal and associated mobile terminal | |
| CN105468720A (en) | Method for integrating distributed data processing systems, corresponding systems and data processing method | |
| CN107103011B (en) | Method and device for realizing terminal data search | |
| CN113568924B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN111382182A (en) | Data processing method and device, electronic equipment and storage medium | |
| JP2013530449A (en) | How to improve queries for searching databases | |
| CN115438087B (en) | Data query method, device, storage medium and equipment based on cache library | |
| CN112416991A (en) | Data processing method and device and storage medium | |
| CN110737634A (en) | Log searching method, client, server and computer readable storage medium | |
| CN111047434B (en) | Operation record generation method and device, computer equipment and storage medium | |
| CN111767574A (en) | User permission determining method and device, electronic equipment and readable storage medium | |
| CN116611411A (en) | Business system report generation method, device, equipment and storage medium | |
| JP2016162016A (en) | Management information acquisition program, management information acquisition method, and management information acquisition device | |
| CN110515979B (en) | Data query method, device, equipment and storage medium | |
| CN111125226B (en) | Configuration data acquisition method and device | |
| CN110008243B (en) | Data table processing method and device | |
| CN109063215B (en) | Data retrieval method and device | |
| CN112491943A (en) | Data request method, device, storage medium and electronic equipment | |
| CN113722337B (en) | Service data determination method, device, equipment and storage medium | |
| CN114153438A (en) | API automatic generation system | |
| CN115185973A (en) | Data resource sharing method, platform, device and storage medium | |
| CN113434585A (en) | Resource saving method and equipment | |
| CN113778996A (en) | Large data stream data processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |