WO2023087673A1 - Procédé et appareil de récupération de données hiérarchiques et dispositif - Google Patents

Procédé et appareil de récupération de données hiérarchiques et dispositif Download PDF

Info

Publication number
WO2023087673A1
WO2023087673A1 PCT/CN2022/097345 CN2022097345W WO2023087673A1 WO 2023087673 A1 WO2023087673 A1 WO 2023087673A1 CN 2022097345 W CN2022097345 W CN 2022097345W WO 2023087673 A1 WO2023087673 A1 WO 2023087673A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
retrieval
database
type
hierarchical
Prior art date
Application number
PCT/CN2022/097345
Other languages
English (en)
Chinese (zh)
Inventor
田标
崔伟
陈献青
邓捷
王刚
王家宾
黄文文
Original Assignee
天翼数字生活科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 天翼数字生活科技有限公司 filed Critical 天翼数字生活科技有限公司
Publication of WO2023087673A1 publication Critical patent/WO2023087673A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2228—Indexing structures
    • G06F16/2246—Trees, e.g. B+trees
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/242—Query formulation
    • G06F16/2433—Query languages
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution

Definitions

  • the invention relates to the technical field of data retrieval, in particular to a hierarchical data retrieval method, device and equipment.
  • a hierarchical data retrieval method provided by the first aspect of the present invention includes:
  • the retrieval type is index database retrieval
  • the step of querying the attribution information corresponding to the user identifier from a preset database, and determining the database hierarchy type corresponding to the user identifier includes:
  • the affiliation information includes a first affiliation organization ID
  • the relational database includes a plane hierarchical tree information table and a first organization equipment association information table; when the database hierarchy type is a limited hierarchy type, the if If the retrieval type is a relational database retrieval, then the relational database corresponding to the database hierarchy type is retrieved according to the attribution information, and the step of obtaining the first target hierarchy data includes:
  • the retrieval type is a relational database retrieval, then query the equipment information table of the first organization according to the ID of the first affiliation organization, and determine the corresponding equipment data and the branch ID of the hierarchical tree;
  • An association between each branch data of the hierarchical tree and the device data is established to obtain first target hierarchical data.
  • the affiliation information includes a second affiliation organization ID and the superior node ID
  • the relational database includes a tree node information table, a tree node association relationship table, and a second organization equipment association information table; when the database level
  • the retrieval type is a relational database retrieval, the step of retrieving the relational database corresponding to the database hierarchical type according to the attribution information to obtain the first target hierarchical data includes:
  • the retrieval type is a relational database retrieval, then query the equipment information table of the second organization according to the ID of the second affiliation organization, and determine the corresponding equipment data, the ID of the superior node and the node ID of the hierarchical tree;
  • the affiliation information includes a third organization affiliation ID
  • the ES index library includes a flat hierarchical tree index table and a first organization equipment association index table
  • the database hierarchy type is a limited hierarchy type
  • the retrieval type is an index database retrieval, then parsing the data retrieval request to obtain a search condition and a target data format;
  • the affiliation information includes a fourth organization affiliation ID
  • the ES index library includes a tree node information index table and a second organization equipment association index table; when the database hierarchy type is an unrestricted hierarchy type, the If the retrieval type is index database retrieval, then according to the data retrieval request and the attribution information, retrieve the ES index database corresponding to the database hierarchy type, and obtain the second target hierarchy data, including:
  • the retrieval type is an index database retrieval, then parsing the data retrieval request to obtain a search condition;
  • the tree nodes corresponding to each of the hierarchical tree node IDs are associated according to the second node association relationship, and an association with the device data is established to obtain second target hierarchical data.
  • An association between the additional user identifier and the additional device data is established, and an additional ES index table is obtained and stored in the ES index library.
  • the second aspect of the present invention provides a hierarchical data retrieval device, including:
  • a user identification acquisition module configured to obtain a user identification corresponding to the user terminal in response to a login request sent by any user terminal
  • a database hierarchy type determining module configured to query the attribution information corresponding to the user identifier from a preset database, and determine the database hierarchy type corresponding to the user identifier;
  • a retrieval type acquisition module configured to obtain the retrieval type corresponding to the data retrieval request when receiving the data retrieval request sent by the client;
  • a relational data retrieval module configured to retrieve the relational database corresponding to the database hierarchy type according to the attribution information if the retrieval type is a relational database retrieval, to obtain the first target hierarchical data;
  • An index data retrieval module configured to retrieve the ES index library corresponding to the database hierarchy type according to the data retrieval request and the attribution information if the retrieval type is index database retrieval, to obtain second target hierarchical data;
  • a hierarchical data returning module configured to return the first target hierarchical data or the second target hierarchical data to the client.
  • the third aspect of the present invention also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor is executed as described in the first aspect of the present invention.
  • a computer program is stored in the memory, and when the computer program is executed by the processor, the processor is executed as described in the first aspect of the present invention.
  • the present invention has the following advantages:
  • the present invention obtains the user identification corresponding to the user terminal by responding to a login request sent by any user terminal; queries the attribution information corresponding to the user identification from a preset database, and determines the database hierarchy type corresponding to the user identification; When a data retrieval request is made, obtain the retrieval type corresponding to the data retrieval request; if the retrieval type is a relational database retrieval, retrieve the relational database corresponding to the database hierarchy type according to the attribution information, and obtain the first target hierarchical data; if the retrieval type is an index database retrieval , then according to the data retrieval request and the ES index library corresponding to the attribution information search database level type, obtain the second target level data; return the first target level data or the second target level data to the client.
  • hierarchical tree data can be obtained through one SQL statement or several queries. Furthermore, the efficiency of the data retrieval process under the amount of hundreds of millions of users is more effectively improved.
  • FIG. 1 is a flow chart of the steps of a hierarchical data retrieval method provided by Embodiment 1 of the present invention
  • FIG. 2 is a flow chart of steps of a hierarchical data retrieval method provided by Embodiment 2 of the present invention
  • FIG. 3 is a flow chart of steps for an incremental synchronization process of hierarchical data provided by an embodiment of the present invention
  • Fig. 4 is a flow chart of the steps of the retrieval query process of a kind of ES index library provided by the embodiment of the present invention.
  • FIG. 5 is a structural block diagram of a hierarchical data retrieval device provided by Embodiment 3 of the present invention.
  • the embodiment of the present invention provides a hierarchical data retrieval method, device and equipment, which are used to solve the problem that the traditional hierarchical data retrieval method has low retrieval efficiency in the Internet business search process with hundreds of millions of users due to the limitation of its structure and retrieval path. Low, technical problems that cannot meet the needs of Internet business development.
  • FIG. 1 is a flowchart of steps of a hierarchical data retrieval method provided by Embodiment 1 of the present invention.
  • a hierarchical data retrieval method provided by the present invention includes:
  • Step 101 in response to a login request sent by any client, obtain a user ID corresponding to the client;
  • User ID refers to the client ID carried in the login request when any client sends a login request.
  • the device responds to the login request sent by any client, and performs login verification on the login request, for example, whether the client has the corresponding login authority by checking whether the client is registered, etc., if the login If the verification is passed, the user ID corresponding to the client is obtained from the login request, so as to obtain the data basis for the subsequent client to query hierarchical data.
  • Step 102 query the attribution information corresponding to the user ID from the preset database, and determine the database hierarchy type corresponding to the user ID;
  • the preset database refers to a relational database that stores information such as a client-organization association table, a hierarchical tree branch ID set, and the like.
  • Affiliation information refers to the information indicating the association between the user ID and multiple affiliation organizations, user types and branches of the affiliation tree, including but not limited to the ID of the first affiliation organization, the ID of the second affiliation organization, and the ID of the third affiliation organization , the ID of the fourth affiliation organization, the ID of the superior node, and other information.
  • the user ID After obtaining the user ID carried in the login request, the user ID can be used as a search word to query the corresponding attribution information in the preset database, and at the same time, according to the user ID combined with the login request, the hierarchical database to which the user ID belongs can be judged Database hierarchy type.
  • Step 103 when receiving the data retrieval request sent by the client, obtain the retrieval type corresponding to the data retrieval request;
  • the data retrieval request sent by the client After determining the database level type corresponding to the user ID, if the data retrieval request sent by the client is received, the data retrieval request can be further responded to, and the corresponding retrieval type can be obtained to determine the retrieval method required by the data retrieval request. For example, relational database retrieval or indexed database retrieval, etc.
  • Step 104 if the search type is relational database search, search the relational database corresponding to the database level type according to the attribution information, and obtain the first target level data;
  • a relational database refers to a database built on the basis of a relational database model. It uses concepts and methods such as set algebra to process data in the database. It is also a set of tables organized into a formal descriptive table. Essence is a special collection of data items loaded with data in these tables that can be accessed or recalled in many different ways without reorganizing the database tables.
  • the definition of a relational database results in a table of metadata or in a formal description of tables, columns, scopes and constraints. Each table (sometimes called a relation) contains one or more data types represented by columns. Each row contains a unique data entity of the type defined by the column.
  • the SQL language is the interface of standard users and applications to relational databases. Its advantage is that it is easy to expand, and after the initial database creation, a new data type can be added without modifying all existing application software.
  • the mainstream relational databases include oracle, db2, sqlserver, sybase, mysql, etc.
  • attribution information can be used as a keyword or keyword, and the retrieval method of relational database retrieval is in the The relational database corresponding to the database hierarchy type is retrieved to obtain the first target hierarchy data.
  • Step 105 if the retrieval type is index database retrieval, according to the data retrieval request and the ES index database corresponding to the attribution information retrieval database hierarchy type, the second target hierarchy data is obtained;
  • the ES index library refers to a distributed, highly scalable, high real-time search and data analysis engine.
  • ES indexes are equivalent to tables in relational databases.
  • An index can contain multiple fields of different data types, but ES indexes cannot be associated with tables. It can easily enable large amounts of data to be searched, analyzed and explored. Taking full advantage of the horizontal scalability of Elasticsearch can make data more valuable in the production environment.
  • the implementation principle of Elasticsearch is mainly divided into the following steps. First, the user submits the data to the Elasticsearch database, and then uses the word segmentation controller to segment the corresponding sentence, and stores its weight and word segmentation results into the data. When the user searches for data At that time, the results will be ranked and scored according to the weight, and then the returned results will be presented to the user.
  • the retrieval type is judged to be indexed database retrieval, it indicates that the retrieval method at this time is fuzzy search or full-text search.
  • compound keywords or key words can be formed based on the search conditions specified in the data retrieval request and attribution information. word, search the ES index database corresponding to the database hierarchy type, so as to obtain the second target hierarchy data.
  • Step 106 return the first target level data or the second target level data to the client.
  • the first target level data or the second target level data After the first target level data or the second target level data is acquired, it is returned to the client for display, so that the user can check the level data.
  • the user identifier corresponding to the client is obtained; the attribution information corresponding to the user identifier is queried from a preset database, and the database hierarchy type corresponding to the user identifier is determined;
  • the retrieval type corresponding to the data retrieval request if the retrieval type is a relational database retrieval, then retrieve the relational database corresponding to the database hierarchy type according to the attribution information, and obtain the first target level data; if the retrieval If the type is index database retrieval, then according to the data retrieval request and the ES index library corresponding to the attribution information retrieval database level type, obtain the second target level data; return the first target level data or the second target level data to the client.
  • hierarchical tree data can be obtained through one SQL statement or several queries. Furthermore, the efficiency of the data retrieval process under the amount of hundreds of millions of users is more effectively improved.
  • FIG. 2 is a flowchart of steps of a hierarchical data retrieval method provided by Embodiment 2 of the present invention.
  • a hierarchical data retrieval method provided by the present invention includes:
  • Step 201 in response to a login request sent by any client, obtain a user ID corresponding to the client;
  • step 201 the specific implementation process of step 201 is similar to step 101, and will not be repeated here.
  • Step 202 query the attribution information corresponding to the user ID from the preset database, and determine the database hierarchy type corresponding to the user ID;
  • step 202 may include the following sub-steps:
  • the attribution information may include various contents.
  • the preset database may be queried according to the user identification to obtain the attribution information corresponding to the user identification and cache.
  • the database hierarchy type corresponding to the user ID is limited. At this time, it can be determined that the database hierarchy type corresponding to the user ID is a limited hierarchy type; if it contains a superior node ID, it indicates that the affiliation organization ID in the attribution information at this time has an associated superior Node ID and hierarchical data are stored in the form of node information and node association relationship. At this time, it can be determined that the database hierarchical type corresponding to the user ID is an unrestricted hierarchical type.
  • the affiliation information may also include a user type, which is used to limit the type to which the user terminal belongs, for example, the user terminal belongs to a certain organization, and the embodiment of the present invention does not limit this, such as the user terminal being an independent organization.
  • Step 203 when receiving the data retrieval request sent by the client, obtain the retrieval type corresponding to the data retrieval request;
  • step 203 is similar to that of step 103, and will not be repeated here.
  • a relational database can be constructed in the following ways:
  • an organization information table (saving organization id, name, address, contact information and other organization's own data), an organization and equipment association table, a user information table, a user and organization association table in the relational database, according to SQL
  • the query condition indexes the table.
  • the user information table is used to save the user terminal id, name, mobile phone number and other self-information.
  • the field names of each node should be the abbreviation of the business entity name as much as possible, and the field names numbered in sequence should be used only in special cases.
  • the user-institution association table is used to record the association relationship between the user and the institution to which it belongs.
  • t_region_tree Taking the hierarchical tree from province to street as an example, the flat hierarchical tree information table t_region_tree can be shown in Table 1 below:
  • the primary key is also the hierarchical tree branch id State / province province varchar the province/municipality id province_id bigint the City area city varchar the city/district id city_id bigint the District/County county varchar the district/county id county_id bigint the street/town street varchar the street/town id street_id bigint the
  • a table is used to record the complete branches from the top to the bottom of the hierarchical tree and the relationship information of each node in it.
  • the order of the fields in the table from left to right corresponds to the actual order of the nodes from top to bottom of the hierarchical tree.
  • the equipment information table of the first institution may be shown in Table 2 below:
  • the main fields of the device information table of the first organization may include but not limited to ID of the first affiliated organization, device id, device name, IP address of the device, etc. It can also be a hierarchical tree branch id.
  • the name of the province, city, county, and district as the node name changes very little. If the node name will change frequently, and the number of child nodes in a tree may be relatively large at the same time, in order to avoid the influence of the database update operation, you can also extract all the node names to create a node information table, the main fields include node id, node name, node Encoded so that only one record here needs to be updated each time the node name is updated.
  • the flat hierarchical tree information table only the id or code of the node is saved for each hierarchical tree node, and the id and code are constructed by continuous self-incremental data. At this time, a record can store up to hundreds of hierarchical trees.
  • Table 1 and Table 2 The structural advantages of the above Table 1 and Table 2 include: 1) meet the business needs of hierarchical tree management and storage; 2) can obtain all directory trees of an organization through a SQL query; 3) can obtain a node containing a node through a SQL query 4) Support user data permissions and data security, ensuring that a user can only see the data allowed by his permissions; 5) The impact of management operations such as adding, deleting, and modifying business data on other operations can be controlled within A limited range that does not grow larger as the amount of data increases.
  • the data that needs to be saved is generally large in scale. If the above-mentioned relational database of the restricted hierarchical type is used to save, it will easily lead to slow reading.
  • the above-mentioned flat hierarchical tree information table can be split into The tree node information table t_node_link and the tree node association relationship table t_node_link are established at the same time as the second institutional equipment association information table.
  • the tree node information table records each tree node ID, node name and other fields that can be added according to the business.
  • the tree node association table can be shown in Table 3 below:
  • Step 204 if the search type is relational database search, search the relational database corresponding to the database level type according to the attribution information, and obtain the first target level data;
  • the affiliation information includes a first affiliation organization ID
  • the relational database includes a plane hierarchical tree information table and a first organization equipment association information table
  • step 204 may include the following subsections step:
  • the retrieval type is a relational database retrieval, query the equipment information table of the first organization according to the ID of the first affiliation organization, and determine the corresponding equipment data and the branch ID of the hierarchical tree;
  • An association between branch data of each hierarchical tree and device data is established to obtain the first target hierarchical data.
  • the database hierarchy type is a limited hierarchy type and the retrieval type is a relational database retrieval
  • the data to be retrieved at this time is a relational database of a limited hierarchy type.
  • the first Affiliation organization ID query the first organization equipment information table according to the first affiliation organization ID, determine the equipment data corresponding to the first affiliation organization ID and the corresponding hierarchical tree branch ID, and then query the plane hierarchical tree information table according to the hierarchical tree branch ID , determine the corresponding hierarchical tree branch data, that is, the above-mentioned entire hierarchical tree branch from the province to the bottom village, and finally establish the association between each hierarchical tree branch data and equipment data to obtain the first target hierarchical data.
  • the branch data of the hierarchical tree After obtaining the branch data of the hierarchical tree, you can put the branch data of the hierarchical tree, all the device data under the branch and the corresponding branch ID of the hierarchical tree into a record, and the device data can be used as a collection type field to form the branch data of the branch ID of the hierarchical tree A property of an object.
  • the affiliation information includes a second affiliation organization ID and a superior node ID
  • the relational database includes a tree node information table, a tree node association relationship table, and a second organization equipment association information table; when the database hierarchy type is non-
  • step 204 may include the following sub-steps:
  • the retrieval type is a relational database retrieval, query the equipment information table of the second organization according to the ID of the second affiliation organization, and determine the corresponding equipment data, upper node ID and hierarchical tree node ID;
  • the tree nodes corresponding to each tree node ID are associated according to the first node association relationship, and the association between the superior node ID and the device data is established to obtain the first target level data.
  • the database hierarchy type is an unrestricted hierarchy type and the retrieval type is a relational database retrieval
  • the relational database to be retrieved at this time is an unrestricted hierarchy type
  • the second affiliation organization ID can be queried.
  • Organization equipment information table determine the equipment data associated with the second affiliation organization ID, superior node ID and hierarchical tree node ID; then use the superior node ID and hierarchical tree node ID to retrieve the tree node information table to determine the hierarchical tree node ID All the tree node IDs of the entire hierarchical tree where the corresponding tree nodes are located, and then respectively retrieve the tree node association relationship table according to each tree node ID, to determine the first node association relationship between the tree nodes corresponding to each tree node ID, according to the The first node association relationship associates the tree nodes corresponding to each tree node ID and establishes an association between the superior node ID and the device data to obtain the first target hierarchical data.
  • the ES index library in order to ensure the stability and accuracy of the index library, is usually built after establishing a certain scale relational database and running for a period of time. At the same time, in order to realize the periodic update of the ES index library, you can Execute the preset configuration file according to the preset update cycle, perform field conversion on the relational data in the relational database to obtain the corresponding ES index field, and then synchronize the ES index field to the ES index library.
  • the data synchronization of the ES index library can be realized through the following steps S11-S15 or steps S21-S26:
  • the Chinese region name needs to be modified in ib/logstash/filters/geoip.rb under the logstash installation directory for each obtained region name
  • ib/logstash/filters/geoip.rb under the logstash installation directory for each obtained region name
  • the IP address information here is saved in a csv file, so you can also maintain your own address library, and you can also call the IP analysis service through the ruby code in geoip.rb to obtain relevant Chinese area information ;
  • the open source data synchronization tool canal can be well decoupled from the business system because it directly operates MySQL's binlog, and its operation puts very little pressure on MySQL.
  • a hierarchical tree branch has not been fully formed Synchronizing it to ES will only increase useless ES update operations. Due to differences in business rules, it is common for business entity data in different states not to be fully synchronized.
  • data processing and verification may require querying other Database or ES, so you need to control the incremental synchronization of data through the canal client.
  • Figure 3 The steps are as follows:
  • the table can be constructed in the following ways:
  • the ES index library of unrestricted hierarchical type may include but not limited to node information index table, flat hierarchical tree index table and first organization device association index table.
  • the flat hierarchical tree index table may be shown in Table 4 below:
  • fields that need to be searched for business can also be added to the index, such as the geographic location of the device, device name, device label, device capability, etc.
  • the first agency equipment association index table may be shown in Table 5 below:
  • special query annotations can be encapsulated for specific types of queries in ES.
  • a new annotation FuzzyQuery and its processing flow can be encapsulated according to the implementation and processing flow of Query annotations, specifying the query that needs to be queried
  • the field, the identification of the corresponding parameter value, and the logical relationship between the fields when assembling are processed by logical AND by default, and the parameters in the method where the FuzzyQuery annotation of the Repository interface is directly replaced in order at runtime; the specified return field list or For the list of fields to be excluded, directly generate the corresponding content through string processing in the painless script of the query request, and keep the same effect as the Query annotation in other respects.
  • the specific construction of the flat hierarchical tree index table is similar to the above table 4, and the node information index table can be shown in the following table 6:
  • the second agency equipment association index table may be shown in Table 7 below:
  • the affiliation information includes a third organization affiliation ID
  • the ES index library includes a flat hierarchical tree index table and a first organization equipment association index table
  • step 205 may include the following Substeps:
  • the retrieval type is indexed database retrieval
  • the data retrieval request is parsed to obtain the search condition and the target data format
  • the attribution ID of the third organization retrieve the organization's equipment association index table, and determine the corresponding equipment data
  • the association between the hierarchical tree information and the equipment data is established and converted into the target data format to obtain the second target hierarchical data.
  • Search criteria refer to data restriction information that may not be exactly the same as the fields in the ES index library but is related.
  • the target data format refers to the hierarchical tree data required by the client or the unique data format of the client.
  • the data retrieval request may be parsed to obtain the search conditions and the target data format.
  • the search conditions and the affiliation ID of the third organization asynchronously search from the organization equipment management index table and the flat hierarchical tree index table to determine the equipment data corresponding to the third organization affiliation ID and the hierarchical tree information that meets the search conditions; and then establish a hierarchy
  • the association between the tree information and the device data is converted into the target data format to obtain the second target level data.
  • the retrieval query of the ES index library can be realized through the following steps S31-S34:
  • the user obtains the attribution ID of the third organization from the cache, and if not found, first queries the relational database MySQL, updates the cache, and then continues;
  • S32 Asynchronously search for eligible data from the institutional device management index table and the flat hierarchical tree index table according to the third organization's affiliation ID and search conditions, and obtain hierarchical tree information and device data that meet the search conditions as search results;
  • the node information of the tree and the relationship between the nodes of the plane hierarchical tree are respectively stored in different indexes, then asynchronously and simultaneously Search the institutional equipment management index table and the flat hierarchical tree index table to obtain the search results.
  • you do not need to assemble the search results into tree data you can directly return the results requested by the client, otherwise you can query MySQL again through the cache to obtain the tree node relationship data, and finally the application service assembles the result data into the required data from the client. returned after the data.
  • the affiliation information includes a fourth organization affiliation ID
  • the ES index library includes a tree node information index table and a second organization equipment association index table
  • retrieval type is index database retrieval
  • the tree nodes corresponding to the tree node IDs of each hierarchy are associated according to the second node association relationship, and the association with the device data is established to obtain the second target hierarchy data.
  • the retrieval type is index database retrieval and the index database is an unrestricted hierarchical type
  • the data retrieval request is parsed to obtain the search conditions
  • the second organization's equipment association index table is retrieved according to the fourth organization's attribution ID, and the corresponding Device data and the branch ID of the hierarchical tree to which it belongs
  • search the tree node information index table according to the search conditions, and obtain multiple hierarchical tree node IDs that meet the search conditions
  • Step 206 return the first target level data or the second target level data to the client.
  • step 206 is similar to that of step 106, and will not be repeated here.
  • Step 207 when receiving the device reallocation request sent by the additional client, create an additional user identifier corresponding to the additional client;
  • Step 208 determine the corresponding additional device data according to the device reallocation request
  • Step 209 establishing the association between the additional user identifier and the additional device data, obtaining an additional ES index table and saving it to the ES index library.
  • the client is usually under a certain organization, and the same organization can include multiple client terminals at the same time, but due to business requirements, it may be necessary to allocate equipment for a single user.
  • the additional client when receiving When the device reallocation request is sent by the additional client, create an additional user ID corresponding to the additional client, and determine the additional device data required by the additional client according to the device reallocation request, such as device name, device ID, and device location , device label, device capability, etc., and then establish the association between the additional user identifier and the additional device data to obtain the additional ES index table.
  • the additional ES index table can be saved to the ES index library.
  • Table 8 shows an additional ES index table of the embodiment of the present invention:
  • the user's role or the above attribute data of the user are respectively extracted from the database when the user logs in, and then cached in the Redis cluster.
  • the subsequent user initiates a search request, it can be judged based on the value of the cached data that the corresponding Users search this part of the index, so as to realize the control of user data permissions synchronously.
  • the device data can be updated in the following ways: There are two main technical methods for the perception of device going offline events: Based on the establishment and disconnection of the socket through the TCP/IP communication connection, based on the reported data of the device heartbeat, the latter will receive the heartbeat data of the device regularly and if it has not been received after a certain period of time, it will be considered that the device has been offline. Both require a dedicated device access service, which can receive device communication requests through the MQTT protocol, a custom protocol based on Netty, or even a device heartbeat request of the HTTP protocol.
  • the device access service instance can first save the device id (continuously increasing number) and its status in the memory through ConcurrentHashMap, and then confirm the device When the connection is established or disconnected, or when the device heartbeat request is received, or the device request is not received after a timeout, set the device online status to true and false respectively.
  • ConcurrentHashMap When multiple devices have status changes at the same time, directly call ConcurrentHashMap concurrently through multiple threads
  • the put() method can modify the state of the corresponding device id.
  • the application service instance in the device access service cluster regularly refreshes these data to the multi-master and multi-slave Redis cluster at regular intervals (the interval is allowed to be configured), and Redis saves the online status of the device through its bitset, and the master instance of each Redis.
  • the key of the bitset is the id of the branch of the hierarchical tree; when the maximum number of levels is not limited, the key is the id of the device next to the upper node. If the id changes, the key cached here needs to be updated synchronously.
  • a group of tree branch ids or device IDs of adjacent nodes can be sent in batches through the pipeline mechanism of Redis to obtain the total number of online devices of a key through BITCOUNT, and then the application The service assembles the resulting data and returns it. If the user can only access some devices under one node, the Redis pipeline mechanism can be used to send batches of commands to get the online status of a device id through GETBIT, and the device corresponding to the table name of 1 is returned online, and then accumulated in the application service memory The total number of online devices is returned to the client.
  • the total number of devices of each node can be queried in the database and cached in the Redis cluster for subsequent query of the total number of devices under the node to be obtained directly from the cache.
  • the id of the tree branch or node can be used as the key through hash, and the total number of devices can be stored as an attribute.
  • the hash value of the corresponding key can be added or subtracted.
  • the total device data of any key in Redis is found to be empty, the data is loaded from the data source and cached in the hash of Redis.
  • the implementation effect of the user-side hierarchical tree is to open the hierarchical tree step by step, and only open the first-level nodes at the beginning by default, then the implementation is simpler: as long as the user double-clicks to open a node, directly open the next-level tree node,
  • the number of online devices can be obtained directly from Redis through the BITCOUNT command according to the id of this node. If the user can only access some of the devices, obtain the online status of each device from Redis in batches through the pipeline and accumulate them in memory. The result of accumulating the devices owned by the user is the total number of the current node. Number of online devices. Since the whole process is mainly carried out in memory, and the good scalability and availability of Redis multi-resident multi-slave cluster, the calculation of the number of online devices can not only be performed in real time, but also have better performance.
  • the user identifier corresponding to the client is obtained; the attribution information corresponding to the user identifier is queried from a preset database, and the database hierarchy type corresponding to the user identifier is determined;
  • the retrieval type corresponding to the data retrieval request if the retrieval type is a relational database retrieval, then retrieve the relational database corresponding to the database hierarchy type according to the attribution information, and obtain the first target level data; if the retrieval If the type is index database retrieval, then according to the data retrieval request and the ES index library corresponding to the attribution information retrieval database level type, obtain the second target level data; return the first target level data or the second target level data to the client.
  • hierarchical tree data can be obtained through one SQL statement or several queries. Furthermore, the efficiency of the data retrieval process under the amount of hundreds of millions of users is more effectively improved.
  • FIG. 5 is a structural block diagram of a hierarchical data retrieval device provided by Embodiment 3 of the present invention.
  • An embodiment of the present invention provides a hierarchical data retrieval device, including:
  • the user identification acquisition module 501 is configured to obtain a user identification corresponding to the user terminal in response to a login request sent by any user terminal;
  • a database hierarchy type determining module 502 configured to query the attribution information corresponding to the user identifier from a preset database, and determine the database hierarchy type corresponding to the user identifier;
  • a retrieval type obtaining module 503, configured to obtain the retrieval type corresponding to the data retrieval request when receiving the data retrieval request sent by the client;
  • a relational data retrieval module 504 configured to retrieve the relational database corresponding to the database hierarchy type according to the attribution information if the retrieval type is a relational database retrieval, to obtain the first target hierarchical data;
  • the index data retrieval module 505 is used for if the retrieval type is an index database retrieval, then according to the data retrieval request and the ES index library corresponding to the attribution information retrieval database hierarchy type, obtain the second target hierarchy data;
  • Hierarchical data return module 506, used to return the first target hierarchical data or the second target hierarchical data to the client.
  • the database hierarchy type determination module 502 is specifically used for:
  • the affiliation information includes the first affiliation organization ID
  • the relational database includes a plane hierarchical tree information table and a first organization equipment association information table; when the database hierarchy type is a limited hierarchy type, the relational data retrieval module 504 is specifically used for:
  • the retrieval type is a relational database retrieval, query the equipment information table of the first organization according to the ID of the first affiliation organization, and determine the corresponding equipment data and the branch ID of the hierarchical tree;
  • An association between branch data of each hierarchical tree and device data is established to obtain the first target hierarchical data.
  • the affiliation information includes a second affiliation organization ID and a superior node ID
  • the relational database includes a tree node information table, a tree node association relationship table and a second organization equipment association information table; when the database hierarchy type is an unrestricted hierarchy type,
  • the relational data retrieval module 504 is specifically used for:
  • the retrieval type is a relational database retrieval, query the equipment information table of the second organization according to the ID of the second affiliation organization, and determine the corresponding equipment data, upper node ID and hierarchical tree node ID;
  • the tree nodes corresponding to each tree node ID are associated according to the first node association relationship, and the association between the superior node ID and the device data is established to obtain the first target level data.
  • the affiliation information includes a third organization affiliation ID
  • the ES index storehouse includes a flat hierarchical tree index table and a first organization equipment association index table
  • the index data retrieval module 505 includes:
  • the retrieval type is indexed database retrieval
  • the data retrieval request is parsed to obtain the search condition and the target data format
  • the attribution ID of the third organization retrieve the organization's equipment association index table, and determine the corresponding equipment data
  • the association between the hierarchical tree information and the equipment data is established and converted into the target data format to obtain the second target hierarchical data.
  • the affiliation information includes a fourth organization affiliation ID
  • the ES index library includes a tree node information index table and a second organization equipment association index table
  • the index data retrieval module 505 includes:
  • retrieval type is index database retrieval
  • each hierarchical tree node ID respectively retrieve the second mechanism equipment association index table, and determine the second node association relationship corresponding to each hierarchical tree node ID;
  • the tree nodes corresponding to the tree node IDs of each hierarchy are associated according to the second node association relationship, and the association with the device data is established to obtain the second target hierarchy data.
  • the device also includes:
  • a user identification creation module configured to create an additional user identification corresponding to the additional user end when receiving a device reallocation request sent by the additional user end;
  • An additional equipment data determining module configured to determine corresponding additional equipment data according to the equipment reallocation request
  • the association establishing module is used to establish the association between the additional user identification and the additional device data, obtain the additional ES index table and save it to the ES index library.
  • the device also includes:
  • the field conversion module is used to execute the preset configuration file according to the preset update cycle, perform field conversion on the relational data in the relational database, and obtain the ES index field;
  • the field synchronization module is used to synchronize the ES index fields to the ES index library.
  • An embodiment of the present invention also provides an electronic device, including a memory and a processor, and a computer program is stored in the memory.
  • the processor executes the hierarchical data retrieval method according to any embodiment of the present invention. step.
  • the disclosed devices and methods can be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division. In actual implementation, there may be other division methods.
  • multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.
  • the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
  • the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present invention.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc., which can store program codes. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

L'invention concerne un procédé et un appareil de récupération de données hiérarchiques, ainsi qu'un dispositif. Le procédé consiste à : en réponse à une requête d'ouverture de session envoyée par n'importe quel terminal d'utilisateur, acquérir un identifiant d'utilisateur correspondant au terminal d'utilisateur (101) ; interroger, à partir d'une base de données prédéfinie, des informations d'attribution correspondant à l'identifiant d'utilisateur, puis déterminer un type hiérarchique de base de données correspondant à l'identifiant d'utilisateur (102) ; lorsqu'une requête de récupération de données envoyée par le terminal d'utilisateur est reçue, acquérir un type de récupération correspondant à la requête de récupération de données (103) ; si le type de récupération est une récupération de base de données relationnelle, récupérer, en fonction des informations d'attribution, une base de données relationnelle correspondant au type hiérarchique de base de données, de façon à obtenir des premières données hiérarchiques cibles (104) ; si le type de récupération est une récupération de base de données d'index, récupérer, en fonction de la requête de récupération de données et des informations d'attribution, une base de données d'index ES correspondant au type hiérarchique de base de données, de façon à obtenir des secondes données hiérarchiques cibles (105) ; et renvoyer les premières données hiérarchiques cibles ou les secondes données hiérarchiques cibles au terminal d'utilisateur (106). Sur la base de la simplification d'une architecture de récupération et d'un trajet de récupération, l'efficacité d'un processus de récupération de données pour des centaines de millions d'utilisateurs est efficacement améliorée.
PCT/CN2022/097345 2021-11-19 2022-06-07 Procédé et appareil de récupération de données hiérarchiques et dispositif WO2023087673A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111389373.9A CN114116716A (zh) 2021-11-19 2021-11-19 一种层次数据检索方法、装置和设备
CN202111389373.9 2021-11-19

Publications (1)

Publication Number Publication Date
WO2023087673A1 true WO2023087673A1 (fr) 2023-05-25

Family

ID=80439691

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/097345 WO2023087673A1 (fr) 2021-11-19 2022-06-07 Procédé et appareil de récupération de données hiérarchiques et dispositif

Country Status (2)

Country Link
CN (1) CN114116716A (fr)
WO (1) WO2023087673A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116484066A (zh) * 2023-06-21 2023-07-25 广东广宇科技发展有限公司 一种多种类数据处理方法
CN117094301A (zh) * 2023-10-17 2023-11-21 北京云枢创新软件技术有限公司 一种验证计划的创建系统
CN117290893A (zh) * 2023-09-25 2023-12-26 北京万里开源软件有限公司 一种基于数据标签的数据库访问方法及系统

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114116716A (zh) * 2021-11-19 2022-03-01 天翼数字生活科技有限公司 一种层次数据检索方法、装置和设备
CN114996297B (zh) * 2022-04-14 2023-09-26 建信金融科技有限责任公司 数据处理方法、装置、设备及介质
CN115033608B (zh) * 2022-08-12 2022-11-04 广东采日能源科技有限公司 一种储能系统信息的分级处理方法及系统
CN116756301B (zh) * 2023-08-17 2023-11-10 北京睿企信息科技有限公司 一种用户信息搜索的方法、电子设备及存储介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20010021929A1 (en) * 2000-02-21 2001-09-13 International Business Machines Corporation User-oriented method and system for database query
CN102769709A (zh) * 2012-06-25 2012-11-07 百度在线网络技术（北京）有限公司 一种用于向用户提供服务联系信息的方法与装置
CN110489445A (zh) * 2019-08-02 2019-11-22 四川宏力信息科技有限责任公司 一种基于多形态复合的海量数据快速查询方法
CN110580255A (zh) * 2018-06-08 2019-12-17 深圳艾派网络科技股份有限公司 一种存储并检索数据的方法以及系统
CN110837515A (zh) * 2019-11-06 2020-02-25 北京天融信网络安全技术有限公司 一种基于数据库的数据处理方法及电子设备
CN112347105A (zh) * 2020-11-06 2021-02-09 中国电子科技集团公司第三十八研究所 一种基于资源树的通用数据服务平台
CN114116716A (zh) * 2021-11-19 2022-03-01 天翼数字生活科技有限公司 一种层次数据检索方法、装置和设备

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20010021929A1 (en) * 2000-02-21 2001-09-13 International Business Machines Corporation User-oriented method and system for database query
CN102769709A (zh) * 2012-06-25 2012-11-07 百度在线网络技术（北京）有限公司 一种用于向用户提供服务联系信息的方法与装置
CN110580255A (zh) * 2018-06-08 2019-12-17 深圳艾派网络科技股份有限公司 一种存储并检索数据的方法以及系统
CN110489445A (zh) * 2019-08-02 2019-11-22 四川宏力信息科技有限责任公司 一种基于多形态复合的海量数据快速查询方法
CN110837515A (zh) * 2019-11-06 2020-02-25 北京天融信网络安全技术有限公司 一种基于数据库的数据处理方法及电子设备
CN112347105A (zh) * 2020-11-06 2021-02-09 中国电子科技集团公司第三十八研究所 一种基于资源树的通用数据服务平台
CN114116716A (zh) * 2021-11-19 2022-03-01 天翼数字生活科技有限公司 一种层次数据检索方法、装置和设备

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116484066A (zh) * 2023-06-21 2023-07-25 广东广宇科技发展有限公司 一种多种类数据处理方法
CN116484066B (zh) * 2023-06-21 2023-12-01 广东广宇科技发展有限公司 一种多种类数据处理方法
CN117290893A (zh) * 2023-09-25 2023-12-26 北京万里开源软件有限公司 一种基于数据标签的数据库访问方法及系统
CN117094301A (zh) * 2023-10-17 2023-11-21 北京云枢创新软件技术有限公司 一种验证计划的创建系统
CN117094301B (zh) * 2023-10-17 2024-01-26 北京云枢创新软件技术有限公司 一种验证计划的创建系统

Also Published As

Publication number Publication date
CN114116716A (zh) 2022-03-01

Similar Documents

Publication Publication Date Title
WO2023087673A1 (fr) Procédé et appareil de récupération de données hiérarchiques et dispositif
US11816126B2 (en) Large scale unstructured database systems
RU2409847C2 (ru) Отображение модели файловой системы в объект базы данных
US6502088B1 (en) Method and system for improved access to non-relational databases
EP2874077B1 (fr) Mémoire cache de base de données sans état
CN107451225A (zh) 用于半结构化数据的可缩放分析平台
JP2022534215A (ja) ハイブリッド・インデックス作成方法、システム、プログラム
US20030195889A1 (en) Unified relational database model for data mining
US20110238703A1 (en) Time in databases and applications of databases
WO2014182852A1 (fr) Système permettant de gérer les interrogations de graphe concernant les relations entre entités au moyen d'un index de graphe
CN102054025A (zh) 交通信息资源整合处理方法及系统
CA2717673A1 (fr) Detection et application des changements apportes aux schemas de bases de donnees sur les etats
CN102999563A (zh) 基于资源描述框架的网络资源语义检索方法及系统
US7624117B2 (en) Complex data assembly identifier thesaurus
US10762068B2 (en) Virtual columns to expose row specific details for query execution in column store databases
CN101833568A (zh) Web数据管理系统
EP4155964A1 (fr) Référentiel centralisé de métadonnées avec identifiants de pertinence
CN112699100A (zh) 一种基于元数据管理分析系统
Arputhamary et al. A review on big data integration
Bruckner et al. A framework for a multidimensional OLAP model using topic maps
Pokorny Databases in the 3rd millennium: trends and research directions
Senapati et al. A survey on data warehouse architecture
EP4170516A1 (fr) Éléments de métadonnées à identifiants persistants
Muñoz-Sánchez et al. Managing Physical Schemas in MongoDB Stores
KR100743927B1 (ko) 공간 데이터 웨어하우스에서 시공간 데이터의 집계정보관리방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22894216

Country of ref document: EP

Kind code of ref document: A1