CN112988857A - Service data processing method and device - Google Patents
Service data processing method and device Download PDFInfo
- Publication number
- CN112988857A CN112988857A CN201911272856.3A CN201911272856A CN112988857A CN 112988857 A CN112988857 A CN 112988857A CN 201911272856 A CN201911272856 A CN 201911272856A CN 112988857 A CN112988857 A CN 112988857A
- Authority
- CN
- China
- Prior art keywords
- attribute
- service data
- name
- attribute name
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/31—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
Abstract
The invention discloses a method and a device for processing service data, and relates to the technical field of computers. One embodiment of the method comprises: acquiring service data corresponding to one or more service systems, wherein the service data indicates one or more entity classes, and the entity classes have one or more attributes; acquiring an attribute name and an attribute type corresponding to the attribute from the service data; determining a column name corresponding to the attribute name in a database for storing the service data according to the attribute name and the attribute type, wherein the database is an unstructured database; and storing the service data into the database according to the attribute name and the column name corresponding to the attribute name. The embodiment unifies the storage structure of the service data and simplifies the processing flow of the service data.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for processing service data.
Background
In order to analyze the service data generated by the service system, the service data is often pushed to a big data analysis platform. At present, a common solution is to establish a corresponding service data table in a service system to form structured data, and then store the structured data in a relational database such as MYSQL. Correspondingly, in a big data analysis platform or a corresponding data processing tool, a corresponding ETL (Extract-Transform-Load) strategy is configured, and data is captured from a database for analysis.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art: data stored by the business system and used for being analyzed by the big data analysis platform is not valuable to the business system, but occupies and consumes database resources of the business system and the like, so that resource waste is caused to a certain extent; the service data of different service systems have different data structures, different data ETL strategies need to be configured in a large data platform or corresponding data processing tools, and the operation is complex.
Disclosure of Invention
In view of this, the present invention provides a method and an apparatus for processing service data, which can store corresponding service data of different service systems by using a uniform data structure, configure a uniform data ETL policy, and are simple to operate, and a database storing service data is set independently of a service system, and does not need to occupy service system resources, thereby improving processing efficiency of the service system.
In order to achieve the above object, according to an aspect of the present invention, there is provided a method for processing service data, including:
acquiring service data corresponding to one or more service systems, wherein the service data indicates one or more entity classes, and the entity classes have one or more attributes;
acquiring an attribute name and an attribute type corresponding to the attribute from the service data;
determining a column name corresponding to the attribute name in a database for storing the service data according to the attribute name and the attribute type, wherein the database is an unstructured database;
and storing the service data into the database according to the attribute name and the column name corresponding to the attribute name.
Optionally, the acquiring service data corresponding to one or more service systems includes:
acquiring the service data pushed by the one or more service systems;
pushing the service data to a message server;
and acquiring the service data from the message server.
Optionally, the determining, according to the attribute name and the attribute type, a column name of the attribute name corresponding to a database for storing the service data includes:
sorting the attribute names corresponding to the same attribute type to generate attribute sequence numbers corresponding to the attribute names;
and generating a column name corresponding to the attribute name according to the attribute type and the attribute sequence number corresponding to the attribute name.
Optionally, the method further comprises:
before generating the column name corresponding to the attribute name, judging whether the attribute name has a corresponding preset column name;
and under the condition that the attribute name has a corresponding preset column name, the preset column name is the column name corresponding to the attribute name.
Optionally, the method further comprises:
before generating the column name corresponding to the attribute name, judging whether the attribute name has a corresponding preset attribute sequence number;
and under the condition that the attribute name has a corresponding preset attribute sequence number, generating a column name corresponding to the attribute name according to the attribute name and the preset attribute sequence number.
Optionally, the method further comprises:
and storing the column name corresponding to the attribute name to a local cache so as to obtain the column name corresponding to the attribute name from the local cache under the condition of obtaining the attribute name and the attribute type corresponding to the attribute from the service data.
Optionally, the database is a Hbase database.
Optionally, service data corresponding to one or more service systems is obtained through an RPC remote call interface.
To achieve the above object, according to another aspect of the present invention, there is provided a service data processing apparatus, including: the system comprises a service data acquisition module, a service data processing module and a service data storage module; wherein the content of the first and second substances,
the service data acquisition module is used for acquiring service data corresponding to one or more service systems, wherein the service data indicates one or more entity classes, and the entity classes have one or more attributes;
the service data processing module is used for acquiring an attribute name and an attribute type corresponding to the attribute from the service data; determining a column name corresponding to the attribute name in a database for storing the service data according to the attribute name and the attribute type, wherein the database is an unstructured database;
and the service data storage module is used for storing the service data into the database according to the attribute name and the column name corresponding to the attribute name.
Optionally, the acquiring service data corresponding to one or more service systems includes:
acquiring the service data pushed by the one or more service systems;
pushing the service data to a message server;
and acquiring the service data from the message server.
Optionally, the determining, according to the attribute name and the attribute type, a column name of the attribute name corresponding to a database for storing the service data includes:
sorting the attribute names corresponding to the same attribute type to generate attribute sequence numbers corresponding to the attribute names;
and generating a column name corresponding to the attribute name according to the attribute type and the attribute sequence number corresponding to the attribute name.
Optionally, the service data processing module is further configured to,
before generating the column name corresponding to the attribute name, judging whether the attribute name has a corresponding preset column name;
and under the condition that the attribute name has a corresponding preset column name, the preset column name is the column name corresponding to the attribute name.
Optionally, the service data processing module is further configured to,
before generating the column name corresponding to the attribute name, judging whether the attribute name has a corresponding preset attribute sequence number;
and under the condition that the attribute name has a corresponding preset attribute sequence number, generating a column name corresponding to the attribute name according to the attribute name and the preset attribute sequence number.
Optionally, the service data processing module is further configured to,
and storing the column name corresponding to the attribute name to a local cache so as to obtain the column name corresponding to the attribute name from the local cache under the condition of obtaining the attribute name and the attribute type corresponding to the attribute from the service data.
Optionally, the database is a Hbase database.
Optionally, the service data acquiring module is configured to acquire service data corresponding to one or more service systems through an RPC remote call interface.
To achieve the above object, according to still another aspect of the present invention, there is provided an electronic device for processing service data, comprising: one or more processors; a storage device, configured to store one or more programs, which when executed by the one or more processors, cause the one or more processors to implement any one of the above-described methods for processing service data.
To achieve the above object, according to still another aspect of the present invention, there is provided a computer-readable medium on which a computer program is stored, the program, when executed by a processor, implementing any one of the methods of processing service data as described above.
The invention has the following advantages or beneficial effects: by replacing the attribute names in the service data with column names in the unstructured database, the storage structures of the service data of different service systems are unified, the ETL (extract transform load) strategy of the data is convenient to configure, and the flow is simplified; the method reduces the influence of the service data volume of the service system on the service system by sending the service data of the service system to the message server and then acquiring the service data from the message server; in addition, the special processing of different service data is realized by presetting column names, attribute serial numbers and the like; and the attribute names and the corresponding column names are stored in a local cache, so that the speed of determining or searching the existing column names of the attribute names is increased, and the efficiency of processing the service data is improved.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
fig. 1 is a schematic diagram of a main flow of a business data processing method according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a main flow of another business data processing method according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a main flow of a further business data processing method according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a main flow of a processing method of further service data according to an embodiment of the present invention;
fig. 5 is a schematic diagram of main blocks of a traffic data processing apparatus according to an embodiment of the present invention;
FIG. 6 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 7 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of a main flow of a business data processing method according to an embodiment of the present invention; as shown in fig. 1, the method for processing service data may specifically include the following steps:
step S101, obtaining service data corresponding to one or more service systems, wherein the service data indicates one or more entity classes, and the entity classes have one or more attributes.
The service system includes, but is not limited to, any service system that performs service processing according to actual needs, such as a logistics information system, an e-commerce system, and the like, and the service data refers to any data generated during the operation of the service system. The entity class refers to abstraction or modeling of a real object during business system design, for example, in a shopping system, a commodity, an order and the like are entity classes in the shopping system, the entity classes can have attributes of the entity classes, such as a commodity entity class, and can have multiple attributes of a commodity name, a commodity price, a commodity production date and the like, and attribute values corresponding to each attribute constitute business data. The commodity name, the commodity price and the like are attribute names of each attribute, each attribute has a corresponding attribute type according to different contents, namely a character string type, an integer type and the like, if the attribute type corresponding to the commodity price is the integer type, and the commodity name is the character string type.
In an optional implementation manner, service data corresponding to one or more service systems is acquired through an RPC remote call interface. Therefore, the acquisition and processing of the service data can be decoupled from the service system, and a device or a tool for specifically processing the service data can be set independently of the service system.
In an optional implementation manner, the obtaining service data corresponding to one or more service systems includes: acquiring the service data pushed by the one or more service systems; pushing the service data to a message server; and acquiring the service data from the message server. The message server can be independently arranged independent of the service data processing device, so that the pressure caused by the sharp increase of the service data volume pushed by the service system to the service system or the service data processing device can be buffered, the normal operation of the service system is facilitated, and the isolation between the service system and the service data processing device is further realized. In addition, by means of the mode that the service system actively pushes the service data, resources consumed by analyzing the service data request and the like when the service system sends the service data according to the service data request and the like are avoided, and pressure of obtaining the service data on the service system is further reduced.
Step S102, obtaining the attribute name and attribute type corresponding to the attribute from the service data. Specifically, taking an entity class as product information as an example for explanation, the attribute name and the corresponding attribute type of the entity class are as follows:
product name, character string type
Product description, string type
Number of products, type of shaping
Price and type of product
That is, the attributes of the string type specifically included in the entity class are product information and product description, respectively, and the attributes of the integer type included in the entity class are product quantity and product price, respectively.
Step S103, determining a column name corresponding to the attribute name in a database for storing the service data according to the attribute name and the attribute type, wherein the database is an unstructured database.
That is, the attribute names, the column names corresponding to the attribute names, the attribute values corresponding to the attribute names in the service data, and the like are stored in the column database. Databases that may be used to store business data include, but are not limited to: hbase, Sybase IQ, infinibb, infobright, ParaAccel, Vertica and the like. In a preferred embodiment of the invention, the database is the Hbase database. The Hbase database is a distributed and column-oriented open source database, provides Bigtable-like capability above Hadoop, is a sub-item of Hadoop items of Apache, but is different from a general relational database, which is a database suitable for unstructured data storage, and is a column-based rather than row-based storage schema.
It can be understood that, according to the attribute name and the attribute type, there are various ways to determine the corresponding column name of the attribute name in the database for storing the business data, including but not limited to: generating a column name corresponding to the attribute name according to a preset rule, directly using a preset column name appointed according to an actual requirement, using a preset attribute sequence number appointed according to the actual requirement and the like to generate the column name corresponding to the attribute name.
In an optional implementation manner, the determining, according to the attribute name and the attribute type, a column name of the attribute name corresponding to a database for storing the service data includes: sorting the attribute names corresponding to the same attribute type to generate attribute sequence numbers corresponding to the attribute names; and generating a column name corresponding to the attribute name according to the attribute type and the attribute sequence number corresponding to the attribute name.
Specifically, taking an entity class as the product information as an example for explanation, the attributes of the string types specifically included in the entity class are product information and product description, and the attributes of the integer types included are product quantity and product price, respectively, and the attribute names corresponding to the same attribute type are sorted, and if the attributes are sorted in the dictionary order, the following examples of the column names corresponding to the attribute types and the attribute name orders are generated:
the product name is as follows: string _0
Product description: character string _1
Product quantity: integer _0
The product price is as follows: integer _1
Therefore, the column names corresponding to different attribute names included in the same entity class can be ensured to be different. It can be understood that the above generated attribute names correspond to column names: string _0, string _1, integer _0, integer _1, etc. are merely exemplary and may also be represented as string 0, string/0, string-0, etc. as desired.
In an optional implementation manner, before generating a column name corresponding to the attribute name, determining whether the attribute name has a corresponding preset column name; and under the condition that the attribute name has a corresponding preset column name, the preset column name is the column name corresponding to the attribute name.
Specifically, still taking the entity class as the product information for example, if the attribute product information and the product description of the string type specifically included in the entity class are specified as the preset column name A, B and the like according to the requirements of the user or the service system, the preset column name A, B is directly used as the column name corresponding to the attribute name, which is specifically as follows:
the product name is as follows: a. the
Product description: b is
Product quantity: integer _0
The product price is as follows: integer _1
Therefore, a customized mode for determining the column names corresponding to the attribute names can be provided, certain expansibility is achieved, and customized requirements of different users or service systems can be met.
In addition, before generating the column name corresponding to the attribute name, judging whether the attribute name has a corresponding preset attribute serial number; and under the condition that the attribute name has a corresponding preset attribute sequence number, generating a column name corresponding to the attribute name according to the attribute name and the preset attribute sequence number. That is, it is not only possible to directly define or designate a column name corresponding to an attribute name, but also possible to define and generate a column name corresponding to an attribute name as a used attribute number. Specifically, still taking an entity class as the product information for example, if the attribute product information and the product description of the string type specifically included in the entity class have been specified as the preset attribute numbers 1-1, 1-2 and the like according to the requirements of the user or the service system, the preset attribute numbers 1-1, 1-2 are directly used as the column names corresponding to the attribute names, which is specifically as follows:
the product name is as follows: string _1-1
Product description: string _1-2
Product quantity: integer _0
The product price is as follows: integer _1
Therefore, a customized mode for determining the column names corresponding to the attribute names can be further provided, certain expansibility is achieved, and customized requirements of different users or service systems can be met.
In an optional implementation manner, the column name corresponding to the attribute name is stored in a local cache, so that when the attribute name and the attribute type corresponding to the attribute are acquired from the service data, the column name corresponding to the attribute name is acquired from the local cache. Based on the above, the column name corresponding to the attribute name determined according to the attribute name and the attribute type and stored in the local cache can be utilized, so that the efficiency of obtaining the column name corresponding to the attribute name can be rapidly improved. In addition, because the storage space of the local cache is limited, the storage time, the storage priority and the like of the column names corresponding to the stored attribute names can be set according to actual conditions, and regular cleaning is performed, so that the storage space of the local cache is effectively utilized, and meanwhile, the local cache is prevented from generating bugs and the like.
And step S104, storing the service data to the database according to the attribute name and the column name corresponding to the attribute name.
Specifically, still taking the entity class as the product information as an example for explanation, the column name corresponding to the attribute name determined according to the attribute name and the attribute type is as follows:
the product name is as follows: string _0
Product description: character string _1
Product quantity: integer _0
The product price is as follows: integer _1
When the service data is stored in the database such as Hbase, the attribute name and the column name corresponding to the attribute name can be stored in the form of key-vakue, where key is the attribute name such as the product name and the product description, and value is the column name of the attribute name corresponding to the database such as Hbase, i.e. character string _0, character string _1, and the like; meanwhile, the attribute name and the corresponding attribute value in the service data are stored in a key-vakue form, or the column name and the corresponding attribute value corresponding to the attribute name are stored. Therefore, the business data from different business systems are stored through the mapping relation between the attribute names and the column names corresponding to the attribute names and one or more databases such as Hbase with the same structure, so that the business data have the same storage structure, and the same ETL (Extract-Transform-Load) strategy can be configured when the business data are subjected to big data analysis, thereby simplifying the flow and improving the efficiency.
Based on the above embodiment, the method for processing service data provided by the present invention unifies the storage structures of service data of different service systems by replacing the attribute names in the service data with the column names in the unstructured database, thereby facilitating the configuration of the ETL policy of data and simplifying the process; the method reduces the influence of the service data volume of the service system on the service system by sending the service data of the service system to the message server and then acquiring the service data from the message server; in addition, the special processing of different service data is realized by presetting column names, attribute serial numbers and the like; and the attribute names and the corresponding column names are stored in a local cache, so that the speed of determining or searching the existing column names of the attribute names is increased, and the efficiency of processing the service data is improved.
Referring to fig. 2, on the basis of the foregoing embodiment, an embodiment of the present invention provides another method for processing service data, where the method specifically includes the following steps:
step S201, obtaining the service data pushed by the one or more service systems. The obtained service data indicates one or more entity classes having one or more attributes. Specifically, service data corresponding to one or more service systems is acquired through an RPC remote call interface. Therefore, the acquisition and processing of the service data can be decoupled from the service system, and a device or a tool for specifically processing the service data can be set independently of the service system.
Step S202, the service data is pushed to a message server.
The message server can be set independently of the service system, so that the capacity of receiving the service data is prevented from changing due to the change of the quantity of the service data pushed by the service system, and the normal operation of the service system is ensured to be improved.
Step S203, obtaining the service data from the message server.
Step S204, obtaining the attribute name and the attribute type corresponding to the attribute from the service data.
Step S205, determining a column name corresponding to the attribute name in a database for storing the service data according to the attribute name and the attribute type, where the database is an unstructured database. It can be understood that, according to the attribute name and the attribute type, there are various ways to determine the corresponding column name of the attribute name in the database for storing the business data, including but not limited to: generating a column name corresponding to the attribute name according to a preset rule, directly using a preset column name appointed according to an actual requirement, using a preset attribute sequence number appointed according to the actual requirement and the like to generate the column name corresponding to the attribute name.
Step S206, storing the service data to the database according to the attribute name and the column name corresponding to the attribute name. Databases that may be employed include, but are not limited to: hbase, Sybase IQ, infinibb, infobright, ParaAccel, Vertica, etc. column databases.
Referring to fig. 3, on the basis of the foregoing embodiment, an embodiment of the present invention provides another service data processing method, which may specifically include the following steps:
step S301, obtaining service data corresponding to one or more service systems, where the service data indicates one or more entity classes, and the entity classes have one or more attributes.
Step S302, obtaining an attribute name and an attribute type corresponding to the attribute from the service data.
Step S303, sorting the attribute names corresponding to the same attribute type to generate an attribute sequence number corresponding to the attribute name.
Step S304, generating a column name corresponding to the attribute name according to the attribute type and the attribute sequence number corresponding to the attribute name. The column name corresponding to the attribute name refers to a column name corresponding to the attribute name in a database for storing the service data, and the database is an unstructured database such as Hbas.
Step S305, storing the service data in the database according to the attribute name and the column name corresponding to the attribute name.
Referring to fig. 4, on the basis of the foregoing embodiment, an embodiment of the present invention provides another service data processing method, which may specifically include the following steps:
on the basis of acquiring the service data corresponding to one or more service systems, the following steps are continuously executed, wherein the acquired service data indicate one or more entity classes, and the entity classes have one or more attributes.
Step S401, obtaining an attribute name and an attribute type corresponding to the attribute from the service data.
Step S402, determining whether the local cache has a column name corresponding to the attribute name, if yes, performing step S408, and if not, performing step S403. Specifically, the column name corresponding to the attribute name generated when the service data is processed before is stored in the local cache, so that the column name corresponding to the attribute name can be acquired from the local cache when the attribute name and the attribute type corresponding to the attribute are acquired from new service data. Based on the above, the column name corresponding to the attribute name determined according to the attribute name and the attribute type and stored in the local cache can be utilized, so that the efficiency of obtaining the column name corresponding to the attribute name can be rapidly improved. In addition, because the storage space of the local cache is limited, the storage time, the storage priority and the like of the column names corresponding to the stored attribute names can be set according to actual conditions, and regular cleaning is performed, so that the storage space of the local cache is effectively utilized, and meanwhile, the local cache is prevented from generating bugs and the like.
Step S403, determining whether the attribute name has a corresponding preset column name, if yes, performing step S404, and if not, performing step S405.
Step S404, the preset column name is a column name corresponding to the attribute name. Therefore, a customized mode for determining the column names corresponding to the attribute names can be provided, certain expansibility is achieved, and customized requirements of different users or service systems can be met.
Step S405, determining whether the attribute name has a corresponding preset attribute sequence number, if yes, performing step S406, and if not, performing step S407.
Step S406, generating a column name corresponding to the attribute name according to the attribute name and the preset attribute sequence number. Therefore, a customized mode for determining the column names corresponding to the attribute names can be further provided, certain expansibility is achieved, and customized requirements of different users or service systems can be met.
Step S407, generating a column name corresponding to the attribute name according to the attribute type and the attribute sequence number corresponding to the attribute name. That is, the attribute names corresponding to the same attribute type are sorted to generate the attribute sequence numbers corresponding to the attribute names; and generating a column name corresponding to the attribute name according to the attribute type and the attribute sequence number corresponding to the attribute name.
Step S408, storing the service data into the database according to the attribute name and the column name corresponding to the attribute name.
Referring to fig. 5, on the basis of the foregoing embodiment, an embodiment of the present invention provides a device 500 for processing service data, including: a service data acquisition module 501, a service data processing module 502 and a service data storage module 503; wherein the content of the first and second substances,
the service data acquiring module 501 is configured to acquire service data corresponding to one or more service systems, where the service data indicates one or more entity classes, and the entity classes have one or more attributes;
the service data processing module 502 is configured to obtain an attribute name and an attribute type corresponding to the attribute from the service data; determining a column name corresponding to the attribute name in a database for storing the service data according to the attribute name and the attribute type, wherein the database is an unstructured database;
the service data storage module 503 is configured to store the service data in the database according to the attribute name and the column name corresponding to the attribute name.
In an optional implementation manner, the obtaining service data corresponding to one or more service systems includes: acquiring the service data pushed by the one or more service systems; pushing the service data to a message server; and acquiring the service data from the message server.
In an optional implementation manner, the determining, according to the attribute name and the attribute type, a column name of the attribute name corresponding to a database for storing the service data includes: sorting the attribute names corresponding to the same attribute type to generate attribute sequence numbers corresponding to the attribute names; and generating a column name corresponding to the attribute name according to the attribute type and the attribute sequence number corresponding to the attribute name.
In an optional implementation manner, the service data processing module 502 is further configured to,
before generating the column name corresponding to the attribute name, judging whether the attribute name has a corresponding preset column name; and under the condition that the attribute name has a corresponding preset column name, the preset column name is the column name corresponding to the attribute name.
In an optional implementation manner, the service data processing module 502 is further configured to, before generating a column name corresponding to the attribute name, determine whether the attribute name has a corresponding preset attribute sequence number; and under the condition that the attribute name has a corresponding preset attribute sequence number, generating a column name corresponding to the attribute name according to the attribute name and the preset attribute sequence number.
In an optional implementation manner, the service data processing module 502 is further configured to store the column name corresponding to the attribute name in a local cache, so as to obtain the column name corresponding to the attribute name from the local cache when the attribute name and the attribute type corresponding to the attribute are obtained from the service data.
In an alternative embodiment, the database is the Hbase database.
In an optional implementation manner, the service data obtaining module 501 is configured to obtain service data corresponding to one or more service systems through an RPC remote call interface.
Fig. 6 shows an exemplary system architecture 600 of a business data processing method or a business data device to which an embodiment of the present invention may be applied.
As shown in fig. 6, the system architecture 600 may include terminal devices 601, 602, 603, a network 604, and a server 605. The network 604 serves to provide a medium for communication links between the terminal devices 601, 602, 603 and the server 605. Network 604 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.
A user may use the terminal devices 601, 602, 603 to interact with the server 605 via the network 604 to receive or send messages or the like. Various communication client applications, such as shopping applications, web browser applications, search applications, instant messaging tools, mailbox clients, social platform software, and the like, may be installed on the terminal devices 601, 602, and 603.
The terminal devices 601, 602, 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 605 may be a server that provides various services, such as a background management server that supports shopping websites browsed by users using the terminal devices 601, 602, and 603. The background management server can analyze and process the received data such as the product information inquiry request and feed back the processing result to the terminal equipment.
It should be noted that the method for processing the service data provided by the embodiment of the present invention is generally executed by the server 605, and accordingly, the processing device for the service data is generally disposed in the server 605.
It should be understood that the number of terminal devices, networks, and servers in fig. 6 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 7, shown is a block diagram of a computer system 700 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 7 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU)701, which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data necessary for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
The following components are connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 701.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor comprises a service data acquisition module, a service data processing module and a service data storage module. The names of the modules do not form a limitation on the modules themselves in some cases, for example, the service data acquiring module may also be described as a "module for acquiring service data corresponding to one or more service systems".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: acquiring service data corresponding to one or more service systems, wherein the service data indicates one or more entity classes, and the entity classes have one or more attributes; acquiring an attribute name and an attribute type corresponding to the attribute from the service data; determining a column name corresponding to the attribute name in a database for storing the service data according to the attribute name and the attribute type, wherein the database is an unstructured database; and storing the service data into the database according to the attribute name and the column name corresponding to the attribute name.
According to the technical scheme of the embodiment of the invention, the attribute names in the service data are replaced by the column names in the unstructured database, so that the storage structures of the service data of different service systems are unified, the ETL (extract transform load) strategy of the data is convenient to configure, and the flow is simplified; the method reduces the influence of the service data volume of the service system on the service system by sending the service data of the service system to the message server and then acquiring the service data from the message server; in addition, the special processing of different service data is realized by presetting column names, attribute serial numbers and the like; and the attribute names and the corresponding column names are stored in a local cache, so that the speed of determining or searching the existing column names of the attribute names is increased, and the efficiency of processing the service data is improved.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (11)
1. A method for processing service data is characterized by comprising the following steps:
acquiring service data corresponding to one or more service systems, wherein the service data indicates one or more entity classes, and the entity classes have one or more attributes;
acquiring an attribute name and an attribute type corresponding to the attribute from the service data;
determining a column name corresponding to the attribute name in a database for storing the service data according to the attribute name and the attribute type, wherein the database is an unstructured database;
and storing the service data into the database according to the attribute name and the column name corresponding to the attribute name.
2. The method for processing service data according to claim 1, wherein the obtaining service data corresponding to one or more service systems includes:
acquiring the service data pushed by the one or more service systems;
pushing the service data to a message server;
and acquiring the service data from the message server.
3. The method for processing business data according to claim 1, wherein the determining, according to the attribute name and the attribute type, a column name corresponding to the attribute name in a database for storing the business data includes:
sorting the attribute names corresponding to the same attribute type to generate attribute sequence numbers corresponding to the attribute names;
and generating a column name corresponding to the attribute name according to the attribute type and the attribute sequence number corresponding to the attribute name.
4. The method for processing service data according to claim 3, further comprising:
before generating the column name corresponding to the attribute name, judging whether the attribute name has a corresponding preset column name;
and under the condition that the attribute name has a corresponding preset column name, the preset column name is the column name corresponding to the attribute name.
5. The method for processing service data according to claim 3, further comprising:
before generating the column name corresponding to the attribute name, judging whether the attribute name has a corresponding preset attribute sequence number;
and under the condition that the attribute name has a corresponding preset attribute sequence number, generating a column name corresponding to the attribute name according to the attribute name and the preset attribute sequence number.
6. The method for processing service data according to any one of claims 1 to 5, further comprising:
and storing the column name corresponding to the attribute name to a local cache so as to obtain the column name corresponding to the attribute name from the local cache under the condition of obtaining the attribute name and the attribute type corresponding to the attribute from the service data.
7. The method for processing service data according to claim 1,
the database is an Hbase database.
8. The method for processing service data according to claim 2,
and acquiring service data corresponding to one or more service systems through the RPC remote call interface.
9. A device for processing service data, comprising: the system comprises a service data acquisition module, a service data processing module and a service data storage module; wherein the content of the first and second substances,
the service data acquisition module is used for acquiring service data corresponding to one or more service systems, wherein the service data indicates one or more entity classes, and the entity classes have one or more attributes;
the service data processing module is used for acquiring an attribute name and an attribute type corresponding to the attribute from the service data; determining a column name corresponding to the attribute name in a database for storing the service data according to the attribute name and the attribute type, wherein the database is an unstructured database;
and the service data storage module is used for storing the service data into the database according to the attribute name and the column name corresponding to the attribute name.
10. An electronic device for processing business data, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-8.
11. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911272856.3A CN112988857A (en) | 2019-12-12 | 2019-12-12 | Service data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911272856.3A CN112988857A (en) | 2019-12-12 | 2019-12-12 | Service data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112988857A true CN112988857A (en) | 2021-06-18 |
Family
ID=76332079
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911272856.3A Pending CN112988857A (en) | 2019-12-12 | 2019-12-12 | Service data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112988857A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113342686A (en) * | 2021-06-30 | 2021-09-03 | 建信金融科技有限责任公司 | Test data generation method and device |
-
2019
- 2019-12-12 CN CN201911272856.3A patent/CN112988857A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113342686A (en) * | 2021-06-30 | 2021-09-03 | 建信金融科技有限责任公司 | Test data generation method and device |
| CN113342686B (en) * | 2021-06-30 | 2023-01-10 | 中国建设银行股份有限公司 | Test data generation method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110019080B (en) | Data access method and device | |
| CN111190888A (en) | Method and device for managing graph database cluster | |
| CN110689268B (en) | Method and device for extracting indexes | |
| CN110795315A (en) | Method and device for monitoring service | |
| CN110858202A (en) | Method and device for generating where clause in database query statement | |
| CN110909022A (en) | Data query method and device | |
| CN112818026A (en) | Data integration method and device | |
| CN113190517B (en) | Data integration method and device, electronic equipment and computer readable medium | |
| CN111241189A (en) | Method and device for synchronizing data | |
| CN113761565B (en) | Data desensitization method and device | |
| CN113190558A (en) | Data processing method and system | |
| CN112988857A (en) | Service data processing method and device | |
| CN110704481B (en) | Method and device for displaying data | |
| CN112948138A (en) | Method and device for processing message | |
| CN113704242A (en) | Data processing method and device | |
| CN107665241B (en) | Real-time data multi-dimensional duplicate removal method and device | |
| CN113377808A (en) | SQL optimization method and device | |
| CN112579673A (en) | Multi-source data processing method and device | |
| CN113742321A (en) | Data updating method and device | |
| CN112925772A (en) | Data dynamic splitting method and device | |
| CN113779018A (en) | Data processing method and device | |
| CN113495891A (en) | Data processing method and device | |
| CN112783914A (en) | Statement optimization method and device | |
| CN112862554A (en) | Order data processing method and device | |
| CN112148705A (en) | Data migration method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |