CN111563112A - Data search and display system based on cross-border trade big data - Google Patents
Data search and display system based on cross-border trade big data Download PDFInfo
- Publication number
- CN111563112A CN111563112A CN202010365885.0A CN202010365885A CN111563112A CN 111563112 A CN111563112 A CN 111563112A CN 202010365885 A CN202010365885 A CN 202010365885A CN 111563112 A CN111563112 A CN 111563112A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- library
- target
- border trade
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/31—Indexing; Data structures therefor; Storage structures
- G06F16/313—Selection or weighting of terms for indexing
Abstract
The utility model provides a data search and display system based on cross border trade big data relates to data processing technology field, and this system includes: firstly, a data searching module searches data of different types of data storage systems to obtain original cross-border trade data; then the data storage module classifies the original cross-border trade data, and distributes and stores the original cross-border trade data in different resource libraries according to classification results; and finally, the data query module determines and displays the target data in the resource library according to the query information of the user. The invention can improve the diversity of data search and better meet the individual requirements of user data retrieval.
Description
Technical Field
The disclosure relates to the technical field of data processing, in particular to a data searching and displaying system based on cross-border trade big data.
Background
Generally, a search index only supports chassis data of a single platform, however, at present, a large amount of data is distributed in each dispersed system, and data dispersed in different systems cannot be searched uniformly so as to be integrated in one platform. In addition, the requirements for data search are also different for different user groups, such as: the angle of data analysis, the requirement of data retrieval, the facing business requirement and the like are all different; however, the content of the data retrieval result is relatively fixed at present, and cannot meet the personalized demand of the user.
Disclosure of Invention
In order to solve the technical problems or at least partially solve the technical problems, the present disclosure provides a data search and display system based on cross-border trade big data, which can not only improve the diversity of data search, but also better meet the personalized requirements of user data retrieval.
The present disclosure provides a data search and presentation system based on cross-border trade big data, including: the data search module is used for searching data of different types of data storage systems to obtain original cross-border trade data; the data storage module is used for classifying the original cross-border trade data and storing the original cross-border trade data in different resource libraries in a distributed manner according to a classification result; and the data query module is used for acquiring query information of a user, determining target data in different resource libraries according to the query information, and displaying the target data.
Further, the raw cross-border trade data comprises: bottom layer raw data and unstructured data; the resource pool comprises: a data chassis library, an analysis subject library and a full text retrieval library; the data storage module includes: the first data processing unit is used for performing data processing operation on the bottom layer original data according to the service type to obtain first data and storing the first data in a data chassis library in an Open Data Processing Service (ODPS) database; wherein the data processing operations comprise: at least one of a data screening operation, a data conversion operation, and a data cleansing operation; the second data processing unit is used for screening the first data in the data chassis library according to a service theme to obtain second data and storing the second data in an analysis theme library in the ODPS database; the data synchronization unit is used for synchronizing the data chassis library and the analysis subject library into an automatic equipment specification (ADS) database; and the third data processing unit is used for extracting data from the unstructured data and the first data and the second data stored in the ODPS database to obtain third data, and storing the third data in a full-text search library.
Further, the data query module is further configured to: identifying the data type of the data to be queried in the query information; determining a target database in a plurality of resource libraries according to the data type; and performing data retrieval in the target database according to the query information to obtain target data.
Further, the data query module is further configured to: acquiring resource library selection information of a user;
determining a target database in the ODPS database and the ADS database according to the resource library selection information; and performing data retrieval in the target database according to the query information to obtain target data.
Further, the data query module is further configured to: performing data retrieval in a preset initial database according to the query information to obtain candidate data; wherein the initial database is in the ODPS database or the ADS database; monitoring whether a switching operation of a user is received; if not, the candidate data is taken as target data; if so, switching the initial database into a target database according to the switching operation; the target database is a database which is not preset as the initial database in the ODPS database or the ADS database; and performing data retrieval in the target database according to the query information to obtain the target data.
Further, the system also comprises a capacity expansion module, and the capacity expansion module is used for: acquiring the number of the nodes of the data nodes in the current resource library after the data nodes are added; the current resource library comprises a newly added data node and an initial data node; performing split expansion on the data on the initial data node in the current resource library according to the number of the data nodes to obtain expanded fragment data; and redistributing the expanded fragment data in the current resource library so that the fragment number of the fragment data on each of the newly added data node and the initial data node meets a preset condition.
Further, the data search module is further configured to: and acquiring a preset data search strategy and rule, and searching data of different types of data storage systems according to the data search strategy and rule. .
The disclosure provides a data searching and displaying method based on cross-border trade big data, which comprises the following steps: data searching is carried out on different types of data storage systems to obtain original cross-border trade data; classifying the original cross-border trade data, and storing the original cross-border trade data in different resource libraries in a distributed manner according to classification results; acquiring query information of a user, determining target data in different resource libraries according to the query information, and displaying the target data.
The present disclosure provides an electronic device, including: a processor and a storage device; the storage means has stored thereon a computer program which, when executed by the processor, performs the method described above.
The present disclosure provides a computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, performs the steps of the above-mentioned method.
Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has the following advantages:
the embodiment of the disclosure provides a data searching and displaying system based on cross-border trade big data, which comprises a data searching module, a display module and a display module, wherein the data searching module is used for searching data of different types of data storage systems to obtain original cross-border trade data; then the data storage module classifies the original cross-border trade data, and distributes and stores the original cross-border trade data in different resource libraries according to classification results; and finally, the data query module determines and displays the target data in the resource library according to the query information of the user. In the mode, the diversification and richness of the original cross-border trade data search can be improved by searching data of different types of data storage systems, and the diversified data access requirement is better improved; original cross-border trade data are stored in different resource libraries according to classification results, and data storage can be flexibly and pertinently realized; meanwhile, when the user retrieves data, personalized data retrieval can be realized based on diversified resource libraries. Therefore, the method can improve the diversity of data search and better meet the individual requirements of user data retrieval.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present disclosure, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
FIG. 1 is a block diagram of a cross-border trade big data based data search and presentation system according to an embodiment of the present disclosure;
FIG. 2 is a schematic diagram of data flow provided by an embodiment of the present disclosure;
fig. 3 is a diagram of a data architecture provided by an embodiment of the present disclosure.
Detailed Description
In order that the above objects, features and advantages of the present disclosure may be more clearly understood, aspects of the present disclosure will be further described below. It should be noted that the embodiments and features of the embodiments of the present disclosure may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure may be practiced in other ways than those described herein; it is to be understood that the embodiments disclosed in the specification are only a few embodiments of the present disclosure, and not all embodiments.
The first embodiment is as follows:
the embodiment provides a data searching and displaying system based on cross-border trade big data, referring to a structural block diagram of the data searching and displaying system based on cross-border trade big data shown in fig. 1, the system mainly includes a data searching module 102, a data storage module 104 and a data query module 106 connected to each other:
and the data searching module 102 is configured to perform data searching on different types of data storage systems to obtain original cross-border trade data.
In this embodiment, a customs service system is taken as an example, and in a general case, a plurality of distributed subsystems are included in the customs service system, and in the customs service system, data interaction with other various service systems is also required. Based on the data search method, data search can be carried out on different types of data storage systems, data extraction in different places is carried out, and multi-source data search is achieved, so that the data application requirements of the customs service system are met, and data exchange is not required to be generated by direct connection with other systems.
And the data storage module 104 is configured to classify the original cross-border trade data, and distribute and store the original cross-border trade data in different resource libraries according to a classification result. Wherein, according to the Data storage time, the resource library can be divided into an ODPS (Open Data Processing Service) database and an ADS (Automation device specification) database; the ODPS database is used to store all data, and the ADS database is used to store data within a preset time (e.g., within three years).
Typically, both the ODPS database and the ADS database contain various sub-databases, such as by classification of data, which may include one or more of a data chassis library, an analysis topic library, and a full-text search library. In this case, the raw cross-border trade data may be distributed and stored in the data base, the analysis subject library and the full-text search library according to the classification result of the raw cross-border trade data.
And the data query module 106 is configured to obtain query information of a user, determine target data in different resource libraries according to the query information, and display the target data.
In the embodiment, the query information of the user is information such as data keywords, data filtering conditions, and the like. Specifically, data retrieval can be performed in a targeted manner in a data chassis library, an analysis subject library and a full text retrieval library belonging to an ODPS database or an ADS database according to the query information, so as to obtain target data meeting personalized requirements of users.
In the data searching and displaying system based on the cross-border trade big data provided by the embodiment, firstly, the data searching module 102 performs data searching on different types of data storage systems to obtain original cross-border trade data; then the data storage module 104 classifies the original cross-border trade data, and distributes and stores the original cross-border trade data in different resource libraries according to the classification result; and finally, the data query module 106 determines target data in the resource library according to the query information of the user. In the mode, the diversification and richness of the original cross-border trade data search can be improved by searching data of different types of data storage systems, and the diversified data access requirement is better improved; original cross-border trade data are stored in different resource libraries according to classification results, and data storage can be flexibly and pertinently realized; meanwhile, when the user retrieves data, personalized data retrieval can be realized based on diversified resource libraries. Therefore, the method can improve the diversity of data search and better meet the individual requirements of user data retrieval.
In an embodiment, in the data search module 102, the implementation process of performing data search on different types of data storage systems may specifically refer to the following: and acquiring a preset data search strategy and rule, and searching data of different types of data storage systems according to the data search strategy and rule.
Specifically, the preset data search policy and rule may be a search policy and rule customized by a user according to a data search requirement, such as a table field, view content, multi-table union, a cycle interval, search time, and a rule supporting incremental search; and searching data for different types of data storage systems based on the data searching strategy and the rules. When based on rules that support incremental searching, incremental searching may be performed on newly added, deleted, or modified data after the first full data search.
There are a variety of data storage systems that are supported during the actual data search process, and several examples of different types of data storage systems are given herein. Example one: the data search of mainstream databases such as Oracle, My SQL, SQL Server, DB2, Sybase, PostgreSQL and the like is supported; example two: the data intercommunication of an online real-time data analysis engine, an offline batch processing calculation engine, an object storage data service and the like is supported; example three: the open source Hadoop platform is supported, and data of HDFS and Hive can be imported; example four: all basic data types are supported, various files stored in a large field are supported, and content extraction and processing of various documents (such as PDF, Office, Html and the like) stored in a database are supported.
The original cross-border trade data obtained according to the data searching embodiment needs to be stored in a resource library. For ease of understanding, the present embodiment describes the manner in which the original cross-border trade data is stored in the data storage module 104.
In this embodiment, the raw cross-border trade data includes: bottom layer raw data and unstructured data; the resource pool comprises: a data chassis library, an analysis subject library and a full text retrieval library; based on this, referring to the data flow diagram shown in fig. 2, the process of classifying the original cross-border trade data and storing the original cross-border trade data in different resource libraries in a distributed manner according to the classification result may be implemented by the following first data processing unit, second data processing unit, data synchronization unit, and third data processing unit:
the first data processing unit is used for performing data processing operation on the bottom-layer original data according to the service class to obtain first data and storing the first data in a data chassis library in the ODPS database; among these, data processing operations include, but are not limited to: at least one of a data screening operation, a data conversion operation, and a data cleansing operation.
And the second data processing unit is used for screening the first data in the data chassis library according to the service theme to obtain second data and storing the second data in an analysis theme library in the ODPS database. The step is further processing the first data, and the formed analysis topic library can be used for data retrieval of the user according to the topics.
And the data synchronization unit is used for synchronizing the data chassis library and the analysis subject library into the ADS database. It is understood that the functions implemented by the data synchronization unit can be executed after the first data processing unit and the second data processing unit complete the corresponding functions, or can be executed synchronously with the first data processing unit and the second data processing unit. Specifically, a data chassis library belonging to an ODPS database is formed by storing first data, and the data chassis library is synchronized to an ADS database; and synchronizing the analysis topic library to the ADS database while forming an analysis topic library belonging to the ODPS database by storing the second data.
And the third data processing unit is used for extracting data from the unstructured data (OSS) and the first data and the second data stored in the ODPS database to obtain third data, and storing the third data in the full-text search library.
In addition, in another implementation manner, in addition to the data base library, the analysis subject library and the full-text search library, a management library may be provided, and the management library is used for managing various configurations, indexes, models, parameters, logs and the like of the cloud engine.
Referring to the data architecture diagram shown in fig. 3, several examples of distributing and storing original cross-border trade data in different resource pools according to classification results in a customs service scenario are given. In the data architecture diagram, the displayed original cross-border trade data are bottom-layer original data, and the resource library is a data chassis library, an analysis subject library and a management library. The underlying raw data includes business data (such as general data and direct custom characteristic data) in customs, external commission data (such as industry and commerce, tax, foreign exchange … …) and internet data (such as commodity). The first data stored in the data chassis library are data searched according to actual business requirements, and provide uniform data access resources including document data, recorded data, law enforcement results and the like for each system running in the big data cloud; the analysis subject library is various service subjects which are combed for upper-layer application and comprise enterprises, commodities, personnel, documents and the like; the management base stores data generated by the conventional management functions of the system, including metrics, models, parameters, logs, and the like.
In the embodiment, the ADS database can store data of recent years, the ODPS database can store all data, and the data in the ADS database and the data in the ODPS database can be displayed in a switched manner in the application process to meet the data query requirements of users.
In addition, in the process of system development or after online operation, if the data chassis needs to be adjusted in the aspects of table structure, calling mode and the like, the system needs to be correspondingly modified in cooperation with the adjustment so as to adapt to data access after the data chassis is adjusted.
Based on the above database, there are various implementation methods for determining the target data in the repository according to the query information in the data query module 106, and examples of the following three implementation methods are given here.
The first method is as follows: (1) identifying the data type of the data to be queried in the query information; data types such as document data, law enforcement results, merchandise data, personnel data, and the like. (2) Determining a target database in a plurality of resource libraries according to the data type; for example, when the data type of the data to be queried is document data, a target database corresponding to the business data is determined to be a data chassis library from a plurality of resource libraries such as a data chassis library, an analysis subject library, a full-text search library and the like. (3) And performing data retrieval in the target database according to the query information to obtain target data.
In the first mode, different types of data are dispersed in different resource libraries, a data resource scheduling function can be added according to the data storage condition, namely, the data are directly positioned to a target database according to the data types, so that the data retrieval amount can be reduced, the retrieval efficiency is improved, and meanwhile, for a user, the operation habit of the user is not changed.
The second method comprises the following steps: (1) acquiring resource library selection information of a user; the resource library selection information is information for selecting an ODPS database or an ADS database. (2) Determining a target database in an ODPS database and an ADS database according to the resource library selection information; (3) and performing data retrieval in the target database according to the query information to obtain target data. For example, if the resource library selection information is that the ODPS database is selected, the ODPS database is determined to be the target database, and the target data is obtained by retrieving from the ODPS database.
The third method comprises the following steps: (1) performing data retrieval in a preset initial database according to the query information to obtain candidate data; wherein the initial database is an ODPS database or an ADS database; typically, the initial database is a default setting, such as setting the ADS database as the initial database.
(2) Monitoring whether a switching operation of a user is received; the switching operation is an operation of switching the initial database to another database. If not, the following step (3) is performed, and if received, the following steps (4) and (5) are performed.
(3) And taking the candidate data as target data.
(4) If so, switching the initial database into a target database according to the switching operation; the target database is an ODPS database or an ADS database which is not preset as an initial database. In one scenario, if the initial database is the ADS database, the ADS database is switched to the ODPS database according to the switching operation, and the ODPS database is also the target database after the switching.
(5) And performing data retrieval in the target database according to the query information to obtain target data.
In the second and third modes, the data in the ADS database and the ODPS database are displayed in a switched manner, so that more data meeting the query requirement can be provided for the user, and the data in two dimensions, namely the ODPS database or the ADS database, can be obtained.
With the development of services, data expansion is necessary, and based on this, the data search and display system based on cross-border trade big data provided in this embodiment may further include a capacity expansion module, where the capacity expansion module is used to implement an expansion method for a data node of a resource library, as shown in the following:
(I) acquiring the number of the nodes of the data nodes in the current resource library after the data nodes are added; and the current resource library comprises a newly added data node and an initial data node.
And (II) splitting and expanding the data on the initial data nodes in the current resource library according to the number of the data nodes to obtain expanded fragment data. Among them, split spreads may include, but are not limited to: and splitting and expanding the original fragments in the initial data nodes according to a preset multiple to obtain expanded fragment data, so that the number of the fragments of the expanded fragment data meets the capacity expansion requirement of the data nodes containing the newly added data nodes.
And (III) redistributing the expanded fragment data in the current resource library so that the fragment number of the fragment data on each of the newly added data node and the initial data node meets a preset condition. The predetermined conditions involved in the present embodiment include, but are not limited to: whether the fragmentation load of each data node is balanced.
The capacity expansion module provided by the embodiment can support efficient addition of nodes for transverse expansion, so that the resource library has high expandability. It is understood that, in the above embodiment, the extended resource base may be an ODPS database and/or an ADS database, and may also be a sub-database belonging to the ODPS database and the ADS database: a data chassis library, an analysis subject library, a full text retrieval library and a management library.
In addition, the resource library also supports data redundancy backup, and a user can configure the redundancy number, so that the service can be provided without losing data under the condition that nodes within half of the number fail, and the resource library has high availability. And the system supports automatic synchronization, can be operated as a system process or background service, and automatically completes a search task according to a rule set by a user.
Example two:
the embodiment provides a data searching and displaying method based on cross-border trade big data, which is realized by the data searching and displaying system based on cross-border trade big data provided by the first embodiment; the method comprises the following steps:
step S102, data searching is carried out on different types of data storage systems to obtain original cross-border trade data;
step S104, classifying the original cross-border trade data, and distributing and storing the original cross-border trade data in different resource libraries according to classification results;
and step S106, acquiring query information of a user, determining target data in different resource libraries according to the query information, and displaying the target data.
The embodiment of the disclosure provides a data searching and displaying method based on cross-border trade big data, which can improve the diversity and richness of the original cross-border trade data search and better improve the diversified data access requirement by searching data of different types of data storage systems; original cross-border trade data are stored in different resource libraries according to classification results, and data storage can be flexibly and pertinently realized; meanwhile, when the user retrieves data, personalized data retrieval can be realized based on diversified resource libraries. Therefore, the method can improve the diversity of data search and better meet the individual requirements of user data retrieval.
In one embodiment, the raw cross-border trade data includes: bottom layer raw data and unstructured data; the resource pool comprises: a data chassis library, an analysis subject library and a full text retrieval library; classifying the original cross-border trade data, and storing the original cross-border trade data in different resource libraries in a distributed manner according to classification results, wherein the method comprises the following steps: performing data processing operation on the bottom-layer original data according to the service category to obtain first data, and storing the first data in a data chassis library in an Open Data Processing Service (ODPS) database; wherein the data processing operations include: at least one of a data screening operation, a data conversion operation, and a data cleansing operation; screening the first data in a data chassis library according to the service theme to obtain second data, and storing the second data in an analysis theme library in an ODPS database; synchronizing the data chassis library and the analysis subject library into an automatic equipment specification (ADS) database; and extracting data from the unstructured data and the first data and the second data stored in the ODPS database to obtain third data, and storing the third data in a full-text search library.
In one embodiment, the step of determining target data in the repository based on the query information includes: identifying the data type of the data to be queried in the query information; determining a target database in a plurality of resource libraries according to the data type; and performing data retrieval in the target database according to the query information to obtain target data.
In one embodiment, the step of determining target data in the repository based on the query information includes: : acquiring resource library selection information of a user; determining a target database in an ODPS database and an ADS database according to the resource library selection information; and performing data retrieval in the target database according to the query information to obtain target data.
In one embodiment, the step of determining target data in the repository based on the query information includes: performing data retrieval in a preset initial database according to the query information to obtain candidate data; wherein the initial database is an ODPS database or an ADS database; monitoring whether a switching operation of a user is received; if not, the candidate data is taken as target data; if so, switching the initial database into a target database according to the switching operation; the target database is an ODPS database or an ADS database which is not preset as an initial database; and performing data retrieval in the target database according to the query information to obtain target data.
In one embodiment, the method further comprises: acquiring the number of the nodes of the data nodes in the current resource library after the data nodes are added; the current resource library comprises a newly added data node and an initial data node; performing split expansion on data on the initial data nodes in the current resource library according to the number of the data nodes to obtain expanded fragment data; and redistributing the expanded fragment data in the current resource library so that the fragment number of the fragment data on each of the newly added data node and the initial data node meets a preset condition.
In one embodiment, the step of performing a data search on different types of data storage systems comprises: and acquiring a preset data search strategy and rule, and searching data of different types of data storage systems according to the data search strategy and rule.
An embodiment of the present invention further provides an electronic device, which includes: a processor and a storage device;
the storage device stores thereon a computer program which, when executed by the processor, executes the data searching and presenting method based on cross-border trade big data in the above embodiment.
The embodiment of the invention also provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the cross-border trade big data-based data searching and displaying method in the embodiment are executed.
Finally, it is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present disclosure, which enable those skilled in the art to understand or practice the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A cross-border trade big data-based data searching and displaying system is characterized by comprising:
the data search module is used for searching data of different types of data storage systems to obtain original cross-border trade data;
the data storage module is used for classifying the original cross-border trade data and storing the original cross-border trade data in different resource libraries in a distributed manner according to a classification result;
and the data query module is used for acquiring query information of a user, determining target data in different resource libraries according to the query information, and displaying the target data.
2. The system of claim 1, wherein the raw cross-border trade data comprises: bottom layer raw data and unstructured data; the resource pool comprises: a data chassis library, an analysis subject library and a full text retrieval library;
the data storage module includes:
the first data processing unit is used for performing data processing operation on the bottom layer original data according to the service type to obtain first data and storing the first data in a data chassis library in an Open Data Processing Service (ODPS) database; wherein the data processing operations comprise: at least one of a data screening operation, a data conversion operation, and a data cleansing operation;
the second data processing unit is used for screening the first data in the data chassis library according to a service theme to obtain second data and storing the second data in an analysis theme library in the ODPS database;
the data synchronization unit is used for synchronizing the data chassis library and the analysis subject library into an automatic equipment specification (ADS) database;
and the third data processing unit is used for extracting data from the unstructured data and the first data and the second data stored in the ODPS database to obtain third data, and storing the third data in a full-text search library.
3. The system of claim 2, wherein the data query module is further configured to:
identifying the data type of the data to be queried in the query information;
determining a target database in a plurality of resource libraries according to the data type;
and performing data retrieval in the target database according to the query information to obtain target data.
4. The system of claim 2, wherein the data query module is further configured to:
acquiring resource library selection information of a user;
determining a target database in the ODPS database and the ADS database according to the resource library selection information;
and performing data retrieval in the target database according to the query information to obtain target data.
5. The system of claim 2, wherein the data query module is further configured to:
performing data retrieval in a preset initial database according to the query information to obtain candidate data; wherein the initial database is in the ODPS database or the ADS database;
monitoring whether a switching operation of a user is received;
if not, the candidate data is taken as target data;
if so, switching the initial database into a target database according to the switching operation; the target database is a database which is not preset as the initial database in the ODPS database or the ADS database;
and performing data retrieval in the target database according to the query information to obtain the target data.
6. The system of claim 1, further comprising a capacity expansion module, the capacity expansion module configured to:
acquiring the number of the nodes of the data nodes in the current resource library after the data nodes are added; the current resource library comprises a newly added data node and an initial data node;
performing split expansion on the data on the initial data node in the current resource library according to the number of the data nodes to obtain expanded fragment data;
and redistributing the expanded fragment data in the current resource library so that the fragment number of the fragment data on each of the newly added data node and the initial data node meets a preset condition.
7. The system of claim 1, wherein the data search module is further configured to:
and acquiring a preset data search strategy and rule, and searching data of different types of data storage systems according to the data search strategy and rule.
8. A data searching and displaying method based on cross-border trade big data is characterized by comprising the following steps:
data searching is carried out on different types of data storage systems to obtain original cross-border trade data;
classifying the original cross-border trade data, and storing the original cross-border trade data in different resource libraries in a distributed manner according to classification results;
acquiring query information of a user, determining target data in different resource libraries according to the query information, and displaying the target data.
9. An electronic device, comprising: a processor and a storage device;
the storage device has stored thereon a computer program which, when executed by the processor, performs the method of claim 8.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method as set forth in claim 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010365885.0A CN111563112A (en) | 2020-04-30 | 2020-04-30 | Data search and display system based on cross-border trade big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010365885.0A CN111563112A (en) | 2020-04-30 | 2020-04-30 | Data search and display system based on cross-border trade big data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111563112A true CN111563112A (en) | 2020-08-21 |
Family
ID=72071763
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010365885.0A Pending CN111563112A (en) | 2020-04-30 | 2020-04-30 | Data search and display system based on cross-border trade big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111563112A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106557486A (en) * | 2015-09-25 | 2017-04-05 | 阿里巴巴集团控股有限公司 | A kind of storage method and device of data |
| CN106980672A (en) * | 2017-03-28 | 2017-07-25 | 中国电子科技集团公司第二十九研究所 | The data-storage system and method for a kind of isomerous environment Based on Object-Oriented Technology |
| CN107622091A (en) * | 2017-08-23 | 2018-01-23 | 阿里巴巴集团控股有限公司 | A kind of data base query method and device |
| CN109086409A (en) * | 2018-08-02 | 2018-12-25 | 泰康保险集团股份有限公司 | Micro services data processing method, device, electronic equipment and computer-readable medium |
| CN109145225A (en) * | 2017-06-27 | 2019-01-04 | 阿里巴巴集团控股有限公司 | A kind of data processing method and device |
| CN110347719A (en) * | 2019-06-24 | 2019-10-18 | 华南农业大学 | A kind of enterprise's foreign trade method for prewarning risk and system based on big data |
-
2020
- 2020-04-30 CN CN202010365885.0A patent/CN111563112A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106557486A (en) * | 2015-09-25 | 2017-04-05 | 阿里巴巴集团控股有限公司 | A kind of storage method and device of data |
| CN106980672A (en) * | 2017-03-28 | 2017-07-25 | 中国电子科技集团公司第二十九研究所 | The data-storage system and method for a kind of isomerous environment Based on Object-Oriented Technology |
| CN109145225A (en) * | 2017-06-27 | 2019-01-04 | 阿里巴巴集团控股有限公司 | A kind of data processing method and device |
| CN107622091A (en) * | 2017-08-23 | 2018-01-23 | 阿里巴巴集团控股有限公司 | A kind of data base query method and device |
| CN109086409A (en) * | 2018-08-02 | 2018-12-25 | 泰康保险集团股份有限公司 | Micro services data processing method, device, electronic equipment and computer-readable medium |
| CN110347719A (en) * | 2019-06-24 | 2019-10-18 | 华南农业大学 | A kind of enterprise's foreign trade method for prewarning risk and system based on big data |
Non-Patent Citations (1)
| Title |
|---|
| 徐刚等: "《客户关系管理理论与实务》", 31 May 2016, 上海财经大学出版社 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220284014A1 (en) | Deriving metrics from queries | |
| US11921715B2 (en) | Search integration | |
| US6507839B1 (en) | Generalized term frequency scores in information retrieval systems | |
| US9626411B1 (en) | Self-described query execution in a massively parallel SQL execution engine | |
| US11360986B2 (en) | Enterprise search | |
| US20080222105A1 (en) | Entity recommendation system using restricted information tagged to selected entities | |
| US9747349B2 (en) | System and method for distributing queries to a group of databases and expediting data access | |
| WO2006076805A1 (en) | Method and apparatus for ordering items within datasets | |
| CN106294695A (en) | A kind of implementation method towards the biggest data search engine | |
| US20080059432A1 (en) | System and method for database indexing, searching and data retrieval | |
| CN111931027A (en) | Intelligent news public opinion early warning system | |
| CN112269816B (en) | Government affair appointment correlation retrieval method | |
| CN103917970B (en) | Keyword search of customer interest in an enterprise | |
| JP2001350656A (en) | Integrated access method for different data sources | |
| CN114461603A (en) | Multi-source heterogeneous data fusion method and device | |
| US20190095481A1 (en) | Generating a query | |
| US20100191730A1 (en) | Efficiency in processing queries directed to static data sets | |
| US20170270149A1 (en) | Database systems with re-ordered replicas and methods of accessing and backing up databases | |
| US10394826B1 (en) | System and methods for searching query data | |
| CN113032436B (en) | Searching method and device based on article content and title | |
| US11556515B2 (en) | Artificially-intelligent, continuously-updating, centralized-database-identifier repository system | |
| US20120066222A1 (en) | Web architecture for green design and construction | |
| CN111563112A (en) | Data search and display system based on cross-border trade big data | |
| CN108615182A (en) | A kind of method and system that product intelligent is recommended | |
| Rats et al. | Using of cloud computing, clustering and document-oriented database for enterprise content management |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200821 |
|
| RJ01 | Rejection of invention patent application after publication |