CN111898003A

CN111898003A - Industrial high-grade data rapid cluster storage and search system and method

Info

Publication number: CN111898003A
Application number: CN202010885245.2A
Authority: CN
Inventors: 刘伟铭
Original assignee: Individual
Current assignee: Individual
Priority date: 2020-08-28
Filing date: 2020-08-28
Publication date: 2020-11-06
Anticipated expiration: 2040-08-28
Also published as: CN111898003B

Abstract

The invention discloses a system and a method for quickly storing and searching industrial high-grade data in a cluster, which comprises a product ID compiling and tracking system, a high-speed data acquisition system, a product data collection module, a data classification module and a cluster storage system, wherein the high-speed data acquisition and operation module correspondingly acquires production data of process production equipment and stores the production data and the product ID as a process data packet; the product data collecting module is used for classifying according to the product ID and sequentially storing and merging the process data packets of the same product ID into the product data packets of the same product ID according to the sequence of process time, process length, concentration or voltage and the like, and the data classifying module is used for respectively storing the product data packets into the data file servers of the cluster storage system. The invention realizes the storage and operation of industrial high-grade data according to rules, and the multiple processes respectively monopolize data link resources to search, thereby greatly reducing the task amount, task difficulty and execution time of single-process search query and being beneficial to quickly obtaining conventional and big data analysis results.

Description

Industrial high-grade data rapid cluster storage and search system and method

Technical Field

The invention relates to the field of industrial data storage and search query, in particular to a system and a method for quickly clustering, storing and searching industrial high-grade data.

Background

In the high-precision numerical control assembly line processing process, a product is processed according to a production assembly line, a plurality of production processes are often required to be combined on the production assembly line, each production process position corresponds to a high-precision numerical control machine tool or other precision processing equipment (collectively referred to as process production equipment), and production data of the production equipment of each process is required to be stored in the high-precision product production, so that various conventional and big data analysis methods are used for finding out a proper way to improve production or find out factors influencing the product quality at a later stage. The production data of high-precision products are relatively various, for example, a high-precision numerical control machine tool usually performs finish machining on the production and machining of the high-precision products at a control period of 0.1ms to 10ms, so that all the production data of the whole high-precision product form industrial high-score data, wherein the high score refers to the high resolution of physical quantities such as time, length and the like, and is acquired every 10ms or every 0.1ms, for example, every 1m or every 1cm (the distance of the high-precision product moving when the high-precision product is moved during machining) for one point. In order to improve the product yield and quality, important production process data is collected in a whole process, such as 3 ten thousand real numbers (for example, 10ms data collected once a day with the data volume of 30000 × 24 × 60 × 1000/10 × 4 bytes 1036.8G, 378T data in one year, which is generally stored for at least 10 years), and only then various conventional and big data analysis methods can be used to find out a suitable way to improve the production or find out factors affecting the product quality. At present, the SQL sentences are generally stored in various databases and are used for query, and the query time is usually small due to huge data volume, so that the time is very time-consuming and labor-consuming.

Disclosure of Invention

Aiming at the defects existing in the prior art, the invention aims to provide a system and a method for rapidly storing and searching industrial high-grade data in a cluster mode, so that the industrial high-grade data are stored and calculated according to rules, multiple processes respectively monopolize data link resources to search, the task quantity, the task difficulty and the execution time of single-process search query are greatly reduced, the search query and data feedback can be completed at the second level or the minute level, the working efficiency is improved, and the system and the method are favorable for rapidly realizing finding a proper way to improve the production or find out factors influencing the product quality by using various conventional and big data analysis methods.

The purpose of the invention is realized by the following technical scheme:

the utility model provides a quick cluster storage search system of industry high score data, includes industrial production system, industrial production system includes a plurality of process production equipment according to production procedure in proper order, its characterized in that: the system also comprises a product ID compiling and tracking system, a high-speed data acquisition system, a product data collection module, a data classification module and a cluster storage system, wherein the high-speed data acquisition system comprises a plurality of high-speed data acquisition and operation modules, the product ID compiling and tracking system is connected with the high-speed data acquisition system, the product ID compiling and tracking system is used for writing or reading a product ID into or from the same product in an industrial production system and completing the writing or reading work of all products, the product ID is a unique identity code of the product, the high-speed data acquisition and operation modules of the high-speed data acquisition system correspond to process production equipment of a plurality of industrial production systems (namely one high-speed data acquisition and operation module can correspondingly acquire the production data of a plurality of process production equipment), and a certain number or a plurality of process production equipment can share one high-speed data acquisition and operation module, namely, the same high-speed data acquisition and operation module realizes the corresponding acquisition operation of the production data of some or a plurality of process production devices, and one high-speed data acquisition and operation module can also correspond to one process production device. The high-speed data acquisition and operation module correspondingly acquires production data of the process production equipment and stores the production data and the product ID as a process data packet; the high-speed data acquisition system is connected with a product data collection module, the product data collection module classifies the product data according to the product ID and stores and combines the process data packets with the same product ID into the product data packets with the same product ID (the product data collection module of the invention classifies the product data according to the product ID and sequentially stores and combines the process data packets with the same product ID into the product data packets with the same product ID according to the data length or other modes), and the product data collection module finishes the classification and combination of the product data packets with all the product IDs; the product data collection module is connected with the data classification module, the data classification module is connected with the cluster storage system, the cluster storage system comprises a plurality of data file cluster servers, and the data classification module stores product data packets into the data file cluster servers of the cluster storage system respectively. The data file cluster server has no redundancy and backup functions, high-grade data in industrial high-grade data refers to high resolution of physical quantities such as time, length and the like, for example, the high-grade data is acquired once in 10ms and 0.1ms, one point is acquired in 1m and 1cm, data aggregation is calculated according to tracking ID numbers of products in various working procedures, and a tracking method can be labels, positions calculated by a computer control program, working procedure position trigger signals and the like. The high-speed data acquisition and operation module can be internally provided with a complex mathematical operation module independently, input signals of the complex mathematical operation module are physical quantities which are determined according to search conditions and are acquired directly, the search conditions are definite, limited and planned in advance, output signals of the complex mathematical operation module are collected into product data, the output signals can be acquired only through complex data operation because the output signals are not acquired directly, and if the signals are calculated during searching, a large amount of time is consumed, and the search time cannot meet the requirements. The data file cluster server is a data file cluster server, data are stored in a file mode, and the searching efficiency is high and the searching speed is high; secondly, the high-set data file cluster server stores data in a high-speed cluster database, supports SQL query statements, and can adopt a high-set database mode on the premise that the search time can meet the requirement; only one type of server is typically installed in a system.

In order to better realize the invention, the process data packet comprises characteristic item data, the characteristic item data is a set of characteristic items, and the high-speed data acquisition and operation module performs characteristic extraction operation on the acquired production data and stores the characteristic item data together with the product ID into the characteristic item data; the product data package comprises product attribute data, the product attribute data is a set of product attribute items, the product attribute items correspond to the characteristic items, and the product data collection module collects the characteristic item data in the process data package into the product attribute data according to the product ID; the high-speed data acquisition operation module of the high-speed data acquisition system respectively stores the process data packets in sequence according to a time sequence and a logic sequence, the product data collection module stores the product data packets in sequence according to the logic sequence, the logic sequence is one or more of a length sequence, a concentration sequence or a voltage sequence and the like which are closely related to the yield and quality of a product, the time sequence is the time of the acquired production data, the length sequence is the product length of the acquired production data, the concentration sequence is the liquid concentration of the acquired production data (such as the concentration of a certain substance in a solution required by production, the concentration of the certain substance in the solution is closely related to the chemical reaction speed, and the chemical reaction speed is closely related to the yield, in this case, the concentration is taken as a sequence with the concentration of 0.5% 1.5% … 80% and the like), the voltage sequence is the voltage power generation characteristic of the collected production data (for example, the voltage discharge characteristic is 100V200V1000V10000V100000V, and the voltage is taken as a sequence when the discharge characteristic is various voltages).

The further technical scheme is as follows: the invention also comprises a total data backup server, wherein the total data backup server is connected with the data classification module and backs up and stores the product data packets. According to the invention, data backup storage of all industrial high-quality data (including all product data packets) can be realized through the total data backup server.

The invention also comprises a search client system which is in network communication connection with the cluster storage system and comprises a plurality of search clients, the search clients of the search client system send search instructions to the data file cluster servers of the cluster storage system, and the data file cluster servers of the cluster storage system execute the search instructions and transmit search result data to the search clients of the search client system in a communication way.

Preferably, the data file cluster server of the cluster storage system of the present invention has a plurality of search engine processes therein, each search engine process has an exclusive data link, each data link has an independent hard disk, a memory block and a CPU core, each search engine process has two independent thread modules, the two thread modules are respectively an index thread module and a search thread module, the index thread module establishes indexes and data associations for corresponding mappings of all product data packets or incremental product data packets stored in the data file cluster server, the indexes correspond to product attribute items, and the search instructions correspond to the indexes; the search thread module receives a search instruction and forms a search task, the search thread module forms a search task queue according to the receiving time, and the search thread module sequentially executes the corresponding search instruction according to the search task queue and correspondingly transmits search result data to the search client.

Preferably, the operation priority of the search thread module in the data file cluster server is greater than the operation priority of the index thread module (the search thread module has the highest operation priority); the search instruction includes a number of product attribute items.

The invention provides a first preferred technical scheme of a product ID writing tracking system structure, which comprises the following steps: the product ID compiling and tracking system comprises a plurality of product ID compiling units, the product ID compiling units of the product ID compiling and tracking system correspond to process production equipment of a plurality of industrial production systems (namely, a high-speed data acquisition and operation module can correspondingly acquire production data of the process production equipment), the process production equipment of the industrial production systems is correspondingly provided with a proximity sensor or a counting sensor connected with the product ID compiling units, the product ID compiling units write product IDs according to the same rule method, and each product ID compiling unit in the product ID compiling and tracking system writes the same product ID aiming at the same product. The technical scheme mainly includes that the same product ID is synchronously written into the same product by each product ID writing unit of the product ID writing tracking system (the synchronization is not synchronous in a time sense, but the product ID writing unit corresponding to the first procedure writes the product ID of the first product, the product ID writing unit corresponding to the second procedure writes the product ID … of the first product, the product ID writing unit corresponding to the Mth procedure writes the product ID of the first product into the same product ID according to the same rule method, because the same product appears as the first product in the first procedure and the Mth procedure of the second procedure …, each product ID writing unit writes the product ID according to the same rule method, the product IDs of all products can be written into the same product ID, and the product IDs of the same product are the same).

The invention provides a second preferred technical scheme of a product ID writing tracking system structure, which comprises the following steps: the product ID compiling and tracking system comprises a product ID compiling unit and a plurality of product ID reading modules, wherein the high-speed data acquisition system is correspondingly provided with the product ID compiling unit in the high-speed data acquisition and operation module of the first procedure, and the product ID compiling unit is used for writing an identifiable code with a product ID into a product and simultaneously feeding back the product ID to the high-speed data acquisition and operation module (the product ID recorded by the computer is tracked by the high-speed data acquisition and operation module, and the high-speed data acquisition and operation module also needs to acquire the product ID of each unit); the high-speed data acquisition system is provided with product ID reading modules corresponding to the high-speed data acquisition operation modules in the rest processes respectively, and the product ID reading modules are used for identifying the identifiable codes on the products, reading the product IDs and feeding the product IDs back to the high-speed data acquisition operation modules. The technical scheme mainly includes that a product ID compiling unit is arranged in a first process (the process is short for processing process and the same as the process below), product ID reading modules are arranged except the first process, an identifiable code with a product ID is written in a product through the product ID compiling unit in the first process, and when the product enters a second process, the product ID reading module corresponding to the second process can identify the identifiable code on the product, read the product ID and feed the product ID back to the high-speed data acquisition and operation module. The product ID authoring tracking system may also employ other techniques.

Preferably, the number of the data file cluster servers is related to the search allowable time, the search data volume and the number of search engines of each data file cluster server, and is not related to the number of the search clients; each data file cluster server is provided with a plurality of search engine processes, each search engine is executed by an independent process, each search engine process monopolizes related data link resources, the data link resources comprise hard disks, a memory and CPU (central processing unit) kernels, and each data file cluster server comprises a plurality of physical hard disks, a large memory and a plurality of CPU kernels.

Preferably, the data file cluster server of the present invention may be a high-speed cluster database server, the high-speed cluster database server supports SQL statement query, the search client issues an SQL search instruction to all the high-speed cluster database servers, and the search client reports the comprehensive query result according to all the SQL query results.

A method for quickly cluster-storing and searching industrial high-grade data comprises an industrial production system, a product ID compiling and tracking system, a high-speed data acquisition system, a product data collection module, a data classification module, a cluster storage system and a search client side system, wherein the industrial production system finishes production line production operation of products, the industrial production system sequentially comprises a plurality of process production devices according to a production flow, and each process production device correspondingly processes production of the product produced by the product production line, and the method comprises the following steps:

A. and (3) industrial high-grade data classification cluster storage: the product ID compiling and tracking system writes or reads product IDs into or from the same product in the industrial production system and completes the writing or reading work of all the products, the product IDs are unique identity codes of the products, production equipment of each procedure of the industrial production system has the same product ID for the same product, and the product IDs of the same product, which are acquired by each high-speed data acquisition operation module of the high-speed data acquisition system, are the same;

the high-speed data acquisition and operation module of the high-speed data acquisition and operation system corresponds to process production equipment of a plurality of industrial production systems (namely, one high-speed data acquisition and operation module can correspondingly acquire the production data of a plurality of process production equipment), the high-speed data acquisition and operation module correspondingly acquires the production data of the process production equipment and stores the production data as a process data packet together with a product ID (identity), the process data packet comprises characteristic item data, the characteristic item data is a set of characteristic items, and the high-speed data acquisition and operation module performs characteristic extraction operation on the acquired production data and stores the characteristic items together with the product ID to the characteristic item data;

the product data collecting module is used for classifying according to the product ID and storing and combining the process data packets of the same product ID into the product data packets of the same product ID, each product data packet comprises product attribute data, the product attribute data are a set of product attribute items, the product attribute items correspond to the feature items, and the product data collecting module is used for collecting the feature item data in the process data packets into the product attribute data according to the product ID; the cluster storage system comprises a plurality of data file cluster servers, a product data collection module sequentially finishes the classification and combination work of product data packets of all product IDs, a data classification module stores the product data packets into each data file cluster server of the cluster storage system respectively, the data classification module records a storage path in the storage process, and all the product data packets in the cluster storage system form industrial high-grade data;

the data file cluster server setting method comprises the following steps: the number of data file cluster servers is related to the search allowable time, the search data amount, the number of search engine processes of each data file cluster server, and is not related to the number of search clients. Each data file cluster server is provided with a plurality of search engine processes, each search engine process is executed by an independent process, each search engine process monopolizes related data link resources such as hard disks, memories, CPU kernels and the like, and each server must be provided with a plurality of physical hard disks, large memories, multi-CPU kernels and the like.

B. Searching industrial high-grade data: the search client side system comprises a plurality of search client sides, a query search interface system is arranged in each search client side, the query search interface system supports SQL query, the search client sides respectively issue search instructions to each data file cluster server of the cluster storage system through the query search interface system, and each search instruction comprises a plurality of product attribute items;

each data file cluster server of the cluster storage system is internally provided with a plurality of search engine processes, each search engine process monopolizes one data link, each data link is provided with an independent hard disk, a memory block and a CPU (central processing unit) kernel, each search engine process is provided with two independent thread modules which are respectively an index thread module and a search thread module, the index thread module establishes index and data association for corresponding mapping of all product data packets or incremental product data packets stored in the data file cluster server, the index corresponds to a product attribute item, and a search instruction corresponds to the index; the search thread module receives a search instruction and forms a search task, the search thread module forms a search task queue according to the receiving time, the search thread module sequentially executes the corresponding search instruction according to the search task queue and correspondingly transmits search result data to the corresponding search client, and after all cluster servers are searched, the query client reports comprehensive search results;

the operation priority of the search thread module is greater than that of the index thread module, the search thread module in the data file cluster server has the highest operation priority, and the operation method of the data file cluster server is as follows:

b11, when the search thread module of the same search engine process in the data file cluster server does not receive the search instruction or is idle, the index thread module of the same search engine process in the data file cluster server operates in an autonomous cycle and establishes index and data association;

and B12, when the search thread module of the same search engine process in the data file cluster server receives the search instruction, the index thread module of the same search engine process in the data file cluster server stops working, the search thread module of the same search engine process in the data file cluster server is started and sequentially establishes a search task queue according to the time for receiving the search instruction, the search thread module sequentially executes the corresponding search instruction according to the search task queue and correspondingly transmits the search result data to the corresponding search client, and the index thread module of the same search engine process is started again until the search thread module of the same search engine process finishes executing all the search instructions.

The method for rapidly clustering, storing and searching the industrial high-quality data can also realize total data backup, and the specific technical scheme is as follows: the method comprises a total data backup server and further comprises the following steps:

C. industrial high-score data backup: the total data backup server is connected with the data classification module, the data classification module sequentially transmits all the product data packages to the total data backup server, and the total data backup server backs up and stores the product data packages;

and B, the search client side sends the search instruction in the step B, the modes of broadcasting and the like are adopted to ensure that each process of each cluster server can almost simultaneously receive the search instruction, and the cluster servers of the data files of the cluster storage system immediately respond after receiving the search instruction so as to facilitate the search client side to take measures and execute the search command.

Compared with the prior art, the invention has the following advantages and beneficial effects:

the invention collects all production data according to working procedures respectively, extracts characteristic items of the production data through operation, synchronously stores the characteristic items into working procedure data packets, then collects and arranges the working procedure data packets into product data packets, and stores the product data packets into a cluster storage system in a classified manner, realizes clustered and balanced data storage through each data file cluster server, and simultaneously each data file cluster server can independently establish an index so as to facilitate search and query. The invention provides a brand-new industrial high-grade data cluster storage and search system and method, which realize the storage and operation of industrial high-grade data according to rules, greatly reduce the quantity of search and query tasks, the difficulty of the tasks and the execution time, can realize the completion of various search and query and data feedback at the second level or the minute level, improve the working efficiency, and are beneficial to quickly realizing the finding of a proper way to improve the production or find out factors influencing the product quality by using various conventional and big data analysis methods.

Drawings

FIG. 1 is a block diagram of the schematic structure of a system cluster storage search system of the present invention;

FIG. 2 is a block diagram of the industrial high-score data cluster searching principle of the present invention;

FIG. 3 is a flow diagram of the present invention for industrial high-score data classification cluster storage;

FIG. 4 is a flow diagram of an industrial high-score data search according to the present invention;

FIG. 5 is a flow diagram of an index thread module according to the present invention;

FIG. 6 is a flow diagram of a search thread module according to the present invention;

FIG. 7 is a schematic diagram of a query based on a data file system index structure according to the present invention;

FIG. 8 is a schematic diagram of a database system based query according to the present invention;

FIG. 9 is a flow diagram of the present invention for querying industrial high-score data by SQL.

Detailed Description

The present invention will be described in further detail with reference to the following examples:

examples

As shown in fig. 1 to 9, a fast cluster storage and search system for industrial high-grade data includes an industrial production system, a product ID compiling and tracking system, a high-speed data acquisition system, a product data collection module, a data classification module and a cluster storage system, the industrial production system includes a plurality of process production devices in sequence according to a production flow, the high-speed data acquisition system includes a plurality of high-speed data acquisition and operation modules, the product ID compiling and tracking system is connected with the high-speed data acquisition system, the product ID compiling and tracking system is used for writing or reading a product ID in the same product in the industrial production system and completing writing or reading of all products, the product ID is a unique identity code of the product, and the product ID compiling and tracking system of the present invention can adopt two preferable technical schemes as follows:

the first preferred technical solution is as follows: the product ID compiling and tracking system comprises a plurality of product ID compiling units, the product ID compiling units of the product ID compiling and tracking system correspond to the process production equipment of a plurality of industrial production systems (namely, one high-speed data acquisition and operation module can correspondingly acquire the production data of a plurality of process production equipment), and a plurality of process production equipment can share one high-speed data acquisition and operation module, namely, the same high-speed data acquisition and operation module can realize the corresponding acquisition operation of the production data of the plurality of process production equipment. The process production equipment of the industrial production system is correspondingly provided with a proximity sensor or a counting sensor connected with a product ID compiling unit, the product ID compiling unit writes product IDs according to the same rule method, and each product ID compiling unit in the product ID compiling tracking system writes the same product ID aiming at the same product. The technical scheme mainly includes that the same product ID is synchronously written into the same product by each product ID writing unit of the product ID writing tracking system (the synchronization is not synchronous in a time sense, but the product ID writing unit corresponding to the first procedure writes the product ID of the first product, the product ID writing unit corresponding to the second procedure writes the product ID … of the first product, the product ID writing unit corresponding to the Mth procedure writes the product ID of the first product into the same product ID according to the same rule method, because the same product appears as the first product in the first procedure and the Mth procedure of the second procedure …, each product ID writing unit writes the product ID according to the same rule method, the product IDs of all products can be written into the same product ID, and the product IDs of the same product are the same).

The second preferred technical solution is as follows: the product ID compiling and tracking system comprises a product ID compiling unit and a plurality of product ID reading modules, wherein the product ID compiling unit is correspondingly arranged on the high-speed data collecting and operating module in the first process, and the product ID compiling unit is used for writing an identifiable code with a product ID into a product and simultaneously feeding back the product ID to the high-speed data collecting and operating module (the product ID recorded by the computer is tracked by the high-speed data collecting and operating module, and the high-speed data collecting and operating module also collects the product ID of each unit). The high-speed data acquisition system is provided with product ID reading modules corresponding to the high-speed data acquisition operation modules in the rest processes respectively, and the product ID reading modules are used for identifying the identifiable codes on the products, reading the product IDs and feeding the product IDs back to the high-speed data acquisition operation modules. The technical scheme mainly includes that a product ID compiling unit is arranged in a first process (the process is short for processing process and the same as the process below), product ID reading modules are arranged except the first process, an identifiable code with a product ID is written in a product through the product ID compiling unit in the first process, and when the product enters a second process, the product ID reading module corresponding to the second process can identify the identifiable code on the product, read the product ID and feed the product ID back to the high-speed data acquisition and operation module.

The high-speed data acquisition operation module of the high-speed data acquisition system corresponds to the process production equipment of a plurality of industrial production systems (namely, one high-speed data acquisition operation module can correspondingly acquire the production data of a plurality of process production equipment), and the high-speed data acquisition operation module correspondingly acquires the production data of the process production equipment and stores the production data as a process data packet together with the product ID. The process data packet comprises feature item data, the feature item data is a set of feature items, and the high-speed data acquisition and operation module performs feature extraction operation on the acquired production data and stores the feature items together with the product ID into the feature item data. The product data collection module performs data collection, complex mathematical operation, single product process data calculation and complete single product data calculation, and the data classification module realizes intelligent classification, cluster storage and total data backup independence (the data classification module and the product data collection module have no rapidity requirement), so that preparation is provided for subsequent rapid search. And determining which physical quantities are subjected to complex mathematical operation according to the search conditions, wherein the search conditions are clear, limited and planned in advance (the search conditions are product attribute items in product attribute data, the product attribute items contained in the search instruction and the feature items in the feature item data). As shown in fig. 3, the data aggregation method: the tracking method can be a label, a computer program controlled position, a process position trigger signal and the like according to the tracking ID number calculation of the product in each process. Where conditions 1-n are explicit, finite, pre-programmed, and also consistent with the search criteria. Data classification: basically ensuring that all data file cluster servers are load balanced (namely N data file cluster servers of a cluster storage system run in a load balancing mode) under various agreed search conditions, wherein the search completion time depends on the heaviest data file cluster server; classification principle: and classifying the product data packets according to product attributes by combining the data contents already stored by all the data file cluster servers, such as: product ID, date and time of product production, material code, physical size, product performance, model specification, raw material condition and the like. The high-speed data acquisition operation module can be internally provided with a complex mathematical operation module separately, the complex mathematical operation module is specially used for carrying out feature extraction operation on acquired production data, which physical quantities are determined to carry out complex mathematical operation according to search conditions, and a complete product data is stored in a hard disk or a memory obtained by intelligent classification; data of one product can only be stored in the same hard disk or memory block. The data file cluster server does not undertake data redundancy backup work, and total data backup is placed in a single total data backup server. Modules for single product data collection, operation, intelligent classification, cluster storage and the like have no special rapidity requirement. Data collection, complex mathematical operation, single product process data calculation, complete single product data calculation, intelligent classification, cluster storage and total data backup server independence are all prepared for subsequent quick search. The data file cluster server has no redundancy and backup functions, high-grade data in industrial high-grade data refers to high resolution of physical quantities such as time, length and the like, for example, the high-grade data is acquired once in 10ms and 0.1ms, one point is acquired in 1m and 1cm, data aggregation is calculated according to tracking ID numbers of products in various working procedures, and a tracking method can be labels, positions calculated by a computer control program, working procedure position trigger signals and the like. The high-speed data acquisition and operation module can be internally provided with a complex mathematical operation module independently, input signals of the complex mathematical operation module are physical quantities which are determined according to search conditions and are acquired directly, the search conditions are definite, limited and planned in advance, output signals of the complex mathematical operation module are collected into product data, the output signals can be acquired only through complex data operation because the output signals are not acquired directly, and if the signals are calculated during searching, a large amount of time is consumed, and the search time cannot meet the requirements. The data file cluster server is a data file cluster server, data are stored in a file mode, and the searching efficiency is high and the searching speed is high; secondly, the high-set data file cluster server stores data in a high-speed cluster database, supports SQL query statements, and can adopt a high-set database mode on the premise that the search time can meet the requirement; only one type of server is typically installed in a system.

The high-speed data acquisition system is connected with a product data collection module, the product data collection module classifies the product data according to the product ID and stores and merges the process data packets of the same product ID into the product data packets of the same product ID (the product data collection module of the embodiment classifies the product data according to the product ID and sequentially stores and merges the process data packets of the same product ID into the product data packets of the same product ID according to the data length or other modes), and the product data collection module completes the classification and merging work of the product data packets of all the product IDs; the high-speed data acquisition and operation module of the high-speed data acquisition system sequentially stores process data packets according to a logic sequence, the product data collection and operation module sequentially stores product data packets according to a logic sequence, the high-speed data acquisition and operation module of the high-speed data acquisition system sequentially stores the process data packets according to a time sequence and a logic sequence, the product data collection module sequentially stores the product data packets according to a logic sequence, the logic sequence is one or more of a length sequence, a concentration sequence, a voltage sequence and other physical quantity sequences closely related to the yield and quality of a product, the time sequence is the time of the acquired production data, the length sequence is the product length of the acquired production data, the concentration sequence is the liquid concentration of the acquired production data (such as the concentration of a certain substance in a solution required for production, and the concentration of the certain substance in the solution is closely related to the chemical reaction speed, the chemical reaction speed is closely related to the yield, and in this case, the concentration is taken as a sequence of concentrations of 0.5% 1% 1.5% … 80% and the like), and the voltage sequence is the voltage power generation characteristic of the collected production data (for example, the voltage discharge characteristic is taken as a sequence of discharge characteristics at various voltages such as 100V200V1000V10000V100000V and the like). The high-speed data acquisition and operation module can sequentially store and store production data into process data packets according to a time sequence or length sequence and the like (according to the time sequence, the high-speed data acquisition and operation module sequentially acquires the production data according to the time sequence and forms the process data packets; according to the length sequence, the high-speed data acquisition and operation module can be internally provided with a complex mathematical operation module independently, the high-speed data acquisition and operation module acquires the production data at one time, the complex mathematical operation module performs data length operation on the production data, for example, a process device processes a steel product with the length of 100 meters in the length direction, the complex mathematical operation module can calculate the production data in a certain length range of the steel product according to the processing speed and the processing time of the process device, and then the high-speed data acquisition and operation module sequentially acquires the production data according to the length sequence and forms the process data packets, thus, production data located at a certain length position, such as production data at a position with a length of 0-3 meters and production data … … at a position with a length of 3-6 meters are conveniently located, and are sequentially arranged and stored as process data packets, and the product data collecting module classifies the process data packets with the same product ID and sequentially stores the process data packets with the same product ID into the product data packets with the same product ID according to a time sequence, a length sequence, a data length (namely the length of data storage) or other modes (or a concentration sequence, a voltage sequence and the like). The product data package comprises product attribute data, the product attribute data are a set of product attribute items, the product attribute items correspond to the characteristic items, and the product data collection module collects the characteristic item data in the process data package into the product attribute data according to the product ID.

The product data collection module is connected with the data classification module, the data classification module is connected with the cluster storage system, the cluster storage system comprises a plurality of data file cluster servers, the data classification module stores product data packets into the data file cluster servers of the cluster storage system respectively, and the data classification module can also record storage paths in the storage process.

The invention also comprises a search client system which is in network communication connection with the cluster storage system and comprises a plurality of search clients, the search clients of the search client system send search instructions to the data file cluster servers of the cluster storage system, and the data file cluster servers of the cluster storage system execute the search instructions and transmit search result data to the search clients of the search client system in a communication way. The data file cluster server of the cluster storage system is internally provided with a plurality of search engine processes, each search engine process monopolizes a data link, each data link is provided with an independent hard disk, a memory block and a CPU (central processing unit) kernel, which is equivalent to the respective operation of each search engine process, a plurality of search engine processes of the data file cluster server synchronously operate, each search engine process of the data file cluster server monopolizes a data link, and the data links can realize the storage, index establishment and search of product data packet data. The index thread module establishes indexes and data association for the corresponding mapping of all product data packets or incremental product data packets stored in the data file cluster server, the indexes correspond to product attribute items, and the search instructions correspond to the indexes; the search thread module receives a search instruction and forms a search task, the search thread module forms a search task queue according to the receiving time, and the search thread module sequentially executes the corresponding search instruction according to the search task queue and correspondingly transmits search result data to the search client. The technical principle is as follows: the data file cluster server is internally provided with a plurality of search engine processes, each search engine process is provided with two independent thread modules, and the two thread modules are an index thread module and a search thread module respectively. Each search engine process operates independently, and the two thread modules under each search engine process operate independently respectively, so that the index thread module under each search engine process stores a product data packet and establishes index and data association respectively, the search thread module under each search engine process can receive a search instruction and form a search task, then the search thread module forms a search task queue according to the receiving time, and the search thread module sequentially executes corresponding search instructions according to the search task queue and correspondingly transmits search result data to a search client. The data classification module stores the product data packets into each data file cluster server of the cluster storage system respectively, and each search engine process of each data file cluster server completes independent storage of the product data packets respectively, so that each data file cluster server can perform synchronous storage through a plurality of search engine processes respectively, the storage speed is greatly improved, and the index thread modules under the search engine processes automatically establish indexes and data association respectively. When a search client sends a search instruction, all data file cluster servers synchronously receive the search instruction respectively, all data file cluster servers synchronously execute the search instruction and transmit search result data to the search client respectively, when the data file cluster servers search, all search engine processes in the data file cluster servers synchronously execute the search instruction (when the same search engine process receives a plurality of search instructions, a search thread module under the same search engine process receives one search instruction and forms a search task, the search thread module forms a search task queue according to the receiving time, the search thread module sequentially executes the corresponding search instruction according to the search task queue and correspondingly transmits the search result data to the search client), the search thread modules under each search engine process synchronously execute the search instruction and transmit the search result data to the search client respectively, the searching speed is greatly improved. The invention adopts a plurality of data file cluster servers, each data file cluster server adopts a plurality of search engine processes, so the total search task amount is not reduced, the search task amount is distributed or dispersed to each search engine process of each data file cluster server, the search task assignment is more balanced, the loads of each data file cluster server and each search engine process are relatively more balanced, the efficiency is greatly improved, the task amount, the task difficulty and the execution time of single-process search query are greatly reduced, the second-level data search query is realized, and the working efficiency is greatly improved.

The data file cluster server of the cluster storage system is internally provided with a plurality of search engine processes, each search engine process monopolizes one data link, each data link is provided with an independent hard disk, a memory block and a CPU (central processing unit) kernel, each search engine process is provided with two independent thread modules, the two thread modules are respectively an index thread module and a search thread module, the running priority of the search thread module of the same search engine process of the data file cluster server is greater than the running priority of the index thread module (namely the search thread module has the highest running priority), and the search thread module in the data file cluster server has the highest running priority. The search instruction includes a number of product attribute items (i.e., when the search thread module initiates a search task, the index thread module then stops to ensure that the search thread module runs preferentially). The index thread module establishes indexes and data association for all product data packets or incremental product data packets stored in the same search engine process of the data file cluster server in a corresponding mapping mode, the indexes correspond to product attribute items, and the search instructions correspond to the indexes. The search thread module receives a search instruction and forms a search task, the search thread module forms a search task queue according to the receiving time, and the search thread module sequentially executes the corresponding search instruction according to the search task queue and correspondingly transmits search result data to the search client.

The invention provides a search instruction query comprising a product attribute item or a plurality of product attribute items under product attribute data, and a condition search query of AND and or can be carried out on a search instruction which is usually corresponding to one or a plurality of product attribute items in the product attribute data. The product attribute items in the product attribute data correspond to the feature items in the feature item data, the set of the general product attribute items is larger than the set of the feature items, the feature items in the feature item data mainly come from the preset feature items, the product attribute items in the product attribute data mainly come from the feature items in the feature item data, the product attribute items contained in the search instruction can be selected and come from the product attribute items in the product attribute data, and a corresponding relation is established, the feature items in the feature item data can be preset or increased or decreased, the high-speed data acquisition operation module extracts and operates the acquired production data according to the feature items, the feature items of the production data are obtained and are stored in the feature item data together with the product ID, and the feature item data and the production data are collected into a process data packet. And the product data collecting module stores and merges the process data packets of the same product into the product data packet of the same product ID according to the product ID, and the product attribute items of the product attribute data in the product data packet are derived from the characteristic items of the process data packet and are combined and merged (the repeated characteristic items need to be deleted). As shown in fig. 3, condition 1, condition 2 … … in fig. 1 indicates that condition n is a product attribute item, and fig. 3 exemplifies the content of the product attribute item, where the product attribute item is a product ID, and the product ID is a unique identity code of a product (the product ID mentioned above); the product attribute items are date and time of production, wherein the date of production is the recorded year, month and day, and the time is the recorded time (accurate to millisecond); the product attribute items are material codes, such as the material of the product, and additional material of the product (such as a layer of copper attached to the product), such as Pb, Cu, Si, C, S and the like; the product attribute items are physical dimensions, such as physical dimensions of the product before processing and physical dimensions after processing, such as length, width, height and the like; the product attribute items are product performance, such as high temperature resistance, wear resistance, tensile strength, cut-off frequency, applicable environment and the like; the product attribute item is a model specification, such as a product model, a product specification or the like (a certain rule is necessary, and the product attribute item can represent the characteristics of some aspects of a product); the product attribute items are raw material conditions, such as chemical components, pH value, temperature and the like. The product attribute items are only examples, and the contents of the feature items and the product attribute items are determined according to actual situations. Each data file cluster server of the invention is provided with a plurality of search engine processes, each search engine process is executed by an independent process, each search engine process monopolizes a data link, the data file cluster server can establish indexes in a self-circulation manner, a trigger signal is sent out after the search is finished, the number of the data file cluster servers and the search allowable time are in an inverse correlation relationship, the inverse correlation is related to the number of the search engine processes for searching each data quantity and is unrelated to the number of the search clients, the invention can quickly finish the search and immediately return the search result.

The product data package storage system further comprises a total data backup server, the total data backup server is connected with the data classification module, the data classification module sequentially transmits all the product data packages to the total data backup server, and the total data backup server backups and stores the product data packages, so that industrial high-grade data of all the product data packages are stored in the total data backup server.

As shown in fig. 1 to 7, a method for fast cluster storage and search of industrial high-grade data includes an industrial production system, a product ID compiling and tracking system, a high-speed data acquisition system, a product data collection module, a data classification module, a cluster storage system and a search client system, the industrial production system completes product flow line production operation, the industrial production system sequentially includes a plurality of process production devices according to a production flow, each process production device correspondingly processes product process production of product flow line production, and the method includes the following steps:

A. and (3) industrial high-grade data classification cluster storage: the product ID writing and tracking system writes or reads product IDs into or from the same product in the industrial production system and completes the writing or reading work of all the products, the product IDs are unique identity codes of the products, production equipment of each process of the industrial production system has the same product ID for the same product, and the product IDs of the same product, which are acquired by each high-speed data acquisition operation module of the high-speed data acquisition system, are the same.

The high-speed data acquisition and operation module of the high-speed data acquisition system corresponds to process production equipment of a plurality of industrial production systems (namely one high-speed data acquisition and operation module can correspondingly acquire the production data of a plurality of process production equipment), the high-speed data acquisition and operation module correspondingly acquires the production data of the process production equipment and stores the production data as a process data packet together with a product ID (identity), the process data packet comprises characteristic item data, the characteristic item data is a set of characteristic items, and the high-speed data acquisition and operation module performs characteristic extraction and operation on the acquired production data and stores the characteristic items together with the product ID under the characteristic item data.

The product data collecting module classifies the process data packets with the same product ID according to the product ID and stores and merges the process data packets with the same product ID into product data packets with the same product ID (the product data collecting module of the invention classifies the product data packets according to the product ID and sequentially stores and merges the process data packets with the same product ID into the product data packets with the same product ID according to data length or other modes), the product data packets comprise product attribute data, the product attribute data are a set of product attribute items, the product attribute items correspond to the characteristic items, and the product data collecting module collects the characteristic item data in the process data packets into the product attribute data according to the product ID. The cluster storage system comprises a plurality of data file cluster servers, a product data collection module completes classification and combination work of product data packets of all product IDs in sequence, a data classification module stores the product data packets into all the data file cluster servers of the cluster storage system respectively, the data classification module records storage paths during storage, and all the product data packets in the cluster storage system form industrial high-grade data.

B. Searching industrial high-grade data: as shown in fig. 1 to 7, the search client system includes a plurality of search clients, a query search interface system is provided in each search client, the search clients issue search instructions to each data file cluster server of the cluster storage system through the query search interface system, and each search instruction includes a plurality of product attribute items.

As shown in fig. 5 and fig. 6, the index thread module of this embodiment represents thread 1, and the search thread module represents thread 2, and the process control is as follows:

firstly, a search thread module monopolizes hardware resources of related data links such as a hard disk, a memory block, a CPU kernel and the like.

And secondly, when no search task exists, the index thread module automatically and circularly establishes respective data indexes.

And thirdly, the simultaneous search thread module only executes 1 search condition, and the subsequent request is pressed into the queue.

Fourthly, the searching state and the result are returned to the client for initiating the search and a triggering signal for completing the search is sent out.

The search result of this embodiment is propagated in a point-to-point manner, that is, the search client interacts with the data corresponding to the data file cluster server.

And each data file cluster server is provided with a plurality of CPUs (central processing units), and each CPU has multiple cores, so that the number of the data file cluster servers can be reduced.

Each process has 2 threads, wherein the thread 1 is used for establishing indexes and receiving search instructions, the thread 2 is used for executing the search instructions and returning search results, and the search conditions are consistent with those in the figure 3.

Thread 2 has the highest priority, and thread 1 is the normal priority.

As shown in fig. 7, in step B of the present invention, the industrial high-score data search may be performed according to a data file system-based index structure, at this time, the index thread module of the present invention establishes index and data association, the search thread module forms a search task queue according to the receiving time, and the search thread module sequentially executes corresponding search instructions according to the search task queue and correspondingly transmits search result data to corresponding search clients.

As shown in fig. 8 and fig. 9, in step B of the present invention, the industrial high-score data search may be performed according to a data query search based on a database system, and the specific technical principle is as follows: in this embodiment, the query search interface system of the search client system supports SQL query, and the data file cluster server also supports SQL query, and the data file cluster server may also be referred to as a high-volume data file cluster server. The implementation method comprises the following steps: the query search interface system of the search client system supports SQL queries, as shown in fig. 1 to 6 and 8, the search client issues search instructions to each data file cluster server of the cluster storage system through the query search interface system.

Each data file cluster server of the cluster storage system is internally provided with a plurality of search engine processes (an engine in fig. 9 is short for a search engine process), each search engine process has an exclusive data link, each data link is provided with an independent hard disk, an independent memory block and a CPU kernel, each search engine process is provided with two independent thread modules, the two thread modules are respectively an index thread module and a search thread module, the index thread module establishes indexes and data associations for corresponding mapping of all product data packets or incremental product data packets stored in the data file cluster server, the indexes correspond to product attribute items, and search instructions correspond to the indexes.

The search thread module receives a search instruction and forms a search task, the search thread module forms a search task queue according to the receiving time, the search thread module sequentially executes the corresponding search instruction according to the search task queue and correspondingly transmits search result data to the corresponding search client, and the query client reports comprehensive search results after the search of all cluster servers is completed.

The operation priority of the search thread module is greater than that of the index thread module (the search thread module inside the data file cluster server has the highest operation priority, that is, the search thread module has the highest operation priority), and the operation method of the data file cluster server is as follows:

and B11, when the search thread module of the data file cluster server does not receive the search instruction or is idle, the index thread module of the data file cluster server operates in an autonomous loop and establishes index and data association.

B12, when the search thread module of the data file cluster server receives the search instruction, the index thread module of the data file cluster server stops working, the search thread module of the data file cluster server starts and sequentially establishes a search task queue according to the time of receiving the search instruction, the search thread module sequentially executes the corresponding search instruction according to the search task queue and correspondingly transmits the search result data to the corresponding search client, and the index thread module starts working again until the search thread module finishes executing all the search instructions.

The invention relates to a method for rapidly storing and searching a cluster of industrial high-quality data, which comprises a total data backup server and further comprises the following steps:

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.

Claims

1. The utility model provides a quick cluster storage search system of industry high score data, includes industrial production system, industrial production system includes a plurality of process production equipment according to production procedure in proper order, its characterized in that: the system comprises a product ID compiling and tracking system, a high-speed data acquisition system, a product data collection module, a data classification module and a cluster storage system, wherein the high-speed data acquisition system comprises a plurality of high-speed data acquisition and operation modules, the product ID compiling and tracking system is connected with the high-speed data acquisition system, the product ID compiling and tracking system is used for writing or reading a product ID into or from the same product in an industrial production system and completing the writing or reading work of all products, the product ID is a unique identity code of the product, the high-speed data acquisition and operation modules of the high-speed data acquisition system correspond to process production equipment of the plurality of industrial production systems, and the high-speed data acquisition and operation modules correspondingly acquire production data of the process production equipment and store the production data together with the product ID into a process data packet; the high-speed data acquisition system is connected with a product data collection module and a complex mathematical operation module, the data collection module is connected with the complex mathematical operation module, the product data collection module classifies according to product IDs and stores and combines process data packets of the same product ID into product data packets of the same product ID, and the product data collection module completes the classification and combination of the product data packets of all the product IDs; the product data collection module is connected with the data classification module, the data classification module is connected with the cluster storage system, the cluster storage system comprises a plurality of data file cluster servers, and the data classification module stores product data packets into the data file cluster servers of the cluster storage system respectively.

2. The industrial high-score data rapid cluster storage search system of claim 1, wherein: the process data packet comprises characteristic item data, the characteristic item data is a set of characteristic items, and the high-speed data acquisition and operation module performs characteristic extraction operation on the acquired production data and stores the characteristic item data together with the product ID (identity) into the characteristic item data; the product data package comprises product attribute data, the product attribute data is a set of product attribute items, the product attribute items correspond to the characteristic items, and the product data collection module collects the characteristic item data in the process data package into the product attribute data according to the product ID; the high-speed data acquisition operation module of the high-speed data acquisition system respectively stores the process data packets in sequence according to a time sequence and a logic sequence, the product data collection module stores the product data packets in sequence according to the logic sequence, the logic sequence is one or more of a length sequence, a concentration sequence, a voltage sequence and the like, the time sequence is the acquisition time of the production data, and the length sequence is the product length of the acquired production data.

3. The industrial high-score data rapid cluster storage search system of claim 1, wherein: the system further comprises a total data backup server, the total data backup server is connected with the data classification module, and the total data backup server backs up the storage product data packages.

4. The industrial high-grade data rapid cluster storage and search system according to any one of claims 1 to 3, characterized in that: the system comprises a cluster storage system and a search client side system, wherein the cluster storage system is connected with the search client side system in a network communication mode, the search client side system comprises a plurality of search client sides, the search client sides of the search client side systems send search instructions to data file cluster servers of the cluster storage system, and the data file cluster servers of the cluster storage system execute the search instructions and transmit search result data to the search client sides of the search client side systems in a communication mode.

5. The industrial high-score data rapid cluster storage search system of claim 4, wherein: the data file cluster server of the cluster storage system is internally provided with a plurality of search engine processes, each search engine process monopolizes one data link, each data link is provided with an independent hard disk, a memory block and a CPU (central processing unit) kernel, each search engine process is provided with two independent thread modules which are respectively an index thread module and a search thread module, the index thread module establishes index and data association for mapping all product data packets or incremental product data packets stored in the data file cluster server correspondingly, the index corresponds to a product attribute item, and the search instruction corresponds to the index; the search thread module receives a search instruction and forms a search task, the search thread module forms a search task queue according to the receiving time, and the search thread module sequentially executes the corresponding search instruction according to the search task queue and correspondingly transmits search result data to the search client.

6. The industrial high-score data rapid cluster storage search system of claim 5, wherein: the searching thread module in the data file cluster server has the highest operation priority which is greater than that of the indexing thread module, and the searching instruction comprises a plurality of product attribute items.

7. The industrial high-score data rapid cluster storage search system of claim 1, wherein: the product ID writing tracking system can adopt any one of the following two technologies, accurate tracking must be realized, and the ID number and related data can be acquired by a data acquisition system:

the first technique is: the product ID compiling and tracking system comprises a plurality of product ID compiling units, the product ID compiling units of the product ID compiling and tracking system correspond to process production equipment of a plurality of industrial production systems, proximity sensors or counting sensors connected with the product ID compiling units are correspondingly arranged on the process production equipment of the industrial production systems, the product ID compiling units write product IDs according to the same rule method, and the product ID compiling units in the product ID compiling and tracking system write the same product ID for the same product;

the second technique is: the product ID compiling and tracking system comprises a product ID compiling unit and a plurality of product ID reading modules, the high-speed data acquisition system is correspondingly provided with the product ID compiling unit in the high-speed data acquisition and operation module of the first procedure, and the product ID compiling unit is used for writing an identifiable code with a product ID into a product and simultaneously feeding back the product ID to the high-speed data acquisition and operation module; the high-speed data acquisition system is provided with product ID reading modules corresponding to the high-speed data acquisition operation modules in the rest processes respectively, and the product ID reading modules are used for identifying the identifiable codes on the products, reading the product IDs and feeding the product IDs back to the high-speed data acquisition operation modules.

8. The industrial high-score data rapid cluster storage search system of claim 1, wherein: the setting method of the data file cluster server comprises the following steps:

the number of the data file cluster servers is related to the search allowable time, the search data volume and the number of search engines of each data file cluster server, and is not related to the number of the search clients; each data file cluster server is provided with a plurality of search engine processes, each search engine is executed by an independent process, each search engine process monopolizes related data link resources, the data link resources comprise hard disks, internal memories and CPU (central processing unit) kernels, each data file cluster server comprises a plurality of physical hard disks, a large internal memory and a plurality of CPU kernels, and the data file cluster server has no redundancy and backup functions;

the data file cluster server is a high-speed cluster database server, the high-speed cluster database server supports SQL statement query, the search client sends an SQL search instruction to all the high-speed cluster database servers, and the search client reports the comprehensive query result according to all the SQL query results.

9. A method for quickly storing and searching industrial high-grade data clusters is characterized by comprising the following steps: the system comprises an industrial production system, a product ID compiling and tracking system, a high-speed data acquisition system, a product data collection module, a data classification module, a cluster storage system and a search client side system, wherein the industrial production system finishes production line production operation of products, the industrial production system sequentially comprises a plurality of process production devices according to a production flow, and each process production device correspondingly processes production process production of the products produced by the product production line, and the method comprises the following steps:

the high-speed data acquisition and operation module correspondingly acquires production data of the process production equipment and stores the production data together with the product ID into a process data packet, the process data packet comprises characteristic item data, the characteristic item data is a set of characteristic items, and the high-speed data acquisition and operation module performs characteristic extraction operation on the acquired production data and stores the characteristic items together with the product ID into the characteristic item data;

B. searching industrial high-grade data: the search client side system comprises a plurality of search client sides, a query search interface system is arranged in each search client side, the search client sides respectively issue search instructions to each data file cluster server of the cluster storage system through the query search interface system, and each search instruction comprises a plurality of product attribute items;

10. The industrial high-score data rapid cluster storage and search method according to claim 9, characterized in that: the method comprises a total data backup server and further comprises the following steps:

and B, the search client side sends the search instruction in the step B, a broadcast mode is adopted to ensure that each process of each cluster server can almost simultaneously receive the search instruction, and the data file cluster servers of the cluster storage system immediately respond after receiving the search instruction so as to facilitate the search client side to take measures and execute the search command.