CN115964369A

CN115964369A - Aluminum industrial data supplementary acquisition method, device, medium and electronic equipment

Info

Publication number: CN115964369A
Application number: CN202211549136.9A
Authority: CN
Inventors: 王跃勇; 张艳芳; 刘巧云; 李琰; 赵清杰; 周益文; 宋转
Original assignee: Aluminum Corp of China Ltd
Current assignee: Aluminum Corp of China Ltd
Priority date: 2022-12-05
Filing date: 2022-12-05
Publication date: 2023-04-14

Abstract

The application relates to the technical field of aluminum industry, and discloses a method, a device, a medium and electronic equipment for supplementing and acquiring aluminum industry data. The method comprises the following steps: acquiring a data complementary acquisition task list in a preset database at intervals of preset time; sequentially performing data supplement collection on the supplement collection tasks in the data supplement collection task list; monitoring the collection state of the complementary collection task; after each supplementary mining task is completed, generating a collecting record; converting first target data corresponding to the complementary mining task into storable data in an ORC format based on the acquisition record; and storing the storable data to a hadoop data platform or storing first target data corresponding to the complementary mining task to a relational database based on the acquisition record. Through the data acquisition method and device, corresponding acquisition records can be obtained after each time of data acquisition, and the data of the time period which is not acquired can be acquired according to actual needs according to the acquisition records, so that the purpose of data acquisition can be achieved.

Description

Aluminum industrial data supplement acquisition method, device, medium and electronic equipment

Technical Field

The present disclosure relates to the field of aluminum industry technologies, and in particular, to a method, an apparatus, a medium, and an electronic device for supplementing and acquiring data in aluminum industry.

Background

Production enterprises in the aluminum industry generate massive production, operation and other data every day, and the data are generally stored in the enterprises of molecular companies to form data islands and business islands. In order to meet the technical analysis and decision requirements, data needs to be collected from the business systems of all molecular companies for centralized storage, data management is carried out, data quality is improved, and effective data assets are formed. Compared with the fields of e-commerce, finance and service, the informatization of aluminum industry enterprises is relatively backward, and a plurality of enterprise business systems are old in technology and difficult to achieve a good effect on incremental acquisition through database structural design.

For example, a table stores a plurality of service data fields, but only the service date field of the data, and no last update timestamp field of the data. Incremental data acquisition can only be performed based on the business date, and in practice, different departments often update different fields of the same record at different times. If the enterprise updates the data again after incremental collection, data missing can result. The problem of data missing collection can not be effectively solved in the prior art, and then the situations of data confusion, data missing and the like are caused.

Disclosure of Invention

The application provides a method, a device, a medium and electronic equipment for supplementing and acquiring aluminum industrial data, which are used for solving the problem of data missing in aluminum industrial production.

Other features and advantages of the present application will be apparent from the following detailed description, or may be learned by practice of the application.

According to an aspect of an embodiment of the present application, there is provided an aluminum industry data supplementary collection method, including:

acquiring a data complementary acquisition task list in a preset database at each interval of preset time, wherein the data complementary acquisition task list comprises at least one complementary acquisition task, and the complementary acquisition task is used for acquiring first target data of a target object in a target time interval in the database;

sequentially performing data supplement acquisition on the supplement and acquisition tasks in the data supplement and acquisition task list;

monitoring the acquisition state of the complementary mining task, wherein the acquisition state is used for representing the execution state of the complementary mining task;

after each complementary mining task is completed, generating a collecting record, wherein the collecting record comprises collecting starting time;

converting first target data corresponding to the complementary mining task into storable data in an ORC format based on the acquisition record;

and storing the storable data to a hadoop data platform or storing first target data corresponding to the complementary mining task to a relational database based on the acquisition record.

In an embodiment of the present application, based on the foregoing scheme, the performing supplementary acquisition on the supplementary acquisition tasks in the data supplementary acquisition task list in sequence includes:

sequencing each supplementary mining task according to a time sequence based on the data supplementary mining task list to obtain a sequencing result;

and sequentially performing data supplementary acquisition on the supplementary acquisition tasks according to the sequencing result.

In an embodiment of the present application, based on the foregoing scheme, the performing supplementary data acquisition on the supplementary data acquisition task includes:

screening second target data corresponding to the target time interval from the database according to the target time interval;

querying the first target data corresponding to the target object in the second target data;

and performing supplementary acquisition on the first target data.

In an embodiment of the application, based on the foregoing scheme, the performing supplementary acquisition on the first target data includes:

converting first target data corresponding to the complementary mining task into a first target data stream;

adding the acquisition starting time corresponding to the acquisition record to the first target data stream as an attribute to obtain a second target data stream;

and performing supplementary acquisition on the second target data stream.

In an embodiment of the present application, based on the foregoing scheme, the converting the first target data corresponding to the complementary task into a first target data stream includes:

converting the first target data corresponding to the complementary mining task into third target data in a JSON format;

converting the third target data into the first target data stream.

In an embodiment of the present application, based on the foregoing solution, the converting the first target data corresponding to the complementary task into storable data in an ORC format based on the acquisition record includes:

acquiring a second target data stream corresponding to the first target data based on the acquisition record;

converting the second target data stream into storable data in an ORC format.

In an embodiment of the present application, based on the foregoing scheme, the acquisition state includes a completion state, a complementary acquisition state, and a to-be-complementary acquisition state; the monitoring of the collection state of the complementary collection task comprises the following steps:

if the complementary acquisition task completes data complementary acquisition, marking the acquisition state as a completion state;

if the complementary mining task is carrying out data complementary acquisition, marking the acquisition state as a complementary mining in-process state;

and if the supplementary acquisition task does not perform data supplementary acquisition, marking the acquisition state as a to-be-supplemented acquisition state.

According to an aspect of the embodiments of the present application, there is provided an aluminum industry data supplementary acquisition device, comprising: the data supplementing and acquiring system comprises an acquiring unit, a processing unit and a processing unit, wherein the acquiring unit is used for acquiring a data supplementing and acquiring task list in a preset database at intervals of preset time, the data supplementing and acquiring task list comprises at least one supplementing and acquiring task, and the supplementing and acquiring task is used for acquiring first target data of a target object in a target time interval in the database; the acquisition unit is used for sequentially performing data supplement acquisition on the supplement acquisition tasks in the data supplement acquisition task list; the monitoring unit is used for monitoring the acquisition state of the complementary acquisition task, and the acquisition state is used for representing the execution state of the complementary acquisition task; the recording unit is used for generating an acquisition record after each complementary acquisition task is completed, and the acquisition record comprises the acquisition starting time; the conversion unit is used for converting the first target data corresponding to the complementary mining task into storable data in an ORC format based on the acquisition record; and the storage unit is used for storing the storable data to a hadoop data platform or storing first target data corresponding to the complementary mining task to a relational database based on the acquisition record.

According to an aspect of an embodiment of the present application, there is provided a computer-readable storage medium on which a computer program is stored, the computer program comprising executable instructions that, when executed by a processor, implement the aluminum industrial data supplementary acquisition method as described in the above embodiment.

According to an aspect of an embodiment of the present application, there is provided an electronic device including: one or more processors; a memory for storing executable instructions of the processor, which when executed by the one or more processors, cause the one or more processors to implement the aluminum industry data supplemental acquisition method as described in the embodiments above.

According to the technical scheme of the embodiment of the application, the data supplement and acquisition task list in the preset database is obtained at intervals of preset time, and the data supplement and acquisition task list can be obtained in time under the condition of updating. And sequentially performing data supplement acquisition and monitoring the acquisition state of the supplement acquisition tasks through the supplement acquisition tasks in the data supplement acquisition task list, and performing data supplement acquisition in order. After the complementary acquisition task is completed, different fields in the record are updated through the generated acquisition record and the initial time in the acquisition record, so that the corresponding acquisition record is ensured after each time of data acquisition, and the data of the time period which is not acquired can be subjected to complementary acquisition according to the actual requirements according to the acquisition record, so that the purpose of data complementary acquisition is achieved. The first target data corresponding to the complementary mining task is converted into storable data in an ORC format and stored in a hadoop data platform, the required data can be quickly obtained by inquiring the hadoop data platform, or the first target data corresponding to the complementary mining task is stored in a relational database based on the acquisition record, at the moment, the format of the first target data does not need to be converted, and the acquired original format is directly reserved and stored in the relational database. The diversity of data storage modes is increased.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application. It is obvious that the drawings in the following description are only some embodiments of the application, and that for a person skilled in the art, other drawings can be derived from them without inventive effort. In the drawings:

fig. 1 is a flowchart illustrating an aluminum industry data supplementary acquisition method according to an embodiment of the present application;

FIG. 2 is a schematic flow chart illustrating data interaction according to an embodiment of the present application;

FIG. 3 is a schematic diagram illustrating a visualization human machine interface according to an embodiment of the present application;

fig. 4 is a schematic flow chart illustrating specific execution of a complementary mining task in an embodiment according to the present application;

fig. 5 is a schematic flow chart illustrating specific execution of a complementary mining task in another embodiment according to an embodiment of the present application;

fig. 6 is a flowchart illustrating a method for performing supplementary acquisition on the first target data according to an embodiment of the present application;

FIG. 7 is a block diagram of an aluminum industry data supplementary acquisition device according to an embodiment of the present application;

FIG. 8 is a schematic diagram of a computer-readable storage medium shown in accordance with an embodiment of the present application;

fig. 9 is a schematic diagram illustrating a system structure of an electronic device according to an embodiment of the present application.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the subject matter of the present application can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and so forth. In other instances, well-known methods, devices, implementations, or operations have not been shown or described in detail to avoid obscuring aspects of the application.

The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or micro-controlled node means.

The flowcharts shown in the figures are illustrative only and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.

It should be noted that: reference herein to "a plurality" means two or more. "and/or" describes the association relationship of the associated object, indicating that there may be three relationships, for example, a and/or B may indicate: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

The implementation details of the technical solution of the embodiment of the present application are set forth in detail below:

firstly, it should be noted that the aluminum industry data supplement and acquisition scheme provided in the present application can be applied to the related technical field of the aluminum industry, and the scheme is performed based on the NiFi technology, and performs data supplement and acquisition in sequence by acquiring the supplement and acquisition tasks in the data supplement and acquisition task list and monitoring the acquisition state of the supplement and acquisition tasks, and performs data supplement and acquisition in order. Different fields in the records are updated through the generated acquisition records and the initial time in the acquisition records, so that the corresponding acquisition records are ensured to exist after the data are acquired every time, and the data in the time period which is not acquired can be subjected to supplementary acquisition according to the acquisition records and the actual requirements, so that the purpose of supplementary acquisition of the data is achieved.

According to an aspect of the present application, there is provided an aluminum industry data supplementary acquisition method, and fig. 1 is a flowchart illustrating the aluminum industry data supplementary acquisition method according to an embodiment of the present application, where the aluminum industry data supplementary acquisition method includes at least steps 110 to 160, and is described in detail as follows:

in step 110, a data complementary acquisition task list in a preset database is obtained at preset time intervals, where the data complementary acquisition task list includes at least one complementary acquisition task, and the complementary acquisition task is used to acquire, in the database, first target data of a target object in a target time interval.

In the application, the preset time can be set according to actual needs, and the preset database comprises data in databases of various companies or enterprises to be inquired. The target object refers to a company or an enterprise needing data supplementary acquisition; the target time interval refers to a time period corresponding to the data supplementary acquisition, such as acquisition of data between 10 and 11 days of 2022 and 10 and 12 days of 2022.

The data complementary acquisition task list in the preset database can be obtained every preset time, wherein the data complementary acquisition task list can comprise one or more complementary acquisition tasks, and the first target data of the target object in the target time interval are acquired through the complementary acquisition tasks.

As will be further explained with reference to fig. 2, fig. 2 is a visualized human-computer interaction interface. The user can select information such as enterprise names, starting time, ending time, source table names and the like in the interface and write the information into the data complementary acquisition task list, the initial state of each complementary acquisition task is set to be a state to be complementary acquired, if the complementary acquisition task is being complementary acquired, the state is changed to be a state of complementary acquisition, and if the complementary acquisition task is completed, the state is changed to be a state of completion.

The states of all tasks can be checked on the current interface, and if the overtime is not completed, the complementary mining task can be repeatedly issued after the fault is eliminated. In fig. 3, a man-machine interaction component 1, a NiFi data processing component 2and a task list database 3 are included. The task list database 3 corresponds to a data complementary acquisition task list, and the data complementary acquisition task list is obtained through the task list database 3.

As shown in fig. 2, the human-computer interaction component 1 includes five parts, namely an enterprise name selection control 10, a time interval selection control 20, a data source to be acquired selection control 30, an execution button control 40, and a task state table control 50.

Initializing an interface control, filling the accessed enterprise name into an enterprise name selection control, filling the table names of a plurality of data source tables of each enterprise into a data source selection control 30, and selecting a time interval by default to R days before the current date, wherein R can be set to be any integer value larger than 0.

Selecting Enterprise E _i Time interval (T0-T1), and data source table name S _ij After clicking the execution button control, writing the three selected items of information into the task list database 3, and writing the task state ST _ij The initialization is 0. The task state table control 50 periodically refreshes the execution states of all the complementary tasks in the display task list database 3.

The NiFi data processing component 2 is arranged in each molecular company, can access the task list database 3, and can also access a data source database in an enterprise; for each enterprise E _i N data source tables S _ij Respectively adding N data complementary acquisition processing flows in the NiFi data processing component 2, wherein each processing flow F _ij Corresponding to a source data table S _ij 。

As shown in FIG. 4, data processing flow F _ij The data table query component 1 in the ExccuteSQL component 1 adds and regularly takes 1 enterprise name E from the task list database 3 _i Source data table S _ij And the task status is 0 _ij And (4) statement, if the statement is successful, continuing, and if the statement is failed, ending the flow. Data processing flow F _ij The format conversion component converts the convertertavrotojson component, and converts the avr format output in the previous step into the JSON format according to default setting, namely converts the first target data corresponding to the complementary acquisition task into third target data in the JSON format.

Data processing flow F _ij The EvaluateJsonPath component receives the output of the previous step, sets the Destination attribute to flowfile-attribute on the properties page, sets the Return Type to scale, adds two attributes P1 and P2, and sets the corresponding value to the field names of the time intervals T0 and T1 according to the JSONPath syntax format.

As shown in fig. 4, the acceutesql component 2 of the data processing flow Fij receives the output of the previous step, and can dynamically obtain the time intervals T0 and T1 to be complemented from the attributes P1 and P2, splice the T0 and T1 into the SQL Query statement according to the Expression syntax of NiFi, set up to the SQL Post-Query attribute, and then Query the data record to be complemented in the data table. The addition of the ConvertevroToORC and PutHDFS components stores the results into the hadoop big data platform, which is the conversion of the second target data stream into storable data in ORC format as described below.

As shown in fig. 5, in another embodiment of the present application, an execute SQL lrecord component is added to receive the output of the previous step, time intervals T0 and T1 that need to be complemented can be dynamically obtained from attributes P1 and P2, and T0 and T1 are spliced into an SQL Query statement according to the Expression syntax of NiFi and set to an SQL Post-Query attribute; a PutDatabaseRecord component is added to save the second target data stream to the target database. The target database may be a preset database according to actual needs.

The structure of the task list database tasklist table is shown in table 1:

TABLE 1

In an embodiment of the application, as shown in fig. 4, the implementation flow is that, in the time interval of the human-computer interaction component 1, the start time 2022-09-10 and the end time 2022-09-11 are selected, the enterprise selection control selects the first company, the Source library table control selects the slot control process report, and after the first company is clicked and executed, the program stores the numbers 1001, 2022-09-10, and 2022-09-11 corresponding to the first company, the table names gongyi _ rpt and the current dates 2022-10-07 corresponding to the slot control process report, and the task state 0 in the end _ id, ddate _ b, ddate _ e, source _ Name, ddate _ p, and Status fields of the task list database tasklist table.

The slot control process reporting and subsidizing process is established on a NiFi system of a company A, and the scheduling rule is executed once every 6 hours from an ExecuteSQL component 1. The "Database Connection point Service" attribute sets the task list Database access address, and the "SQL select query" attribute sets the record of the query task table tasklist with the filter condition of end _ id =1001and source _name = 'gongyi _ rpt' and Status = 0.

If no record is returned, the process ends and continues if a record is found.

Further, converting the record of the previous step into a json object according to default attributes by ConvertAvroToJSON;

further, the EvaluateJsonPath component adds three attributes mid, rq _ b, and rq _ e, whose corresponding values are set to $. Id, $. Ddate _ b, and $. Ddate _ e, respectively, from which the downstream component will obtain the current record to update the task state, and from which the rq _ b and rq _ e attributes read to start time 2022-09-10 and end time 2022-09-11.

Further, the PutSQL component 1 updates the current task record to set the task state Status =1

Further, the executeSQL component 2 sets a "Database Connection Pooling Service" attribute as the Database Connection information where the enterprise slot control process report gongyi _ rpt is located, and the SQL statement of the "SQL select query" attribute adds a filter condition of RecordDate > = '$ { rq _ b }' and RecordDate < = '$ { rq _ e }', where RecordDate is the Service date field of the slot control process report gongyi _ rpt.

Further, the ConvertAvroToORC component converts the upstream queried data stream into orc format.

Further, the PutHDFS component saves the data in the upstream orc format to the Hadoop data platform specified by the Hadoop Configuration Resources attribute.

Further, putSQL component 2 updates the current task record setting task state Status =2.

In another embodiment of the present application, as shown in fig. 5, the start time 2022-09-08 and the end time 2022-09-11 are selected in the time interval of the human-computer interaction component 1, the company b is selected by the enterprise selection control, the electrolyte assay is selected by the Source library table control, and after the electrolyte assay is clicked and executed, the program saves the numbers 1002, 2022-09-08 and 2022-09-11 corresponding to the company b, the table Name Qdata _ bath corresponding to the electrolyte assay, the current date 2022-10-07, and the task state 0 into the end _ id, ddate _ b, ddate _ e, source _ Name, ddate _ p and Status fields of the task list database tasklist table.

The method is deployed on a NiFi system of company B to establish an electrolyte testing and supplementing flow, and a scheduling rule is executed every 12 hours from an ExecuteQLRecord component 1. The "Database Connection point Service" attribute sets the access address of the task list Database, and the "SQL select query" attribute sets the record in the query task table with the filter condition of end _ id =1002and source_name = 'Qdata _ base' and Status = 0.

Further, a JsonRecordSetWriter component is set, and a Record Writer attribute of ExecuteQLRecord is set as the JsonRecordSetWriter component, so that data is converted into a json format;

further, if the executeQLRecord component does not return a record, the flow ends and the process continues if a record is found.

Further, the EvaluateJsonPath component adds three attributes mid, rq _ b, and rq _ e, whose corresponding values are set to $. Id, $. Ddate _ b, and $. Ddate _ e, respectively, and downstream components will get the current record through the mid attribute to update the task state and read to start time 2022-09-08 and end time 2022-09-11 through the rq _ b and rq _ e attributes.

Further, the executeQLRecord component 2 sets the attribute "Database Connection Pooling Service" as the Database Connection information where the enterprise electrolyte test Database table Qdata _ base is located, and the SQL statement of the attribute "SQL select query" adds the filtering condition of QDate > = '$ { rq _ b }' and QDate < = '$ { rq _ e }', where QDate is the business date field of the electrolyte test Database table Qdata _ base.

Further, the PutDatabaseRecord component stores the data queried in the previous step into the relational database.

With continued reference to fig. 1, in step 120, data supplementary acquisition is performed on the supplementary acquisition tasks in the data supplementary acquisition task list in sequence.

In an embodiment of the present application, the data supplementary acquisition is performed on the supplementary acquisition tasks in the data supplementary acquisition task list in sequence, and may be performed according to steps S1 to S2:

step S1: and sequencing each complementary mining task according to the time sequence based on the data complementary mining task list to obtain a sequencing result.

Step S2: and performing data supplementary acquisition on the supplementary acquisition tasks in sequence according to the sequencing result.

In the application, the complementary acquisition tasks are sequentially sequenced by writing the complementary acquisition tasks into the sequence of the data complementary acquisition task list to obtain a sequencing result, and then the data complementary acquisition is sequentially performed on the complementary acquisition tasks through the sequencing result.

In an embodiment of the present application, as shown in fig. 6, the performing of the supplementary acquisition on the first target data includes steps S21 to S23:

step S21: and converting the first target data corresponding to the complementary acquisition task into a first target data stream.

Step S22: and adding the acquisition starting time corresponding to the acquisition record into the first target data stream as an attribute to obtain a second target data stream.

Step S23: and performing supplementary acquisition on the second target data stream.

In the application, the first target data corresponding to the complementary acquisition task is converted into the first target data stream, that is, the task record obtained by querying in the task list database is converted into the JSON object, the acquisition start time corresponding to the acquisition record is added to the first target data stream as an attribute to obtain the second target data stream, and then the second target data stream is subjected to complementary acquisition.

In an embodiment of the application, the converting the first target data corresponding to the complementary task into the first target data stream includes steps S211 to S212.

Step S211: and converting the first target data corresponding to the complementary acquisition task into third target data in a JSON format.

Step S212: converting the third target data into the first target data stream.

In the application, the first target data corresponding to the complementary acquisition task is converted into third target data in a JSON format and then converted into a first target data stream.

With continued reference to fig. 1, in step 130, an acquisition state of the complementary task is monitored, and the acquisition state is used for characterizing an execution state of the complementary task.

In one embodiment of the present application, the acquisition state includes a completion state, a complementary acquisition state, and a to-be-complementary acquisition state; the monitoring of the acquisition state of the complementary acquisition task comprises the steps S31 to S33:

step S31: and if the complementary acquisition task completes the data complementary acquisition, marking the acquisition state as a completion state.

Step S32: and if the complementary acquisition task is performing data complementary acquisition, marking the acquisition state as a complementary acquisition in-process state.

Step S33: and if the supplementary acquisition task does not perform data supplementary acquisition, marking the acquisition state as a to-be-supplemented acquisition state.

In the application, if the complementary mining task is not subjected to complementary mining, the state is set to be in the state of complementary mining, if the complementary mining task is in the state of complementary mining, the state is changed to be in the state of complementary mining, and if the complementary mining task is finished, the state is changed to be in the state of finished.

With continued reference to fig. 1, in step 140, after each of the complementary tasks is completed, an acquisition record is generated, the acquisition record including a start time of acquisition.

In the application, the data of which time period of which enterprise is subjected to supplementary acquisition can be known by generating the acquisition record, so that repeated acquisition of the data of the same time period can be prevented by checking the acquisition record during subsequent data supplementary acquisition, and the efficiency of data supplementary acquisition is further improved.

With continued reference to fig. 1, in step 150, the first target data corresponding to the complementary tasks is converted into storable data in ORC format based on the acquisition records.

In an embodiment of the present application, the converting the first target data corresponding to the complementary mining task into storable data in an ORC format based on the collection record includes:

converting the second target data stream into storable data in ORC format.

In the application, the second target data stream is converted into the storable data in the ORC format, and then the storable data can be stored in a hadoop big data platform, so that the required data can be conveniently queried subsequently.

With continued reference to fig. 1, in step 160, the storable data is saved to a hadoop data platform or first target data corresponding to the complementary mining task is saved to a relational database based on the collection record.

In the application, the hadoop data platform is an open-source distributed storage and distributed computing platform, and the storable data is stored in the hadoop data platform, so that the data can be conveniently stored and called, and the data reading efficiency is further improved. Or storing the first target data corresponding to the complementary mining task in a relational database based on the acquisition record, wherein the format of the first target data does not need to be converted, and the original format acquired by acquisition is directly reserved and stored in the relational database.

Fig. 7 is a block diagram of an aluminum industry data supplementary acquisition device according to an embodiment of the present application.

Referring to fig. 7, an aluminum industry data supplementary collection device 700 according to an embodiment of the present application, the device 700 includes:

an obtaining unit 701, configured to obtain, at preset intervals, a data complementary acquisition task list in a preset database, where the data complementary acquisition task list includes at least one complementary acquisition task, and the complementary acquisition task is used to acquire, in the database, first target data of a target object in a target time interval;

the acquisition unit 702 is used for sequentially performing data supplementary acquisition on the supplementary acquisition tasks in the data supplementary acquisition task list;

a monitoring unit 703, configured to monitor an acquisition state of the replenishment task, where the acquisition state is used to represent an execution state of the replenishment task;

a recording unit 704, configured to generate a collection record after each of the complementary mining tasks is completed, where the collection record includes a collection start time;

a conversion unit 705, configured to convert the first target data corresponding to the complementary mining task into storable data in an ORC format based on the collection record;

the storage unit 706 is configured to store the storable data in a hadoop data platform or store the first target data corresponding to the complementary mining task in a relational database based on the acquisition record.

Referring to fig. 8, a program product 800 for implementing the above method according to an embodiment of the present application is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present application is not so limited, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

A computer readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

As another aspect, the present application further provides an electronic device capable of implementing the above method.

As will be appreciated by one skilled in the art, aspects of the present application may be embodied as a system, method or program product. Accordingly, various aspects of the present application may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.

An electronic device 900 according to this embodiment of the application is described below with reference to fig. 9. The electronic device 900 shown in fig. 9 is only an example and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.

As shown in fig. 9, electronic device 900 is in the form of a general purpose computing device. Components of electronic device 900 may include, but are not limited to: the at least one processing unit 910, the at least one memory unit 920, and a bus 930 that couples various system components including the memory unit 920 and the processing unit 910.

Wherein the storage unit stores program codes, which can be executed by the processing unit 910, so that the processing unit 910 performs the steps according to various exemplary embodiments of the present application described in the section "method of embodiment" mentioned above in this specification.

The storage unit 920 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM) 921 and/or a cache memory unit 922, and may further include a read only memory unit (ROM) 923.

Storage unit 920 may also include a program/utility 924 having a set (at least one) of program modules 925, such program modules 925 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.

Bus 930 can be any of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

The electronic device 900 may also communicate with one or more external devices 1200 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 900, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 900 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interface 950. Also, the electronic device 900 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet) via the network adapter 960. As shown, the network adapter 960 communicates with the other modules of the electronic device 900 via the bus 930. It should be appreciated that although not shown, other hardware and/or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to make a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) execute the method according to the embodiments of the present application.

Furthermore, the above-described figures are only schematic illustrations of the processes involved in the methods according to exemplary embodiments of the present application and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed, for example, synchronously or asynchronously in multiple modules.

It will be understood that the present application is not limited to the precise arrangements that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims

1. An aluminum industry data supplementary acquisition method, characterized in that the method comprises:

sequentially performing data supplement collection on the supplement collection tasks in the data supplement collection task list;

2. The aluminum industry data supplement collection method according to claim 1, wherein the data supplement collection of the supplementary collection tasks in the data supplement and collection task list is performed in sequence, and the method comprises the following steps:

and performing data supplementary acquisition on the supplementary acquisition tasks in sequence according to the sequencing result.

3. The aluminum industry data supplementary collection method of claim 2, wherein the supplementary collection of data for the supplementary collection task comprises:

querying the second target data for the first target data corresponding to the target object;

and performing supplementary acquisition on the first target data.

4. The aluminum industry data supplementary collection method of claim 3, wherein the supplementary collection of the first target data comprises:

and performing supplementary acquisition on the second target data stream.

5. The aluminum industry data supplementary collection method of claim 4, wherein the converting the first target data corresponding to the supplementary collection task into a first target data stream comprises:

converting the third target data into the first target data stream.

6. The aluminum industry data complementary collection method of claim 5, wherein converting the first target data corresponding to the complementary collection task into storable data in ORC format based on the collection record comprises:

converting the second target data stream into storable data in ORC format.

7. The aluminum industrial data supplement collection method according to claim 1, wherein the collection status includes a completion status, a replenishment in-process status, and a to-be-replenished status; the monitoring of the collection state of the complementary collection task comprises the following steps:

8. An aluminum industry data supplement acquisition device, the device comprising:

the acquisition unit is used for acquiring a data complementary acquisition task list in a preset database at intervals of preset time, wherein the data complementary acquisition task list comprises at least one complementary acquisition task, and the complementary acquisition task is used for acquiring first target data of a target object in a target time interval in the database;

the acquisition unit is used for sequentially performing data supplement acquisition on the supplement acquisition tasks in the data supplement acquisition task list;

the monitoring unit is used for monitoring the acquisition state of the complementary mining task, and the acquisition state is used for representing the execution state of the complementary mining task;

the recording unit is used for generating an acquisition record after each complementary acquisition task is completed, and the acquisition record comprises the acquisition starting time;

a conversion unit, configured to convert the first target data corresponding to the complementary mining task into storable data in an ORC format based on the collection record;

and the storage unit is used for storing the storable data to a hadoop data platform or storing first target data corresponding to the complementary mining task to a relational database based on the acquisition record.

9. A computer-readable storage medium having stored therein at least one program code, the at least one program code being loaded into and executed by a processor to perform operations performed by the method of any one of claims 1 to 7.

10. An electronic device, comprising one or more processors and one or more memories having at least one program code stored therein, the at least one program code being loaded into and executed by the one or more processors to perform operations performed by the method of any one of claims 1 to 7.