CN116939045A - Intelligent optimization method, device, equipment and storage medium - Google Patents
Intelligent optimization method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN116939045A CN116939045A CN202310928846.0A CN202310928846A CN116939045A CN 116939045 A CN116939045 A CN 116939045A CN 202310928846 A CN202310928846 A CN 202310928846A CN 116939045 A CN116939045 A CN 116939045A
- Authority
- CN
- China
- Prior art keywords
- task
- processed
- analysis result
- intelligent
- machine room
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000005457 optimization Methods 0.000 title claims abstract description 58
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000012545 processing Methods 0.000 claims abstract description 20
- 230000008569 process Effects 0.000 claims abstract description 14
- 238000004458 analytical method Methods 0.000 claims description 104
- 238000013508 migration Methods 0.000 claims description 55
- 230000005012 migration Effects 0.000 claims description 55
- 210000004556 brain Anatomy 0.000 claims description 40
- 238000004422 calculation algorithm Methods 0.000 claims description 40
- 238000004364 calculation method Methods 0.000 claims description 40
- 230000005540 biological transmission Effects 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 6
- 239000008280 blood Substances 0.000 claims description 5
- 210000004369 blood Anatomy 0.000 claims description 5
- 238000005516 engineering process Methods 0.000 abstract description 5
- 230000009286 beneficial effect Effects 0.000 abstract description 3
- 238000004891 communication Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000013500 data storage Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000007405 data analysis Methods 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The intelligent optimization method is beneficial to solving the problem that the effective dispatching across the IDC machine room cannot be realized in the related technology, and determining the target Internet data center machine room and the target computing engine to process the task to be processed, so that the processing efficiency of the task to be processed is improved.
Description
Technical Field
The present application relates to the field of computers, and in particular, to an intelligent optimization method, apparatus, device, and storage medium.
Background
The integrated lake and storehouse has become a new direction of large data platform evolution, has data increment reading capability, realizes real-time stream processing scene support, and supports wider application scenes such as BI analysis besides data analysis scenes.
The integrated lake and warehouse platform is provided with a machine room which is physically distributed across IDCs (Internet Data Center, internet data centers), equipment is arranged in the machine room, and the equipment is combined with an engine to process tasks. However, the existing lake and bin integrated platform cannot realize effective dispatching across an IDC machine room, and cannot effectively determine efficient task processing across the IDC machine room.
Disclosure of Invention
The embodiment of the application provides an intelligent optimization method, device, equipment and storage medium, which are used for solving at least one problem existing in the related technology, and the technical scheme is as follows:
in a first aspect, an embodiment of the present application provides a method for intelligent optimization, including:
receiving a task to be processed sent by a lake and warehouse integrated platform;
performing predictive analysis on the task to be processed through an intelligent optimization algorithm to obtain a predictive analysis result;
and sending the prediction analysis result to the lake and warehouse integrated platform, wherein the prediction analysis result is used for the lake and warehouse integrated platform to determine a target internet data center machine room and a target calculation engine so as to process the task to be processed.
In one embodiment, the performing, by using an intelligent optimization algorithm, predictive analysis on the task to be processed, to obtain a predictive analysis result includes:
obtaining a target calculation engine according to the task to be processed and a prediction algorithm;
and obtaining the target internet data center machine room according to the task to be processed and the machine room optimization algorithm.
In one embodiment, the obtaining the target calculation engine according to the task to be processed and the prediction algorithm includes:
predicting the execution time of the task to be processed in different calculation engines to obtain the corresponding predicted execution time of the different calculation engines;
and determining the computing engine with the minimum predicted execution time as a target computing engine.
In one embodiment, the obtaining the target internet data center room according to the task to be processed and the room optimization algorithm includes:
determining machine room distances among a plurality of different internet data center machine rooms;
performing blood margin analysis on the task statement of the task to be processed to obtain data topology information;
and determining the Internet data center machine room with the lowest transmission cost as a target Internet data center machine room according to the machine room distance and the data topology information.
In one embodiment, the method further comprises:
receiving an operation log;
performing heat analysis on the operation data of the operation log to obtain a heat analysis result of the operation data;
when the heat analysis result meets the heat condition, generating a first migration result, otherwise, generating a second migration result;
and sending the first migration result or the second migration result to the lake and warehouse integrated platform, and moving the operation data to a hot data cluster according to the first migration result or moving the operation data to a cold data cluster according to the second migration result by using the lake and warehouse integrated platform.
In a second aspect, an embodiment of the present application provides another intelligent optimization method, including:
the task to be processed is sent to the intelligent brain;
receiving a prediction analysis result returned by the intelligent brain; the prediction analysis result is obtained by performing prediction analysis on the task to be processed by the intelligent brain through an intelligent optimization algorithm;
determining a target internet data center room and a target calculation engine according to the prediction analysis result;
and processing the task to be processed according to the target internet data center machine room and the target calculation engine.
In one embodiment, the method further comprises:
sending an operation log to the intelligent brain;
receiving a first migration result or a second migration result; the first migration result is generated when the heat analysis result meets heat conditions after the intelligent brain performs heat analysis on the operation data of the operation log to obtain the heat analysis result of the operation data, and the second migration result is generated when the heat analysis result does not meet the heat conditions;
and moving the operation data to a hot data cluster according to the first migration result, or moving the operation data to a cold data cluster according to the second migration result.
In a third aspect, an embodiment of the present application provides an intelligent optimization apparatus, including:
an intelligent brain or lake storehouse integrated platform;
the intelligent brain includes:
the first receiving module is used for receiving a task to be processed sent by the integrated lake and warehouse platform;
the analysis module is used for carrying out predictive analysis on the task to be processed through an intelligent optimization algorithm to obtain a predictive analysis result;
the first sending module is used for sending the prediction analysis result to the lake and warehouse integrated platform, and the prediction analysis result is used for the lake and warehouse integrated platform to determine a target internet data center machine room and a target calculation engine so as to process the task to be processed;
the integrative platform of lake storehouse includes:
the second sending module is used for sending the task to be processed to the intelligent brain;
the second receiving module is used for receiving a prediction analysis result returned by the intelligent brain; the prediction analysis result is obtained by performing prediction analysis on the task to be processed by the intelligent brain through an intelligent optimization algorithm;
the determining module is used for determining a target internet data center machine room and a target calculation engine according to the prediction analysis result;
and the processing module is used for processing the task to be processed according to the target internet data center machine room and the target calculation engine.
In a fourth aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory in which instructions are stored, the instructions being loaded and executed by the processor to implement the method of any of the embodiments of the above aspects.
In a fifth aspect, embodiments of the present application provide a computer readable storage medium storing a computer program, which when executed implements a method in any one of the embodiments of the above aspects.
The beneficial effects in the technical scheme at least comprise:
the method comprises the steps of receiving a task to be processed sent by a lake and warehouse integrated platform, carrying out predictive analysis on the task to be processed through an intelligent optimization algorithm to obtain a predictive analysis result, and sending the predictive analysis result to the lake and warehouse integrated platform, wherein the predictive analysis result is used for the lake and warehouse integrated platform to determine a target internet data center machine room and a target computing engine, so that the problem that effective scheduling across an IDC machine room cannot be achieved in the related technology is solved, and the target internet data center machine room and the target computing engine are determined to process the task to be processed, so that the processing efficiency of the task to be processed is improved.
The foregoing summary is for the purpose of the specification only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of the present application will become apparent by reference to the drawings and the following detailed description.
Drawings
In the drawings, the same reference numerals refer to the same or similar parts or elements throughout the several views unless otherwise specified. The figures are not necessarily drawn to scale. It is appreciated that these drawings depict only some embodiments according to the disclosure and are not therefore to be considered limiting of its scope.
FIG. 1 is a flow chart illustrating steps of an intelligent optimization method according to an embodiment of the present application;
FIG. 2 is a flowchart illustrating steps of an intelligent optimization method according to another embodiment of the present application;
FIG. 3 is a block diagram of an intelligent optimization apparatus according to an embodiment of the present application;
fig. 4 is a block diagram of an electronic device according to an embodiment of the application.
Detailed Description
Hereinafter, only certain exemplary embodiments are briefly described. As will be recognized by those of skill in the pertinent art, the described embodiments may be modified in various different ways without departing from the spirit or scope of the present application. Accordingly, the drawings and description are to be regarded as illustrative in nature and not as restrictive.
Referring to FIG. 1, a flow chart of an intelligent optimization method according to an embodiment of the present application is shown, and the intelligent optimization method may at least include steps S100-S300:
and S100, receiving a task to be processed sent by the lake and warehouse integrated platform.
S200, carrying out predictive analysis on the task to be processed through an intelligent optimization algorithm to obtain a predictive analysis result.
And S300, sending a prediction analysis result to the integrated lake and warehouse platform, wherein the prediction analysis result is used for determining a target internet data center machine room and a target calculation engine by the integrated lake and warehouse platform so as to process a task to be processed.
In the embodiment of the application, a user can initiate a task to be processed through the integrated lake and warehouse platform, and the integrated lake and warehouse platform receives the task to be processed and then sends the task to the intelligent brain.
In the embodiment of the application, after the lake and reservoir integrated platform receives the prediction analysis result, the lake and reservoir integrated platform responds to the prediction analysis result to determine the target internet data center machine room and the target calculation engine from a plurality of internet data center machine rooms and a plurality of calculation engines of the lake and reservoir integrated platform.
The intelligent optimization method of the embodiment of the application can be executed by an intelligent brain, and the intelligent brain can be deployed in a terminal such as a computer, a mobile phone, a tablet, a vehicle-mounted terminal or a cloud server.
According to the technical scheme, the intelligent brain receives the task to be processed, which is sent by the integrated lake and warehouse platform, carries out predictive analysis on the task to be processed through the intelligent optimization algorithm, obtains a predictive analysis result, sends the predictive analysis result to the integrated lake and warehouse platform, and the predictive analysis result is used for the integrated lake and warehouse platform to determine the target internet data center machine room and the target calculation engine, so that the problem that effective dispatching across the IDC machine room cannot be achieved in the related technology is solved, the target internet data center machine room and the target calculation engine are determined to process the task to be processed, and the processing efficiency of the task to be processed is improved.
In one embodiment, the intelligent optimization algorithm includes a prediction algorithm and a machine room optimization algorithm, and step S200 includes steps S210-S220:
s210, obtaining a target calculation engine according to the task to be processed and the prediction algorithm.
In one embodiment, the processing of steps S2101-S2102 may be performed on the task to be processed by a prediction algorithm to obtain a target computing engine:
s2101, predicting the execution time of the task to be processed in different calculation engines to obtain the corresponding predicted execution time of the different calculation engines.
In the embodiment of the application, the intelligent brain can read or store all calculation engines and all internet data center machine rooms of the lake and warehouse integrated platform in real time, and as different data analysis tasks consume different resources and time through different calculation engines, the intelligent analysis is beneficial to selecting the optimal calculation engine. Specifically, after receiving a task to be processed, according to a DAG graph (directed acyclic graph) of the task to be processed and input data corresponding to the task to be processed, the execution time of the task to be processed in different computing engines can be predicted, so that predicted execution times corresponding to different computing engines can be obtained. Alternatively, the prediction algorithm for prediction is implemented by a prediction model obtained by training in advance, and the prediction model may be trained by acquiring a history task execution log as training data according to the blood edges of task statements such as SQL statements, task execution plans, and actual time consumption of the history task, so that the prediction model outputs predicted execution time. In the embodiment of the application, a lake and warehouse integrated platform can be provided with calculation engines of big data such as MR, TEZ, spark and the like, a Task execution plan is obtained by taking the Spark calculation engine as an example, the execution plan DAG graph of the Spark calculation engine is adopted, each Spark Task execution process comprises a plurality of stages, stage classifications are divided into different classifications according to processing logic, each Stage comprises a plurality of tasks, a historical Task execution log can obtain the execution time of each Stage comprising the Task, and for a newly submitted Task, the execution time of each Task is predicted through a prediction model according to the DAG graph of the newly submitted Task and input data thereof, and the execution time prediction of the whole Task is obtained through summarization:
T Job ＝{Stage i |0≤i≤M}
Stage i ＝{Task i,j |0≤j≤N}
wherein M is the number of stages of Task execution, N is the number of tasks of each Stage, T Job Representing the execution time predicted by the predictive model.
S2102, determining the computing engine with the minimum predicted execution time as a target computing engine.
In the embodiment of the application, the computing engine with the smallest predicted execution time is determined as the target computing engine, so that the optimal computing engine is determined, and the purposes of intelligent optimization of the computing engine and intelligent selection of the computing engine are achieved.
And S220, obtaining the target internet data center machine room according to the task to be processed and the machine room optimization algorithm.
In one embodiment, the tasks to be processed may be processed in steps S2201-S2203 by a machine room optimization algorithm, to obtain the target internet data center machine room:
s2201, determining the machine room distance among a plurality of different internet data center machine rooms.
Optionally, the independent deployment of the yacn clusters in each internet data center room provides computing resource scheduling, and since the lake and warehouse integrated platform provides multiple kinds of computing engines, the physical yacn clusters distributed in each internet data center room also need to independently deploy computing engine instances, including MR, TEZ, spark, etc., and finally the computing engines are formed by heterogeneous multiple computing engine instances of each yacn cluster of multiple internet data center rooms. Meanwhile, data enters a lake according to a data source nearby principle, the data are scattered in a plurality of different internet data center machine rooms, bandwidth cost among the machine rooms is high, task statement input and output data of cross-domain associated data analysis tasks cross a plurality of cross-IDC clusters, and data transmission quantity among the cross-IDCs caused by executing tasks in the different internet data center machine rooms is different. In the embodiment of the application, the machine room optimization algorithm can determine the machine room distances among a plurality of different internet data center machine rooms, the machine room distances represent the data transmission cost, and the machine room distances can be expressed by a two-bit array.
S2202, performing blood margin analysis on task sentences of the tasks to be processed to obtain data topology information.
In the embodiment of the application, the task statement can be an SQL statement, for example, a machine room optimization algorithm can perform blood-margin analysis on the SQL statement of the task to be processed, and analyze the blood-margin relation of the SQL statement, namely, the logic relation among the SQL statements, such as the father-son relation among the statements, so as to take the task to input the data machine room, the data quantity and output the data machine room, thereby forming the data topology information.
S2203, determining the Internet data center machine room with the lowest transmission cost as the target Internet data center machine room according to the machine room distance and the data topology information.
In the embodiment of the application, the computer room optimization algorithm comprises a graph analysis algorithm, and the computer room distance and the data topology information are combined through the graph analysis algorithm, so that the internet data center computer room with the lowest transmission cost is determined to be the target internet data center computer room, and the target internet data center computer room is taken as the submitting computer room of the task to be processed.
In one embodiment, step S2203 may further include an algorithm modification step S2204: and (3) combining the target internet data center machine room inferred by the graph algorithm, combining the occupation condition of the resources of each machine room, acquiring the occupation condition of the computing resources of the target internet data center machine room, correcting the target internet data center machine room into a low-cost machine room when the occupation condition represents no idle resources, returning to the step S2203 to determine a new target internet data center machine room until the final target internet data center machine room is confirmed, and ensuring the normal and efficient execution of the task to be processed.
In the embodiment of the application, the target internet data center room is determined by the room optimization algorithm, so that the internet data center room with the lowest data transmission cost across the internet data center room is intelligently optimized and intelligently selected.
In one implementation manner, the intelligent optimization method of the embodiment of the present application further includes steps S410 to S440:
s410, receiving the operation log.
Optionally, the operation log may be an operation log generated when the integrated platform of the lake and the storehouse executes various tasks and is sent to the intelligent brain through the integrated platform of the lake and the storehouse.
S420, performing heat analysis on the operation data of the operation log to obtain a heat analysis result of the operation data.
Optionally, the intelligent brain may perform a heat analysis on the operation data of the operation log, analyze the SQL blood margin in the operation data to obtain partition access actions and statistics data table partition level access indexes, obtain a heat analysis result of the operation data, where the heat analysis result includes, for example, but not limited to, the number of accesses on the day, the number of accesses on the week, the number of accesses on the month, and the like, and provide a data index basis for data dynamic migration.
S430, when the heat analysis result meets the heat condition, generating a first migration result, otherwise, generating a second migration result.
It should be noted that the heat condition may be customized according to the need without specific limitation, and when the heat analysis result satisfies the heat condition, the first migration result is generated, and when the heat analysis result does not satisfy the heat condition, the second migration result is generated, and when the heat analysis result satisfies the heat condition, the first migration result is considered to be generated, and the data is considered to be required to be moved to the cold data. Illustratively, hot data and cold data are stored in a hierarchical/partitioned manner in the integrated lake and warehouse platform, for example, the hot data and the cold data are stored by using different clusters, for convenience of describing that the cluster storing the hot data is called a hot data cluster (short for hot cluster), the cluster storing the cold data is called a cold data cluster (short for cold cluster), and the heat condition may be:
1) Migrating operation data in the hot cluster corresponding to the current month access frequency of 0 to the cold cluster, wherein the current month access frequency of 0;
2) The cold data in the cold cluster is accessed more than 3 times within 1 week, and then is migrated to the hot cluster;
3) The hot cluster threshold is greater than 80%, triggering migration of operational data located at the hot clusters to the cold clusters.
4) And other rules for complex migration are designed according to the multi-dimension of the clusters and the data indexes.
S440, the first migration result or the second migration result is sent to the integrated lake and warehouse platform, and the integrated lake and warehouse platform moves the operation data to hot data according to the first migration result or moves the operation data to cold data according to the second migration result.
Optionally, when the first migration result or the second migration result is obtained and sent to the integrated lake and warehouse platform, the first migration result or the second migration result corresponds to a decision of data migration, and the integrated lake and warehouse platform moves the operation data to the hot data cluster based on the first migration result or moves the operation data to the cold data cluster according to the second migration result.
In the embodiment of the application, the intelligent scheduling of cold and hot data is realized through the analysis of the operation log by the intelligent brain, the intelligent migration of the cold and hot data is realized, the data access efficiency is guaranteed, and the comprehensive cost of data storage is reduced.
Referring to fig. 2, a flow chart of an intelligent optimization method according to another embodiment of the present application, which may be performed by a lake and reservoir integrated platform, may include at least steps S500-S800:
s500, sending the task to be processed to the intelligent brain.
S600, receiving a prediction analysis result returned by the intelligent brain.
The prediction analysis result is obtained by performing prediction analysis on the task to be processed by the intelligent brain through an intelligent optimization algorithm, and is not described in detail with reference to step S200.
And S700, determining a target internet data center room and a target calculation engine according to the prediction analysis result.
S800, processing tasks to be processed according to the target Internet data center machine room and the target calculation engine.
Specifically, the task to be processed needs to be submitted to a cluster in a specific internet data center room to be processed by using a specific computing engine, namely, the task to be processed is processed by using a target computing engine through a target internet data center room. For example, the intelligent brain gives out the prediction analysis result as the internet data center room a and the computing engine B, and at this time, the lake and warehouse integrated platform receives the prediction analysis result, analyzes and extracts the content of the prediction analysis result, so as to determine the internet data center room a as the target internet data center room and determine the computing engine B as the target computing engine.
It should be noted that, the integrated platform of the lake and the storehouse in the related art needs to design different development interfaces according to different engine characteristics and adaptation scenes, and has a high development threshold.
In one implementation manner, the intelligent optimization method of the embodiment of the application further includes steps S910 to S930:
s910, sending the operation log to the intelligent brain.
S920, receiving the first migration result or the second migration result.
The first migration result is generated when the heat analysis result meets the heat condition after the intelligent brain performs heat analysis on the operation data of the operation log to obtain the heat analysis result of the operation data, and the second migration result is generated when the heat analysis result does not meet the heat condition, which is described in reference to step S420.
And S930, moving the operation data to the hot data cluster according to the first migration result, or moving the operation data to the cold data cluster according to the second migration result.
In the related art, the integrated lake and warehouse platform adopts a unified storage scheme of cold and hot data, and the comprehensive storage cost is high. In the embodiment of the application, the cold data and the hot data are stored in a grading manner, so that the access efficiency of the application data is ensured and the comprehensive cost of the data is reduced. For ease of description, the cluster storing hot data is referred to as a hot data cluster (hot cluster for short) and the cluster storing cold data is referred to as a cold data cluster (cold cluster for short).
Optionally, the hot cluster prioritizes IO performance, and through typical big data server bearers (typical configuration: 2 x 12core/256G/6 x 12T SATA), data high availability can be achieved through 3 copies, resulting in an effective data storage utilization of 1/3; the cold cluster prioritizes storage costs, adopts a high-density storage server (typical configuration: 2 x 12core/256G/30 x 12t SATA), and the data reliability is realized by adopting an erasure code mechanism, so that the data availability is high and the effective storage utilization rate is 2/3, and the additional 50% storage is required. The single TB data storage cost for cold data storage is reduced by more than 80% over the hot data scheme.
In the embodiment of the application, after the lake and warehouse integrated platform receives the migration decision of the first migration result or the second migration result, the operation data is moved to the hot data according to the first migration result or the operation data is moved to the cold data according to the second migration result, so that the hot data and the cold data are dynamically adjusted, and the high access efficiency of a user is ensured.
In the embodiment of the application, the automatic operation of the integrated lake and bin platform is realized through the interaction of the integrated lake and bin platform and the intelligent brain, the optimization decision is realized through the intelligent technology in the aspects of storage, calculation, unified dispatching across IDCs and the like, the cost is reduced, and the efficiency and the usability are improved.
Referring to FIG. 3, a block diagram of an intelligent optimization apparatus, which may include an intelligent brain or lake-warehouse integrated platform, is shown, according to one embodiment of the present application, wherein:
the intelligent brain includes:
the first receiving module is used for receiving a task to be processed sent by the integrated lake and warehouse platform;
the analysis module is used for carrying out predictive analysis on the task to be processed through the intelligent optimization algorithm to obtain a predictive analysis result;
the first sending module is used for sending a prediction analysis result to the lake and warehouse integrated platform, wherein the prediction analysis result is used for the lake and warehouse integrated platform to determine a target internet data center machine room and a target calculation engine so as to process a task to be processed;
the integrative platform of lake storehouse includes:
the second sending module is used for sending the task to be processed to the intelligent brain;
the second receiving module is used for receiving a prediction analysis result returned by the intelligent brain; the prediction analysis result is obtained by performing prediction analysis on the task to be processed by the intelligent brain through an intelligent optimization algorithm;
the determining module is used for determining a target internet data center machine room and a target calculation engine according to the prediction analysis result;
and the processing module is used for processing the task to be processed according to the target internet data center room and the target calculation engine.
The functions of each module in each device of the embodiments of the present application may be referred to the corresponding descriptions in the above methods, and are not described herein again.
Referring to fig. 4, a block diagram of an electronic device according to an embodiment of the present application is shown, the electronic device including: memory 310 and processor 320, the memory 310 stores instructions executable on the processor 320, and the processor 320 loads and executes the instructions to implement the intelligent optimization method in the above embodiment. Wherein the number of memory 310 and processors 320 may be one or more.
In one embodiment, the electronic device further includes a communication interface 330 for communicating with an external device for data interactive transmission. If the memory 310, the processor 320 and the communication interface 330 are implemented independently, the memory 310, the processor 320 and the communication interface 330 may be connected to each other and communicate with each other through buses. The bus may be an industry standard architecture (Industry Standard Architecture, ISA) bus, peripheral interconnect (Peripheral ComponentInterconnect, PCI) bus, or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, among others. The bus may be classified as an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in fig. 4, but not only one bus or one type of bus.
Alternatively, in a specific implementation, if the memory 310, the processor 320, and the communication interface 330 are integrated on a chip, the memory 310, the processor 320, and the communication interface 330 may communicate with each other through internal interfaces.
An embodiment of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the intelligent optimization method provided in the above embodiment.
The embodiment of the application also provides a chip, which comprises a processor and is used for calling the instructions stored in the memory from the memory and running the instructions stored in the memory, so that the communication equipment provided with the chip executes the method provided by the embodiment of the application.
The embodiment of the application also provides a chip, which comprises: the input interface, the output interface, the processor and the memory are connected through an internal connection path, the processor is used for executing codes in the memory, and when the codes are executed, the processor is used for executing the method provided by the application embodiment.
It should be appreciated that the processor may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (digital signal processing, DSP), application specific integrated circuits (application specific integrated circuit, ASIC), field programmable gate arrays (fieldprogrammablegate array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or any conventional processor or the like. It is noted that the processor may be a processor supporting an advanced reduced instruction set machine (advanced RISC machines, ARM) architecture.
Further, optionally, the memory may include a read-only memory and a random access memory, and may further include a nonvolatile random access memory. The memory may be volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile memory may include a read-only memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an electrically Erasable EPROM (EEPROM), or a flash memory, among others. Volatile memory can include random access memory (random access memory, RAM), which acts as external cache memory. By way of example, and not limitation, many forms of RAM are available. For example, static RAM (SRAM), dynamic RAM (dynamic random access memory, DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous DRAM (SLDRAM), and direct memory bus RAM (DR RAM).
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions in accordance with the present application are fully or partially produced. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. Computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present application, the meaning of "a plurality" is two or more, unless explicitly defined otherwise.
Any process or method description in a flowchart or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process. And the scope of the preferred embodiments of the present application includes additional implementations in which functions may be performed in a substantially simultaneous manner or in an opposite order from that shown or discussed, including in accordance with the functions that are involved.
Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions.
It is to be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the methods of the embodiments described above may be performed by a program that, when executed, comprises one or a combination of the steps of the method embodiments, instructs the associated hardware to perform the method.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing module, or each unit may exist alone physically, or two or more units may be integrated in one module. The integrated modules may be implemented in hardware or in software functional modules. The integrated modules described above, if implemented in the form of software functional modules and sold or used as a stand-alone product, may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic or optical disk, or the like.
The foregoing is merely illustrative of the present application, and the present application is not limited thereto, and any person skilled in the art will readily recognize that various changes and substitutions are possible within the scope of the present application. Therefore, the protection scope of the application is subject to the protection scope of the claims.
Claims (10)
1. An intelligent optimization method is characterized by comprising the following steps:
receiving a task to be processed sent by a lake and warehouse integrated platform;
performing predictive analysis on the task to be processed through an intelligent optimization algorithm to obtain a predictive analysis result;
and sending the prediction analysis result to the lake and warehouse integrated platform, wherein the prediction analysis result is used for the lake and warehouse integrated platform to determine a target internet data center machine room and a target calculation engine so as to process the task to be processed.
2. The intelligent optimization method according to claim 1, wherein: the task to be processed is subjected to predictive analysis through an intelligent optimization algorithm, and the obtaining of a predictive analysis result comprises the following steps:
obtaining a target calculation engine according to the task to be processed and a prediction algorithm;
and obtaining the target internet data center machine room according to the task to be processed and the machine room optimization algorithm.
3. The intelligent optimization method according to claim 2, wherein: the obtaining the target calculation engine according to the task to be processed and the prediction algorithm comprises the following steps:
predicting the execution time of the task to be processed in different calculation engines to obtain the corresponding predicted execution time of the different calculation engines;
and determining the computing engine with the minimum predicted execution time as a target computing engine.
4. The intelligent optimization method according to claim 2, wherein: the obtaining the target internet data center machine room according to the task to be processed and the machine room optimization algorithm comprises the following steps:
determining machine room distances among a plurality of different internet data center machine rooms;
performing blood margin analysis on the task statement of the task to be processed to obtain data topology information;
and determining the Internet data center machine room with the lowest transmission cost as a target Internet data center machine room according to the machine room distance and the data topology information.
5. The intelligent optimization method according to any one of claims 1-4, wherein: further comprises:
receiving an operation log;
performing heat analysis on the operation data of the operation log to obtain a heat analysis result of the operation data;
when the heat analysis result meets the heat condition, generating a first migration result, otherwise, generating a second migration result;
and sending the first migration result or the second migration result to the lake and warehouse integrated platform, and moving the operation data to a hot data cluster according to the first migration result or moving the operation data to a cold data cluster according to the second migration result by using the lake and warehouse integrated platform.
6. An intelligent optimization method is characterized by comprising the following steps:
the task to be processed is sent to the intelligent brain;
receiving a prediction analysis result returned by the intelligent brain; the prediction analysis result is obtained by performing prediction analysis on the task to be processed by the intelligent brain through an intelligent optimization algorithm;
determining a target internet data center room and a target calculation engine according to the prediction analysis result;
and processing the task to be processed according to the target internet data center machine room and the target calculation engine.
7. The intelligent optimization method according to claim 6, wherein: further comprises:
sending an operation log to the intelligent brain;
receiving a first migration result or a second migration result; the first migration result is generated when the heat analysis result meets heat conditions after the intelligent brain performs heat analysis on the operation data of the operation log to obtain the heat analysis result of the operation data, and the second migration result is generated when the heat analysis result does not meet the heat conditions;
and moving the operation data to a hot data cluster according to the first migration result, or moving the operation data to a cold data cluster according to the second migration result.
8. An intelligent optimization device, characterized by comprising: an intelligent brain or lake storehouse integrated platform;
the intelligent brain includes:
the first receiving module is used for receiving a task to be processed sent by the integrated lake and warehouse platform;
the analysis module is used for carrying out predictive analysis on the task to be processed through an intelligent optimization algorithm to obtain a predictive analysis result;
the first sending module is used for sending the prediction analysis result to the lake and warehouse integrated platform, and the prediction analysis result is used for the lake and warehouse integrated platform to determine a target internet data center machine room and a target calculation engine so as to process the task to be processed;
the integrative platform of lake storehouse includes:
the second sending module is used for sending the task to be processed to the intelligent brain;
the second receiving module is used for receiving a prediction analysis result returned by the intelligent brain; the prediction analysis result is obtained by performing prediction analysis on the task to be processed by the intelligent brain through an intelligent optimization algorithm;
the determining module is used for determining a target internet data center machine room and a target calculation engine according to the prediction analysis result;
and the processing module is used for processing the task to be processed according to the target internet data center machine room and the target calculation engine.
9. An electronic device, comprising: a processor and a memory in which instructions are stored, the instructions being loaded and executed by the processor to implement the method of any one of claims 1 to 7.
10. A computer readable storage medium having stored therein a computer program which when executed implements the method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310928846.0A CN116939045A (en) | 2023-07-26 | 2023-07-26 | Intelligent optimization method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310928846.0A CN116939045A (en) | 2023-07-26 | 2023-07-26 | Intelligent optimization method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116939045A true CN116939045A (en) | 2023-10-24 |
Family
ID=88393992
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310928846.0A Pending CN116939045A (en) | 2023-07-26 | 2023-07-26 | Intelligent optimization method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116939045A (en) |
-
2023
- 2023-07-26 CN CN202310928846.0A patent/CN116939045A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109783237B (en) | Resource allocation method and device | |
| CN112465146B (en) | Quantum and classical hybrid cloud platform and task execution method | |
| CN110689121A (en) | Method for realizing neural network model splitting by using multi-core processor and related product | |
| CN113590508B (en) | Dynamic reconfigurable memory address mapping method and device | |
| CN111782185B (en) | Management method, equipment and medium of artificial intelligent development platform | |
| CN111858034A (en) | Resource management method, system, device and medium | |
| CN111897493B (en) | Storage space management method and device, electronic equipment and storage medium | |
| CN115033340A (en) | Host selection method and related device | |
| CN112667405A (en) | Information processing method, device, equipment and storage medium | |
| Choi et al. | An enhanced data-locality-aware task scheduling algorithm for hadoop applications | |
| CN116939045A (en) | Intelligent optimization method, device, equipment and storage medium | |
| CN115934354A (en) | Online storage method and device | |
| US20210149746A1 (en) | Method, System, Computer Readable Medium, and Device for Scheduling Computational Operation Based on Graph Data | |
| CN113918098A (en) | Data processing method, device, equipment and medium | |
| CN113468442A (en) | Resource bit flow distribution method, computing device and computer storage medium | |
| CN109145052B (en) | Data partition storage method, device, system, storage medium and electronic device | |
| CN114138444A (en) | Task scheduling method, device, equipment, storage medium and program product | |
| CN115150402B (en) | Cloud resource allocation method and system | |
| CN116974714A (en) | Data processing method, device, computer equipment and storage medium | |
| CN114398339A (en) | Data scheduling method and device and electronic equipment | |
| CN116483558A (en) | Resource management method, system, device, processor and electronic equipment | |
| CN115033254A (en) | Application pre-deployment method, device, equipment and storage medium | |
| CN116226081A (en) | Database elastic expansion method and device, electronic equipment and storage medium | |
| CN113900799A (en) | Computing resource allocation method and device, electronic equipment and storage medium | |
| CN118034885A (en) | Task processing method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |