CN115129542A - Data processing method, data processing device, storage medium and electronic device - Google Patents

Abstract

The invention discloses a data processing method, device, storage medium and electronic device. The method includes: acquiring a target data package and configuration parameters corresponding to the target data package, wherein the target data package is a data package to be run, and the configuration parameters are used to determine resources to be used for running the target data package; The target placeholder in the file is replaced to obtain the target arrangement file, wherein the field where the target placeholder is located is the field to be adjusted; the target resource space is determined based on the target arrangement file, wherein the target resource space is used for the target The data package provides an independent running resource; the target data package is run based on the target resource space to obtain the running result. The present invention solves the technical problem of low data runtime efficiency in the resource space of the related art.

Description

Data processing method, device, storage medium and electronic device

technical field

The present invention relates to the field of big data, and in particular, to a data processing method, device, storage medium and electronic device.

Background technique

The workflow of billing big data can be simply described as collecting the logs related to login and payment of each game into a high-throughput distributed publish-subscribe messaging system through a lightweight transmission program for forwarding and centralizing log data. For storage, such logs are called business original logs. At present, in the process of running this kind of raw log data, all the jobs are entrusted to a system administrator for management, and the entire running environment is only based on one application of the resource manager. If the node or system administrator becomes unavailable, all the jobs of the entire cluster will be affected, making it difficult to run normally.

For the above problems, no effective solution has been proposed yet.

SUMMARY OF THE INVENTION

At least some embodiments of the present invention provide a data processing method, device, storage medium, and electronic device, so as to at least solve the technical problem of low data runtime efficiency in the resource space of the related art.

According to one embodiment of the present invention, a data processing method is provided, including: acquiring a target data packet and a configuration parameter corresponding to the target data packet, wherein the target data packet is a data packet to be run, and the configuration parameter is used to determine a running target The resource to be used by the data package; replace the target placeholder in the preset arrangement file based on the configuration parameters to obtain the target arrangement file, wherein the field where the target placeholder is located is the field to be adjusted; based on the target arrangement file Determine a target resource space, wherein the target resource space is used to provide an independent operating resource for the target data package; run the target data package based on the target resource space to obtain the operating result.

Optionally, acquiring configuration parameters corresponding to the target data packet includes: calling a first interface to analyze the target data packet, and obtaining target computing resources, where the target computing resources are used to represent resources required for computing the target data packet; The computing resource determines a first parameter, where the first parameter is the number of first components required to calculate the target computing resource in the target resource space; acquires a second parameter, where the second parameter is the first parameter used to manage the target resource space The number of two components; based on the first parameter and the second parameter, the configuration parameter is determined.

Optionally, the method further includes: acquiring a target keyword of an initial arrangement template, where the target keyword is a field to be adjusted in the initial arrangement template; adding a target underline to a preset position of the target keyword to obtain a target placeholder ; Adjust the initial arrangement template based on the target placeholder to obtain a preset arrangement file.

Optionally, the target resource space and the preset resource space share a log directory, and in the process of using the target resource space to run the target data package, the method further includes: collecting log files of the target resource space based on the preset resource space and the log directory; The log file monitors the process of running the target data package in the target resource space, and obtains a monitoring result, wherein the monitoring result is used to indicate whether an abnormality occurs in the process of running the target data package in the target resource space.

Optionally, the method further includes: in response to the target instruction for the preset resource space, stopping monitoring the processing process of the target resource space based on the log file, wherein the target instruction is in a preset time period after the target resource space stops running. generate.

Optionally, acquiring the target data package and the configuration parameters corresponding to the target data package includes: in response to a touch instruction to the first control in the first interface in the target platform, acquiring the target data package, wherein the first interface is used to retrieve the target data package. Determining a target data packet in at least one data packet; in response to a touch command on a second control of a second interface in the target platform, uploading configuration information corresponding to the target data packet to the target platform, wherein the second interface is used to monitor the target data Edit the configuration information corresponding to the package.

Optionally, using the target resource space to process the target data packet to obtain a processing result, including: acquiring first version information corresponding to the target resource space; determining a target image based on the first version information, wherein the target image is used to provide a running target The environment of the resource space; run the target data package based on the target image and the target resource space, and get the operation result.

Optionally, acquiring the first version information corresponding to the target resource space includes: in response to a touch instruction to a third control in the third interface in the target platform, acquiring the first version information corresponding to the target resource space, wherein the third The interface is used to edit the first version information corresponding to the target resource space.

Optionally, the method further includes: in response to an upgrade instruction for the target resource space, acquiring a second interface, where the second interface is associated with second version information, and the second version information is version information corresponding to the upgraded target resource space. ;Add the second interface to the third interface.

Optionally, before analyzing the target data packet based on the first interface to obtain the target computing resource, the method further includes: preprocessing the target data packet to obtain a preprocessing result, wherein the preprocessing result is used to represent the target data. Whether the package can run normally in the target environment; if the preprocessing result is that the target data package can run normally on the target resource, analyze the target data package based on the first interface to obtain the target computing resource.

According to an embodiment of the present invention, a data processing apparatus is also provided, the apparatus includes: an acquisition module, configured to acquire a target data packet and a configuration parameter corresponding to the target data packet, wherein the target data packet is a data packet to be executed , the configuration parameters are used to determine the resources to be used for running the target data package; the replacement module is used to replace the target placeholders in the preset arrangement file based on the configuration parameters to obtain the target arrangement file, where the target placeholder is located The field of the location is the field to be adjusted; the calling module is used to determine the target resource space based on the target arrangement file, wherein the target resource space is used to provide an independent running resource for the target data package; the processing module is used to determine the target resource space based on the target resource The space runs the target data package and obtains the running result.

According to one of the embodiments of the present invention, there is also provided a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, wherein the computer program is configured to be executed by a processor when running any one of the above the data processing method described.

According to one of the embodiments of the present invention, there is also provided an electronic device including a memory and a processor, a computer program is stored in the memory, and the processor is configured to run the computer program to execute any one of the above data processing methods.

In at least some embodiments of the present invention, by first obtaining the target data packet and the configuration parameters corresponding to the target data packet, wherein the target data packet is the data packet to be run, and the configuration parameter is used to determine the resource to be used for running the target data packet; Then, based on the configuration parameters, the target placeholder in the preset arrangement file is replaced to obtain the target arrangement file, wherein the field where the target placeholder is located is the field to be adjusted; then the target resource is determined based on the target arrangement file space, wherein the target resource space is used to provide an independent operating resource for the target data package; finally, the target data package is run based on the target resource space to obtain the operating result, thereby improving the operating efficiency of the target data package. It is easy to notice that the target arrangement file corresponding to the target data package can be quickly determined according to the configuration parameters and the preset arrangement file, and the target resource space corresponding to the target data package can be determined based on the target arrangement file, so as to facilitate the passage of the target data package. The resource space isolates the running process of the target data packet from the running process of other data packets, and provides an independent running resource for the target data packet, which can prevent the target data packet from being affected by others, thereby improving the operation efficiency of the target data packet. Further, the problem of low operation efficiency of the target data packet in the related art can be solved.

Description of drawings

The accompanying drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the attached image:

1 is a block diagram of a hardware structure of a mobile terminal of a data processing method according to an embodiment of the present invention;

2 is a flowchart of a data processing method according to an embodiment of the present invention;

FIG. 3 is a schematic diagram showing a running state according to an embodiment of the present invention;

4 is a schematic diagram of a user submission interface according to an embodiment of the present invention;

5 is a schematic diagram showing an editing operation interface according to an embodiment of the present invention;

FIG. 6 is a schematic diagram showing a startup operation interface according to an embodiment of the present invention;

7 is a schematic diagram of an overall framework according to an embodiment of the present invention;

8 is a structural block diagram of a data processing apparatus according to an embodiment of the present invention;

FIG. 9 is a schematic diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

In order to make those skilled in the art better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only Embodiments are part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "first", "second" and the like in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used may be interchanged under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

According to one of the embodiments of the present invention, an embodiment of a data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and , although a logical order is shown in the flowcharts, in some cases steps shown or described may be performed in an order different from that herein.

The method embodiments may be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, the mobile terminal can be a smart phone (such as an Android mobile phone, an iOS mobile phone, etc.), a tablet computer, a handheld computer, and a mobile Internet device (Mobile Internet Devices, referred to as MID), PAD, game consoles, etc. Terminal Equipment. FIG. 1 is a block diagram of a hardware structure of a mobile terminal according to a data processing method according to an embodiment of the present invention. As shown in FIG. 1 , the mobile terminal may include one or more (only one is shown in FIG. 1 ) processors 102 (the processors 102 may include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), a digital Processing devices for signal processing (DSP) chips, microprocessors (MCU), programmable logic devices (FPGA), neural network processors (NPU), tensor processors (TPU), artificial intelligence (AI) type processors, etc. ) and a memory 104 for storing data. Optionally, the above-mentioned mobile terminal may further include a transmission device 106 for communication functions, an input and output device 108 and a display device 110 . Those of ordinary skill in the art can understand that the structure shown in FIG. 1 is only a schematic diagram, which does not limit the structure of the above-mentioned mobile terminal. For example, the mobile terminal may also include more or fewer components than those shown in FIG. 1 , or have a different configuration than that shown in FIG. 1 .

The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the data processing method in the embodiment of the present invention. A functional application and data processing are implemented, that is, to implement the above-mentioned data processing method. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory located remotely from the processor 102, and these remote memories may be connected to the mobile terminal through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

Transmission device 106 is used to receive or transmit data via a network. The specific example of the above-mentioned network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC for short), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (Radio Frequency, RF for short) module, which is used to communicate with the Internet in a wireless manner.

The input in the input-output device 108 may come from a plurality of human interface devices (Human Interface Device, HID for short). For example: keyboard and mouse, gamepad, other specialized game controllers (eg: steering wheel, fishing rod, dance mat, remote control, etc.). In addition to providing input functions, some human interface devices can also provide output functions, such as force feedback and vibration of gamepads, and audio output of controllers.

Display device 110 may be, for example, a head-up display (HUD), a touch screen-style liquid crystal display (LCD), and a touch display (also referred to as a "touch screen" or "touch display"). The liquid crystal display may enable the user to interact with the user interface of the mobile terminal. In some embodiments, the above-mentioned mobile terminal has a graphical user interface (GUI), and the user can perform human-computer interaction with the GUI by touching finger contacts and/or gestures on the touch-sensitive surface, where the human-computer interaction function is optional Including the following interactions: creating web pages, drawing, word processing, making electronic documents, games, video conferencing, instant messaging, sending and receiving e-mails, calling interfaces, playing digital video, playing digital music and/or web browsing, etc. Executable instructions for machine interaction functions are configured/stored in one or more processor-executable computer program products or readable storage media.

First, the technical terms or technical terms that appear in the embodiments of the present invention are explained as follows:

Flink: Also known as Apache Flink, is a distributed processing engine and framework for stateful computing on bounded and unbounded data streams.

JobManager: The administrator of the Flink system, responsible for accepting and processing Flink job requests, generating execution plans for the jobs, and scheduling TaskManager to serve the job running, and also manages and records the status of the jobs, which can be referred to as JM for short.

TaskManager: The executor of the Flink system is the main component responsible for executing computing tasks, abbreviated as TM, and its essence is JVM, that is, Java Virtual Machine, which can also be called Java Virtual Machine.

CheckPoint: Flink's fault-tolerant recovery mechanism, the user can define the execution frequency, record the current running status of the job at regular intervals and store it in the database. When the job fails and restarts, the stored status can be read from the database and stored in the database. The job is restored to its current state

SavePoint: Flink's job recovery mechanism is a consistent mirror of the execution status of streaming jobs created by the Flink CheckPointing mechanism. Users need to manually execute, create, and delete them. Among them, Flink CheckPoint is used for restarting jobs after unexpected failures, and SavePoint is generally used for jobs. Scheduled restart.

FileBeat: A lightweight delivery program for forwarding and centralizing log data. In departments, it is mainly used for log collection and transmission to Kafka for storage.

Kafka: It is a high-throughput distributed publish-subscribe messaging system that can be used to store raw log information and provide it for consumption by Flink.

ZooKeeper: It is a software project that can provide open source distributed configuration services, synchronization services and naming registration for large-scale distributed computing. In Flink, it can be responsible for storing the metadata of job running and providing metadata required for job failure recovery, referred to as ZK.

RocksDB: It is an architecture engine that provides key-value storage and read-write functions. It can be used in Flink to store job running status during CheckPoint and SavePoint.

Kubernetes: is a portable and extensible open source platform for managing containerized workloads and services that facilitates declarative configuration and automation, or K8S for short, as the main tool for Flink resource regulation in this paper.

PersistentVolumeClaim: Persistent volume claim, which expresses the user's request for storage, referred to as PVC.

PersistentVolume: Persistent volume, a piece of storage in the cluster, referred to as PV.

Container: A container is a portable, executable lightweight image that contains the software and its related dependencies.

Image: It is an image, which is a saved container instance that can package a set of software required for the application to run.

HDFS: Hadoop distributed File System, also known as Hadoop distributed file system, is responsible for storing binary files generated in the process of SavePoint and CheckPoint in Flink.

JFlink: It is a platform for Flink users, which can be used to manage the start and stop of Flink cluster jobs, job life cycle control, job error troubleshooting, and version iteration of program running packages. Through this platform, you can maintain the job version, view and adjust the job status, etc.

Prometheus: Prometheus is a system monitoring and alarm toolkit developed in the form of open source software. Flink has a complete monitoring collection program and reporting process, which provides great help for Flink operation and maintenance.

Monitoring data: data that can reflect the running status of components in real time, often composed of time, monitoring indicators, and monitoring results.

Cluster configuration: Indicates the basic attributes of a Flink cluster, which is the necessary basis for the stable operation of Flink jobs. It is a configuration table customized by users according to the resource requirements of their own jobs.

Operation plan: The return result obtained by the Plan method in the official Flink provides the Flink job topology diagram, Flink operator job analysis and other results, which can pre-expose potential problems that the Flink job will encounter during the startup phase.

Orchestration file: The resource management and resource object orchestration and deployment in the K8S cluster are solved by declarative style files, which are called resource manifest files. In this article, the orchestration files are used to create the necessary components for Flink to run on K8S.

FIG. 2 is a flowchart of a data processing method according to one embodiment of the present invention. A graphical user interface is provided through a terminal device, and the content displayed on the graphical user interface includes a touch area. As shown in FIG. 2 , the method includes the following steps :

Step S202, acquiring the target data package and configuration parameters corresponding to the target data package, wherein the target data package is a data package to be run, and the configuration parameters are used to determine resources to be used for running the target data package.

The above target data package can be uploaded by the user on the job management platform. After the target data package is uploaded, the target data package can be configured to obtain configuration parameters. Optionally, the target data package may include logs related to game login, payment and the like. After obtaining the target data packet, the target data packet can be cleaned and filtered first, the wrong data and duplicate data can be deleted, the format can be converted into a format suitable for the job management platform, and the aggregation operation can be performed to classify it according to the data type. It is divided into different groups, and finally stored in the database. Later, when the target data package needs to be used, it can be retrieved from the database and placed on the job management platform for operation.

The above configuration parameters may include the memory size of the job management platform, the print level of the logger, and the like. Optionally, jobs such as JobManager and TaskManager can be configured, and the number of parallel slots supported by each TaskManager can be determined.

Step S204, the target placeholder in the preset arrangement file is replaced based on the configuration parameter to obtain the target arrangement file, wherein the field where the target placeholder is located is the field to be adjusted.

The above-mentioned preset arrangement file may be a template of an arrangement file reserved on the server in advance, and the configuration file on the template will have a placeholder filled in key fields for subsequent user modification.

The above-mentioned target placeholder can be used to temporarily replace the position of the field to be adjusted, and the field of the placeholder can be replaced with the filled parameter in subsequent use, so that the filled parameter can take effect successfully, and the target arrangement file can be further generated.

Among them, since the target placeholder needs to be frequently modified by the user, and the operation of the job is closely related to this field, therefore, the target placeholder can be represented in the following way, and can be distinguished by adding underscores before and after the placeholder field. In an optional embodiment, assuming that {{__NAMESPACE__}} is a template of an arrangement file, all target arrangement files can be derived according to the template.

For example, taskmanager.memory.process.size:__HEAP_SIZE_TM_M__ is a layout file template, and the parameter prefix and the end with an underscore are target placeholders. The user passes in the parameters corresponding to the target placeholder field through the API (ApplicationProgramming Interface), such as: __HEAP_SIZE_TM_M__: 2GB, then the API will fill in the parameters into the template of the arrangement file and replace it with taskmanager.memory.process.size : 2GB effect, and start the cluster, a set of processes allows users to provide a Flink running cluster that meets their needs according to their own job needs. If the user does not provide parameters, the API will use 2GB of memory as the parameter input by default.

Optionally, the configuration parameters in the template of the orchestration file may include the controller orchestration file of JobManager and TaskManager, the mount disk file, the domain name routing entry file (also called the image file), the Flink cluster configuration file, and the cluster start and stop scripts.

Optionally, by processing the keywords to be modified through placeholders, a preset arrangement file with placeholders can be obtained, thereby facilitating the use of different data packages.

Step S206, determining a target resource space based on the target arrangement file, wherein the target resource space is used to provide an independent operating resource for the target data package.

The above-mentioned target arrangement file may be generated from a preset arrangement file. Optionally, the placeholder of the preset arrangement file may be modified, and parameters may be filled in, thereby generating the target arrangement file. Among them, the target orchestration file is the key to correctly start the Flink cluster.

The above target resource space can be used to provide an independent running resource so that different data packets can be run in isolation, so that when a job's JobManager becomes unavailable, other jobs will not be affected. Optionally , the target resource space can be a Flink cluster.

In an optional embodiment, the namespace concept (also called the NameSpace concept) of K8S can be used to generate an independent NameSpace for each job of the user to ensure that the JobManager and TaskManager under the NameSpace only serve one job. That is, when a job A is submitted to the cluster, a resource space named N_A will be applied for the job A in K8S, and only the number of task managers and the number of JMs required to run the job are on it, where JM can be A model in which jobs exclusively occupy N_A resources and do not interfere with other job resource spaces such as N_B, N_C. In this way, jobs can be isolated from jobs and achieve the running mode of Per Job, that is, each job There is only one JobManager. That is, the target resource space can provide an independent running resource, so that different data packages that are running jobs at the same time can be separated. Even when the JobManager of a job becomes unavailable, it will not affect other jobs. progress.

Step S208, run the target data package based on the target resource space to obtain the running result.

In an optional embodiment, after the independent running resource has been determined, that is, after the target resource space, the target data package can be run to obtain the running result. FIG. 3 is a schematic diagram showing the running state according to one embodiment of the present invention. As shown in FIG. 3 , after the user presses the start button, the interface will display the running state in the form of a progress bar, wherein the progress bar may be a long bar shaped, or fan-shaped.

Through the above steps, first obtain the target data package and the configuration parameters corresponding to the target data package, wherein the target data package is the data package to be run, and the configuration parameters are used to determine the resources to be used for running the target data package; The target placeholder in the preset arrangement file is replaced to obtain the target arrangement file, wherein the field where the target placeholder is located is the field to be adjusted; then the target resource space is determined based on the target arrangement file, wherein the target resource The space is used to provide an independent operating resource for the target data package; finally, the target data package is run based on the target resource space to obtain the operating result, thereby improving the operating efficiency of the target data package. It is easy to notice that the target arrangement file corresponding to the target data package can be quickly determined according to the configuration parameters and the preset arrangement file, and the target resource space corresponding to the target data package can be determined based on the target arrangement file, so as to facilitate the passage of the target data package. The resource space isolates the running process of the target data packet from the running process of other data packets, and provides an independent running resource for the target data packet, which can prevent the target data packet from being affected by others, thereby improving the operation efficiency of the target data packet. Further, the problem of low operation efficiency of the target data packet in the related art can be solved.

Optionally, acquiring configuration parameters corresponding to the target data packet includes: calling a first interface to analyze the target data packet, and obtaining target computing resources, where the target computing resources are used to represent resources required for computing the target data packet; The computing resource determines a first parameter, where the first parameter is the number of first components required to calculate the target computing resource in the target resource space; acquires a second parameter, where the second parameter is the first parameter used to manage the target resource space The number of two components; based on the first parameter and the second parameter, the configuration parameter is determined.

The above-mentioned first interface may be an API interface, which is used to analyze the target data packet to obtain the target computing resource. Optionally, after the target data packet is analyzed to obtain the target computing resource, the user's request will be sent To the API background program, the first parameter is further determined based on the target computing resource.

The above-mentioned first parameter is the number of first components required to calculate the target computing resources in the target resource space, and optionally, the number of first components may be the number of TaskManagers.

The above-mentioned second parameter may be the number of second components for managing the target resource space, and optionally, the number of second components may be the number of JobManagers.

In an optional embodiment, when determining configuration parameters, the number of TaskManagers and the number of JobManagers may be determined first, and then the number of parallel slots supported by TaskManagers and the memory corresponding to JobManagers and TaskManagers may be determined. Size, logger print level and other related configuration parameters.

Optionally, the method further includes: acquiring a target keyword of an initial arrangement template, where the target keyword is a field to be adjusted in the initial arrangement template; adding a target underline to a preset position of the target keyword to obtain a target placeholder ; Adjust the initial arrangement template based on the target placeholder to obtain a preset arrangement file.

The above-mentioned preset positions may be the front and rear positions of the fields to be adjusted.

The above target underline may be an underline added at a preset position.

In an optional embodiment, a target underline may be added before and after the to-be-adjusted field of the initial layout template, and it is determined as the target placeholder to obtain a preset layout file with the target placeholder, optional In the subsequent use process, the user only needs to fill in the keyword used to replace the target placeholder on the target placeholder.

Taking TaskManager-deployment as an example, the key field can be replica data, where the number of replicas can be the number of work components, that is, one work component can process one task process at the same time. This field determines the number of TaskManagers, i.e. the number of main components that compute tasks. The job can run only when the number of TaskManagers is enough to support the maximum number of operators of a job. In the template, you can use the {{__SLOT_TM__}} keyword as a placeholder. When the API analyzes the maximum number of jobs in the user data packet After the number of operators, the placeholders will be replaced with this number to ensure that enough TaskManagers are started in the cluster for jobs to run.

Optionally, the target resource space and the preset resource space share a log directory, and in the process of using the target resource space to run the target data package, the method further includes: collecting log files of the target resource space based on the preset resource space and the log directory; The log file monitors the process of running the target data package in the target resource space, and obtains a monitoring result, wherein the monitoring result is used to indicate whether an abnormality occurs in the process of running the target data package in the target resource space.

The above preset resource space may share a log directory with the target resource space, so as to monitor the target resource space. Optionally, the preset resource space may be a Filebeat cluster.

In an optional embodiment, when collecting Flink logs, two K8S containers (Containers), Flink and Filebeat, can be processed in a shared Pod mode, where the Pod mode is a shared mode, and this mode is used. , Filebeat and Flink can share directories to make the log paths visible to each other, Flink logs can be printed on the log directory, and Filebeat collects log files in the log directory in real time. Among them, Filebeat can be used to forward and centralize log data. The lightweight delivery program is mainly used in the department for log collection and transmission to Kafka for storage to ensure that the log is collected.

In another optional embodiment, a monitoring data collection program (also called exporter) supported by Flink can be used to monitor components. After exporter is enabled in the configuration, the system will automatically collect the Java Virtual Machine (Java Virtual Machine) during Flink running. Also known as JVM) various states and report to a preset address, where the preset address can be set by the user, Prometheus in the same K8S cluster will recognize the preset address reported by the exporter, get the data according to the preset address and For storage, aggregation, and finally, the operation and maintenance personnel draw the data on the table of the open source data visualization tool (also called Grafana). Optionally, the monitoring data is reported to a preset address according to the monitoring state, and the monitoring data is processed based on the preset address, thereby obtaining a target table corresponding to the monitoring data.

Optionally, the method further includes: in response to the target instruction for the preset resource space, stopping monitoring the processing process of the target resource space based on the log file, wherein the target instruction is in a preset time period after the target resource space stops running. generate.

The above target instruction may be a computer instruction generated in a preset time period after the target resource space stops running, wherein the preset time may be specified by the user. Optionally, the target instruction may act on the target resource space, and in response to the target instruction, the monitoring performed during the processing of the target resource space based on the log file may be stopped.

In an optional embodiment, since the integrity of log collection needs to be considered, the following instructions can be added to Filebeat:

Further, using this command can make Filebeat stop the collection of Filebeat after Flink stops running, thus ensuring that the log can be collected completely.

Optionally, acquiring the target data package and the configuration parameters corresponding to the target data package includes: in response to a touch instruction to the first control in the first interface in the target platform, acquiring the target data package, wherein the first interface is used to retrieve the target data package. Determining a target data packet in at least one data packet; in response to a touch command on a second control of a second interface in the target platform, uploading configuration information corresponding to the target data packet to the target platform, wherein the second interface is used to monitor the target data Edit the configuration information corresponding to the package.

The above target platform can be oriented to Flink users, which can manage the start and stop of Flink cluster jobs, job life cycle control, job error troubleshooting, and version iteration of program running packages. Users can maintain job versions, view and adjust job status through this platform. Optionally, the target platform can be the JFlink platform.

The above-mentioned first control may be a control used to obtain the target data packet in the first interface. Optionally, when the user needs to obtain the target data packet, the user may perform a touch operation on the control to issue a touch command, based on the The touch command can acquire the target data packet.

The above-mentioned second control may be a control in the second interface for uploading the configuration information corresponding to the target data packet to the target platform. Optionally, when the user needs to upload the configuration information corresponding to the target data packet to the target platform, the The control performs a touch operation, thereby issuing a touch command, and based on the touch command, the configuration information corresponding to the target data packet can be uploaded to the target platform.

In an optional embodiment, the JFlink platform can manage data packages through job templates. Since data packages frequently iterate versions, data packages of multiple versions can exist in the job template at the same time. When it is introduced into the job template, after the user selects the job template when starting the job, he can view all versions of the data package, and through the "click to upload" operation, it can be uploaded to the data package list under the job template. And start the corresponding version of the data package according to its own needs, so as to complete the upload of the data package to the target platform.

In another optional embodiment, the job name, the Flink job running entry, the person in charge, the startup parameters, and the timed save point can be configured, so as to realize the editing of the job. Among them, the timed savepoint can be used to roll back to the previous state when it is found that the statement is written incorrectly when the previous statement is found to be executed. Optionally, when there are several versions of the background job template, each flink version is different, and the job should start the cluster by default according to the required version. Its main function is to determine the target version owner mainly for receiving alerts. Send alerts when something goes wrong.

Fig. 4 is a schematic diagram showing a user submission interface according to one embodiment of the present invention. As shown in Fig. 4, in the uploading program package interface, the user needs to fill in the template name, identification information, and project name, and the data can be uploaded after the filling is completed. package, the user can click the upload button, at the same time select the version suffix as no suffix, and fill in the version number, and click OK after all information is ensured.

Optionally, using the target resource space to process the target data packet to obtain a processing result, including: acquiring first version information corresponding to the target resource space; determining a target image based on the first version information, wherein the target image is used to provide a running target The environment of the resource space; run the target data package based on the target image and the target resource space, and get the operation result.

The above-mentioned first version information may be the version information of the target resource space before the upgrade, wherein the version information may be a version number, a version name, and the like.

The above target image can be used to provide an environment for running the target resource space. Based on the target image, it can be determined which version of the Flink image K8S will use to start the Flink cluster. Optionally, if the image version is missing, the operation and maintenance personnel can quickly package and make , in the absence of special environmental requirements, you can directly pull or update the specified image from the mirror repository, that is, use the docker pull flink command to obtain the image. If you have special environmental needs, you can follow the steps below:

Step 1: docker pull flink:version.

Pull the latest flink, version is the version required by the user.

Step 2: docker images|grep flink.

Check the image ID you just pulled.

Step 3: docker run-it image ID /bin/bash.

After entering the image, you can verify the next version through the application and keep it running. After entering the image, you can install the required basic software and create the environment according to the user's needs.

Step 4: docker ps to get the container ID.

After obtaining the container ID, you can perform operations such as file copying.

Step 5: File Copy.

docker cp, data container id: /opt/flink/lib/.

Step 6: Log in to the mirror center.

docker login url.

Step 7: Tag the mirror.

docker tag flink:version url:version.

Step 8: Push to mirror center.

docker push url:version.

Further, once a perfect image is made, it can be used by all users to achieve the effect once and for all.

Fig. 5 is a schematic diagram showing an editing job interface according to one embodiment of the present invention. As shown in Fig. 5, in the new job interface, the user can select a job template, and fill in the job name, and the job name can be controlled within 100 characters , and choose to start the cluster by default, specify the entrance, and the main person in charge, where the person in charge can be used to receive alerts and issue alerts when there is a problem with K8S. At the same time, describe the job, fill in the startup parameters, and select the savepoint configuration. The savepoint configuration can be selected from no configuration, configuration every hour, configuration every half day, or configuration every day. Optionally, there are several versions of the background job template, and each flink version is different. The job should start the cluster by default according to the required version. By editing the configuration of the job interface, the target version can be determined.

Optionally, acquiring the first version information corresponding to the target resource space includes: in response to a touch instruction to a third control in the third interface in the target platform, acquiring the first version information corresponding to the target resource space, wherein the third The interface is used to edit the first version information corresponding to the target resource space.

The above-mentioned third control may be a control of a third interface, wherein the third control may issue a touch command, and in response to the touch command, the system will acquire the first version information corresponding to the target resource space.

In an optional embodiment, the user can manually select the version of the data package to start the job, the cluster to start, and manually configure the cluster configuration required for running the job, where the parameters are reserved fields in the orchestration file.

FIG. 6 is a schematic diagram showing a startup job interface according to one embodiment of the present invention. As shown in FIG. 6 , in the startup job interface, you can first select a job template, a currently used version, a startup version, a startup cluster, and a resource policy. After selecting the above information, you need to fill in the number of task managers, the number of slots in each task manager, the memory of the task manager and the model memory, and you can click OK. If the information is wrong, you can click Cancel and try again. fill in.

Optionally, the method further includes: in response to an upgrade instruction for the target resource space, acquiring a second interface, where the second interface is associated with second version information, and the second version information is version information corresponding to the upgraded target resource space. ;Add the second interface to the third interface.

The above-mentioned second version information may be version information of the target resource space after the upgrade, where the version information may be a version number, a version name, and the like.

The above-mentioned second interface may be an interface newly built during the upgrade process.

In an optional embodiment, in the operation of upgrading the K8S cluster, a higher version of K8S can be built. At this time, it is only necessary to build a JAVA application program interface service in the cluster again. JFLink uses the database stored The relationship between the data version and the API. When adding a new version, insert the corresponding version number and the API address corresponding to the version into the database. API), users can view all Flink versions stored in the current database when they choose to start the cluster interface. After the user selects the required version, the correct Flink version cluster can be launched according to the corresponding API. Through such version switching, the user can Migration of jobs can also be done easily.

Using the above deployment mode can make the operation convenient and flexible, allowing users to choose the configuration by themselves to maximize resource utilization, and further, it is also very helpful for cluster migration. After giving an arbitrary K8S cluster, it can be Quickly build a Flink environment.

Optionally, before analyzing the target data packet based on the first interface to obtain the target computing resource, the method further includes: preprocessing the target data packet to obtain a preprocessing result, wherein the preprocessing result is used to represent the target data. Whether the package can run normally in the target environment; if the preprocessing result is that the target data package can run normally on the target resource, analyze the target data package based on the first interface to obtain the target computing resource.

The above-mentioned preprocessing can be a one-step operation on the target data packet before analyzing the target data packet based on the first interface. Through preprocessing, a preprocessing result can be obtained, and the result can be used to indicate whether the target data packet is damaged or not. The environment is blocked and cannot be run. Optionally, when the target data packet is normally available, the target data packet may be analyzed, thereby obtaining the target computing resource.

In an optional embodiment, after acquiring the target data package and the configuration parameters corresponding to the target data package, the job management platform may receive a data processing request sent by the user, and the request may be sent to the API background program, In response to this data processing request, the target data packet can be sent to the Flink client for preprocessing operations. Optionally, this operation can analyze the target data packet of the user, thereby obtaining the maximum number of TaskManagers required for the operation of the target data packet, and at the same time, it can also expose in advance the problem that the target data packet cannot be run due to damage to the target data packet and the environment is blocked. Finally, the target data packets that can run normally and are obtained after analysis will be written into TaskManager and JobManager by the API background program, and these orchestration files will be sent to the cluster brain (that is, the manager in the cluster, who does not participate in the execution, also known as K8Smaster) request, you can pull up a Flink on K8S cluster that satisfies the user's job operation.

In another optional embodiment, after the user submits the job, the user requests to render the parameters into the K8S orchestration file template through the API, and start the script template to pull up the cluster. When the cluster is started, the management address of the JobManager is mapped to the domain name + job ID through the ingress configuration file, and the management address can be exposed. After the configuration is started, the JobManager interface of the cluster can allow users to enter without being isolated by the virtual IP environment of K8S. When the JFlink platform obtains the management address of the JobManager, it can upload the user's data packets to the cluster through the interface. , and then use the interface to start the packet.

Figure 7 is a schematic diagram of the overall framework according to one embodiment of the present invention. As shown in the figure, the job management platform can start and stop the Flink cluster, start and stop jobs, modify the cluster/job configuration, save points regularly, and check the job status. The interface part can perform packet analysis, configure the cluster, and start and stop the cluster operation. The K8S layer can perform rendering configuration, routing domain name, resource scheduling, high availability guarantee, and system log operations. On the Flink cluster, you can start and stop jobs, ensure high availability, PerJob mode, data processing, and read and write database operations. Among them, what is done on the K8S layer and the Flink cluster can be called logging. When calling data in the database, you can call ES data, My SQL data, TiDB data, RocksDB data, and Kafka data, notifications. In the interface part, K8S layer and Flink cluster and database can monitor and alarm operations. Users can also control permissions on the job management platform, interface, K8S layer, Flink cluster, and database. The operating environment for all the above information can be a K8S server or an independent server.

From the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, the technical solutions of the present invention can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products are stored in a storage medium (such as ROM/RAM, magnetic disk, CD-ROM), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

In this embodiment, a data processing apparatus is also provided, and the apparatus is used to implement the above-mentioned embodiments and preferred implementations, and what has been described will not be repeated. As used below, the terms "unit", "module" may be a combination of software and/or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementations in hardware, or a combination of software and hardware, are also possible and contemplated.

FIG. 8 is a structural block diagram of a data processing apparatus according to an embodiment of the present invention. As shown in FIG. 8 , the apparatus includes:

The obtaining module 802 is configured to obtain a target data package and a configuration parameter corresponding to the target data package, wherein the target data package is a data package to be run, and the configuration parameter is used to determine a resource to be used for running the target data package.

The replacement module 804 is configured to replace the target placeholder in the preset arrangement file based on the configuration parameter to obtain the target arrangement file, wherein the field where the target placeholder is located is the field to be adjusted.

The calling module 806 is configured to determine a target resource space based on the target arrangement file, wherein the target resource space is used to provide an independent running resource for the target data package.

The processing module 808 is used for running the target data package based on the target resource space to obtain the running result.

Optionally, the obtaining module 802 includes: a calling unit, configured to call a first interface to analyze the target data packet, and obtain the target computing resource, wherein the target computing resource is used to represent the resource required for computing the target data packet; the first a determining unit, configured to determine a first parameter based on the target computing resource, where the first parameter is the number of first components required to calculate the target computing resource in the target resource space; the first obtaining unit is configured to obtain the second parameter, Wherein, the second parameter is the number of second components used for managing the target resource space; the second determining unit is configured to determine the configuration parameter based on the first parameter and the second parameter.

Optionally, the device further includes: a second obtaining module, configured to obtain a target keyword of the initial arrangement template, wherein the target keyword is a field to be adjusted in the initial arrangement template; adding the target keyword at a preset position of the target keyword Underline, get the target placeholder; adjust the initial arrangement template based on the target placeholder to get the preset arrangement file.

Optionally, the processing module 808 includes: a collection unit, used for collecting log files of the target resource space based on a preset resource space and a log directory; a monitoring unit, used for the process of running the target data package of the target resource space based on the log file Monitoring is performed to obtain a monitoring result, wherein the monitoring result is used to indicate whether an abnormality occurs in the process of running the target data package in the target resource space.

Optionally, the monitoring unit is further configured to stop monitoring the processing process of the target resource space based on the log file in response to the target instruction for the preset resource space, wherein the target instruction is at a preset time after the target resource space stops running. segment generation.

Optionally, the obtaining module 802 further includes: a second obtaining unit, configured to obtain a target data packet in response to a touch instruction to the first control in the first interface in the target platform, wherein the first interface is used to obtain a target data packet from at least one A target data packet is determined in one data packet; the uploading unit is used for uploading the configuration information corresponding to the target data packet to the target platform in response to the touch command on the second control of the second interface in the target platform, wherein the second The interface is used to edit the configuration information corresponding to the target data package.

Optionally, the uploading unit includes: an acquiring subunit, used for acquiring first version information corresponding to the target resource space; a determining subunit, used for determining a target image based on the first version information, wherein the target image is used to provide a running target The environment of the resource space; the operation subunit is used to run the target data package based on the target image and the target resource space, and obtain the operation result.

Optionally, the obtaining subunit is further configured to obtain the first version information corresponding to the target resource space in response to the touch command on the third control in the third interface in the target platform, wherein the third interface is used for controlling the target resource. Edit the first version information corresponding to the space.

Optionally, the obtaining subunit is further configured to obtain a second interface in response to the upgrade instruction for the target resource space, wherein the second interface is associated with the second version information, and the second version information is corresponding to the upgraded target resource space. Version information; add the second interface to the third interface.

Optionally, the calling module 806 includes: a preprocessing unit, configured to preprocess the target data packet to obtain a preprocessing result, wherein the preprocessing result is used to indicate whether the target data packet can normally run in the target environment; The analyzing unit is configured to analyze the target data packet based on the first interface to obtain the target computing resource when the preprocessing result is that the target data packet can normally run on the target resource.

It should be noted that the above-mentioned units and modules can be implemented by software or hardware, and the latter can be implemented in the following ways, but not limited to this: the above-mentioned units and modules are all located in the same processor; Units and modules are located in different processors in any combination.

An embodiment of the present invention also provides a computer-readable storage medium, where a computer program is stored in the non-volatile storage medium, wherein the computer program is configured to execute the steps in any of the above method embodiments when running .

Optionally, in this embodiment, the above-mentioned computer-readable storage medium may be configured to store a computer program for executing the following steps:

S1: Acquire a target data package and configuration parameters corresponding to the target data package, where the target data package is a data package to be run, and the configuration parameters are used to determine resources to be used for running the target data package.

S2: Replacing the target placeholder in the preset arrangement file based on the configuration parameters to obtain the target arrangement file, wherein the field where the target placeholder is located is the field to be adjusted.

S3: Determine a target resource space based on the target arrangement file, where the target resource space is used to provide an independent running resource for the target data package.

S4, run the target data package based on the target resource space to obtain the running result.

Optionally, in this embodiment, the above-mentioned computer-readable storage medium may include, but is not limited to, a USB flash drive, a read-only memory (Read-Only Memory, referred to as ROM), and a random access memory (Random Access Memory, referred to as abbreviated as ROM). Various media that can store computer programs, such as RAM), removable hard disks, magnetic disks or optical disks.

Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

Optionally, the above-mentioned computer-readable storage medium is further configured to store program codes for performing the following steps:

Optionally, acquiring configuration parameters corresponding to the target data packet includes: calling a first interface to analyze the target data packet, and obtaining target computing resources, where the target computing resources are used to represent resources required for computing the target data packet; The computing resource determines a first parameter, where the first parameter is the number of first components required to calculate the target computing resource in the target resource space; acquires a second parameter, where the second parameter is the first parameter used to manage the target resource space The number of two components; based on the first parameter and the second parameter, the configuration parameter is determined.

Optionally, the method further includes: acquiring a target keyword of an initial arrangement template, where the target keyword is a field to be adjusted in the initial arrangement template; adding a target underline to a preset position of the target keyword to obtain a target placeholder ; Adjust the initial arrangement template based on the target placeholder to obtain a preset arrangement file.

Optionally, the target resource space and the preset resource space share a log directory, and in the process of using the target resource space to run the target data package, the method further includes: collecting log files of the target resource space based on the preset resource space and the log directory; The log file monitors the process of running the target data package in the target resource space, and obtains a monitoring result, wherein the monitoring result is used to indicate whether an abnormality occurs in the process of running the target data package in the target resource space.

Optionally, the method further includes: in response to the target instruction for the preset resource space, stopping monitoring the processing process of the target resource space based on the log file, wherein the target instruction is in a preset time period after the target resource space stops running. generate.

Optionally, acquiring the target data package and the configuration parameters corresponding to the target data package includes: in response to a touch instruction to the first control in the first interface in the target platform, acquiring the target data package, wherein the first interface is used to retrieve the target data package. Determining a target data packet in at least one data packet; in response to a touch command on a second control of a second interface in the target platform, uploading configuration information corresponding to the target data packet to the target platform, wherein the second interface is used to monitor the target data Edit the configuration information corresponding to the package.

Optionally, using the target resource space to process the target data packet to obtain a processing result, including: acquiring first version information corresponding to the target resource space; determining a target image based on the first version information, wherein the target image is used to provide a running target The environment of the resource space; run the target data package based on the target image and the target resource space, and get the operation result.

Optionally, acquiring the first version information corresponding to the target resource space includes: in response to a touch instruction to a third control in the third interface in the target platform, acquiring the first version information corresponding to the target resource space, wherein the third The interface is used to edit the first version information corresponding to the target resource space.

Optionally, the method further includes: in response to an upgrade instruction for the target resource space, acquiring a second interface, where the second interface is associated with second version information, and the second version information is version information corresponding to the upgraded target resource space. ;Add the second interface to the third interface.

Optionally, before analyzing the target data packet based on the first interface to obtain the target computing resource, the method further includes: preprocessing the target data packet to obtain a preprocessing result, wherein the preprocessing result is used to represent the target data. Whether the package can run normally in the target environment; if the preprocessing result is that the target data package can run normally on the target resource, analyze the target data package based on the first interface to obtain the target computing resource.

In the computer-readable medium of this embodiment, the target data packet and the configuration parameter corresponding to the target data packet are obtained first, wherein the target data packet is the data packet to be run, and the configuration parameter is used to determine the target data packet to be used for running. resource; then replace the target placeholder in the preset arrangement file based on the configuration parameters to obtain the target arrangement file, in which the field where the target placeholder is located is the field to be adjusted; then determine based on the target arrangement file The target resource space, wherein the target resource space is used to provide an independent operating resource for the target data package; finally, the target data package is run based on the target resource space to obtain the operating result. It is easy to think that, after replacing the target placeholder in the preset arrangement file based on the configuration parameters to obtain the target arrangement file, the target resource space can be determined based on the target arrangement file, thereby further realizing the processing of data. Efficiency and accuracy, thereby solving the technical problem of low data runtime efficiency in the relevant technical resource space.

From the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described herein may be implemented by software, or may be implemented by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, and the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, U disk, mobile hard disk, etc.) or on a network, Several instructions are included to cause a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to perform a method according to an embodiment of the present invention.

In an exemplary embodiment of the present application, a computer-readable storage medium stores a program product capable of implementing the above method of this embodiment. In some possible implementations, various aspects of the embodiments of the present invention can also be implemented in the form of a program product, which includes program codes, and when the program product runs on a terminal device, the program codes are used to make The terminal device performs the steps according to various exemplary embodiments of the present invention described in the above-mentioned "Exemplary Method" section of this embodiment.

A program product for implementing the above method according to an embodiment of the present invention may adopt a portable compact disc read only memory (CD-ROM) and include program codes, and may run on a terminal device, such as a personal computer. However, the program product of the embodiment of the present invention is not limited thereto. In the embodiment of the present invention, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or associated with an instruction execution system, apparatus, or device. In conjunction with.

The above-described program product may employ any combination of one or more computer-readable media. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read only memory (ROM), Erase programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

It should be noted that the program code contained on the computer-readable storage medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the above.

An embodiment of the present invention also provides an electronic device, comprising a memory and a processor, where a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above method embodiments.

Optionally, the above-mentioned electronic device may further include a transmission device and an input-output device, wherein the transmission device is connected to the above-mentioned processor, and the input-output device is connected to the above-mentioned processor.

Optionally, in this embodiment, the above-mentioned processor may be configured to execute the following steps through a computer program:

S1: Acquire a target data package and configuration parameters corresponding to the target data package, where the target data package is a data package to be run, and the configuration parameters are used to determine resources to be used for running the target data package.

S2: Replacing the target placeholder in the preset arrangement file based on the configuration parameters to obtain the target arrangement file, wherein the field where the target placeholder is located is the field to be adjusted.

S3: Determine a target resource space based on the target arrangement file, where the target resource space is used to provide an independent running resource for the target data package.

S4, run the target data package based on the target resource space to obtain the running result.

Optionally, the above-mentioned processor may also be configured to perform the following steps through a computer program: invoking the first interface to analyze the target data packet to obtain the target computing resource, wherein the target computing resource is used to represent the required data for computing the target data packet. resource; determine a first parameter based on the target computing resource, where the first parameter is the number of first components required to calculate the target computing resource in the target resource space; acquire a second parameter, where the second parameter is used to manage the target The number of second components of the resource space; based on the first parameter and the second parameter, the configuration parameter is determined.

Optionally, the above-mentioned processor may also be configured to perform the following steps through a computer program: obtaining a target keyword of an initial layout template, where the target keyword is a field to be adjusted in the initial layout template; presetting the target keyword Add the target underline to the position to obtain the target placeholder; adjust the initial arrangement template based on the target placeholder to obtain the preset arrangement file.

Optionally, the above-mentioned processor can also be configured to execute the following steps through a computer program: collect the log file of the target resource space based on the preset resource space and the log directory; perform the process of running the target data package in the target resource space based on the log file Monitoring is performed to obtain a monitoring result, wherein the monitoring result is used to indicate whether an abnormality occurs in the process of running the target data package in the target resource space.

Optionally, the above-mentioned processor may also be configured to perform the following steps through a computer program: in response to the target instruction for the preset resource space, stop monitoring the processing process of the target resource space based on the log file, wherein the target instruction is in the target resource space. Generated at a preset time period after the resource space stops running.

Optionally, the above-mentioned processor may also be configured to perform the following steps through a computer program: in response to a touch instruction on the first control in the first interface in the target platform, acquire the target data packet, wherein the first interface is used to obtain the target data package from the first interface. Determining a target data packet in at least one data packet; in response to a touch command on a second control of a second interface in the target platform, uploading configuration information corresponding to the target data packet to the target platform, wherein the second interface is used to monitor the target data Edit the configuration information corresponding to the package.

Optionally, the above-mentioned processor may also be configured to perform the following steps through a computer program: acquiring first version information corresponding to the target resource space; determining a target image based on the first version information, wherein the target image is used to provide a running target resource space environment; run the target data package based on the target image and target resource space, and get the running result.

Optionally, the above-mentioned processor may also be configured to perform the following steps through a computer program: in response to a touch instruction to the third control in the third interface in the target platform, obtain the first version information corresponding to the target resource space, wherein, The third interface is used to edit the first version information corresponding to the target resource space.

Optionally, the above-mentioned processor may also be configured to perform the following steps through a computer program: in response to an instruction to upgrade the target resource space, obtain a second interface, where the second interface is associated with second version information, and the second version information is Version information corresponding to the upgraded target resource space; add the second interface to the third interface.

Optionally, the above-mentioned processor may also be configured to perform the following steps through a computer program: preprocessing the target data packet to obtain a preprocessing result, wherein the preprocessing result is used to indicate whether the target data packet can run normally in the target environment. In the case where the preprocessing result is that the target data packet can normally run on the target resource, analyze the target data packet based on the first interface to obtain the target computing resource.

In the electronic device of this embodiment, the target data packet and the configuration parameters corresponding to the target data packet are obtained first, wherein the target data packet is the data packet to be run, and the configuration parameter is used to determine the resource to be used for running the target data packet; Then, based on the configuration parameters, the target placeholder in the preset arrangement file is replaced to obtain the target arrangement file, wherein the field where the target placeholder is located is the field to be adjusted; then the target resource is determined based on the target arrangement file space, wherein the target resource space is used to provide an independent operating resource for the target data package; finally, the target data package is run based on the target resource space to obtain the operating result. It is easy to think that, after replacing the target placeholder in the preset arrangement file based on the configuration parameters to obtain the target arrangement file, the target resource space can be determined based on the target arrangement file, thereby further realizing the processing of data. Efficiency and accuracy, thereby solving the technical problem of low data runtime efficiency in the relevant technical resource space.

FIG. 9 is a schematic diagram of an electronic device according to an embodiment of the present invention. As shown in FIG. 9 , the electronic device 900 is only an example, and should not impose any limitation on the function and scope of use of the embodiments of the present invention.

As shown in FIG. 9, the electronic device 900 is embodied in the form of a general-purpose computing device. Components of the electronic device 900 may include, but are not limited to, the above-mentioned at least one processor 910 , the above-mentioned at least one memory 920 , a bus 930 connecting different system components (including the memory 920 and the processor 910 ), and a display 940 .

The above-mentioned memory 920 stores program codes, which can be executed by the processor 910, so that the processor 910 executes the steps according to various exemplary embodiments of the present invention described in the above-mentioned method part of the embodiments of the present application.

The memory 920 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 9201 and/or a cache storage unit 9202, may further include a read only storage unit (ROM) 9203, and may also include Non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid state memory.

In some instances, memory 920 may also include a program/utility 9204 having a set (at least one) of program modules 9205, such program modules 13205 including, but not limited to: an operating system, one or more application programs, other program modules As well as program data, each or some combination of these examples may include an implementation of a network environment. The memory 920 may further include memories disposed remotely with respect to the processor 910, and these remote memories may be connected to the electronic device 900 through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The bus 930 may represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, the processor 910, or a bureau using any of a variety of bus structures. domain bus.

Display 940 may be, for example, a touch screen type liquid crystal display (LCD) that may enable a user to interact with a user interface of electronic device 900 .

Optionally, the electronic device 900 can also communicate with one or more external devices 900 (such as keyboards, pointing devices, Bluetooth devices, etc.), and can also communicate with one or more devices that enable the user to interact with the electronic device 900, and/or with any device (eg, router, modem, etc.) that enables the electronic device 900 to communicate with one or more other computing devices. Such communication may take place through input/output (I/O) interface 950 . Also, the electronic device 900 may communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN), and/or a public network such as the Internet) through the network adapter 960 . As shown in FIG. 9 , the network adapter 960 communicates with other modules of the electronic device 900 through the bus 930 . It should be appreciated that, although not shown in FIG. 9, other hardware and/or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, Tape drives and data backup storage systems, etc.

The above electronic device 900 may further include: a keyboard, a cursor control device (eg, a mouse), an input/output interface (I/O interface), a network interface, a power supply, and/or a camera.

Those of ordinary skill in the art can understand that the structure shown in FIG. 9 is only a schematic diagram, which does not limit the structure of the above-mentioned electronic device. For example, the electronic device 900 may also include more or fewer components than those shown in FIG. 9 , or have a different configuration than that shown in FIG. 1 . The memory 920 may be used to store computer programs and corresponding data, such as computer programs and corresponding data corresponding to the data processing method in the embodiment of the present invention. The processor 910 executes various functional applications and data processing by running the computer program stored in the memory 920, ie, realizes the above-mentioned data processing method.

The above-mentioned serial numbers of the embodiments of the present invention are only for description, and do not represent the advantages or disadvantages of the embodiments.

In the above-mentioned embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are only illustrative, for example, the division of the units may be a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components may be combined or Integration into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of units or modules, and may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes .

The above are only the preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, without departing from the principles of the present invention, several improvements and modifications can be made. It should be regarded as the protection scope of the present invention.

Claims (13)

1. A method of data processing, the method comprising:

acquiring a target data packet and configuration parameters corresponding to the target data packet, wherein the target data packet is a data packet to be operated, and the configuration parameters are used for determining resources to be used for operating the target data packet;

replacing a target placeholder in a preset layout file based on the configuration parameters to obtain a target layout file, wherein a field at the position of the target placeholder is a field to be adjusted;

determining a target resource space based on the target arrangement file, wherein the target resource space is used for providing an independent running resource for the target data packet;

and operating the target data packet based on the target resource space to obtain an operation result.

2. The method of claim 1, wherein obtaining the configuration parameters corresponding to the target packet comprises:

calling a first interface to analyze the target data packet to obtain a target operation resource, wherein the target operation resource is used for expressing a resource required by operation of the target data packet;

determining a first parameter based on the target computational resource, wherein the first parameter is a number of first components required to compute the target computational resource in the target resource space;

acquiring a second parameter, wherein the second parameter is the number of second components for managing the target resource space;

determining the configuration parameter based on the first parameter and the second parameter.

3. The method of claim 1, further comprising:

acquiring a target keyword of an initial arranging template, wherein the target keyword is a field to be adjusted in the initial arranging template;

adding a target underline at a preset position of the target keyword to obtain the target placeholder;

and adjusting the initial layout template based on the target placeholder to obtain the preset layout file.

4. The method of claim 1, wherein the target resource space shares a log directory with a preset resource space, and during the running of the target data packet by using the target resource space, the method further comprises:

collecting a log file of the target resource space based on the preset resource space and the log directory;

and monitoring the process of operating the target data packet in the target resource space based on the log file to obtain a monitoring result, wherein the monitoring result is used for indicating whether the process of operating the target data packet in the target resource space is abnormal or not.

5. The method of claim 4, further comprising:

and responding to a target instruction of the preset resource space, and stopping monitoring the processing process of the target resource space based on the log file, wherein the target instruction is generated in a preset time period after the target resource space stops running.

6. The method of claim 1, wherein obtaining a target data packet and configuration parameters corresponding to the target data packet comprises:

responding to a touch instruction of a first control in a first interface in a target platform, and acquiring the target data packet, wherein the first interface is used for determining the target data packet from at least one data packet;

and responding to a touch instruction of a second control of a second interface in the target platform, and uploading configuration information corresponding to the target data packet to the target platform, wherein the second interface is used for editing the configuration information corresponding to the target data packet.

7. The method of claim 6, wherein processing the target data packet using the target resource space to obtain a processing result comprises:

acquiring first version information corresponding to the target resource space;

determining a target image based on the first version information, wherein the target image is used for providing an environment for running the target resource space;

and operating the target data packet based on the target mirror image and the target resource space to obtain the operation result.

8. The method of claim 7, wherein obtaining the first version information corresponding to the target resource space comprises:

and responding to a touch instruction of a third control in a third interface in the target platform, and acquiring first version information corresponding to the target resource space, wherein the third interface is used for editing the first version information corresponding to the target resource space.

9. The method of claim 8, further comprising:

responding to the target resource space upgrading instruction, and acquiring a second interface, wherein the second interface is associated with second version information, and the second version information is version information corresponding to the upgraded target resource space;

adding the second interface to the third interface.

10. The method of claim 1, wherein before analyzing the target packet based on the first interface to obtain the target computational resource, the method further comprises:

preprocessing the target data packet to obtain a preprocessing result, wherein the preprocessing result is used for indicating whether the target data packet can normally run in a target environment;

and analyzing the target data packet based on the first interface to obtain the target operation resource under the condition that the preprocessing result is that the target data packet can normally run in the target resource.

11. A data processing apparatus, characterized in that the apparatus comprises:

the device comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a target data packet and configuration parameters corresponding to the target data packet, the target data packet is a data packet to be operated, and the configuration parameters are used for determining resources to be used for operating the target data packet;

the replacing module is used for replacing the target placeholder in the preset layout file based on the configuration parameters to obtain a target layout file, wherein the field of the position of the target placeholder is a field to be adjusted;

the calling module is used for determining a target resource space based on the target layout file, wherein the target resource space is used for providing an independent running resource for the target data packet;

and the processing module is used for operating the target data packet based on the target resource space to obtain an operation result.

12. A computer-readable storage medium, in which a computer program is stored, wherein the computer program is arranged to, when executed by a processor, perform a data processing method as claimed in any one of claims 1 to 10.

13. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and the processor is arranged to execute the computer program to perform the data processing method of any of claims 1 to 10.

Images