CN115129542A

CN115129542A - Data processing method, data processing device, storage medium and electronic device

Info

Publication number: CN115129542A
Application number: CN202210699966.3A
Authority: CN
Inventors: 钟锦豪
Original assignee: Netease Hangzhou Network Co Ltd
Current assignee: Netease Hangzhou Network Co Ltd
Priority date: 2022-06-20
Filing date: 2022-06-20
Publication date: 2022-09-30

Abstract

The invention discloses a data processing method, a data processing device, a storage medium and an electronic device. The method comprises the following steps: acquiring a target data packet and configuration parameters corresponding to the target data packet, wherein the target data packet is a data packet to be operated, and the configuration parameters are used for determining resources to be used for operating the target data packet; replacing the target placeholder in the preset layout file based on the configuration parameters to obtain a target layout file, wherein the field of the position of the target placeholder is a field to be adjusted; determining a target resource space based on the target layout file, wherein the target resource space is used for providing an independent running resource for the target data packet; and operating the target data packet based on the target resource space to obtain an operation result. The invention solves the technical problem of lower efficiency of data operation in the related technology resource space.

Description

Data processing method, data processing device, storage medium and electronic device

Technical Field

The invention relates to the field of big data, in particular to a data processing method, a data processing device, a storage medium and an electronic device.

Background

The workflow of charging big data can be simply described as that log related to login, payment and the like of each game is collected to a high-throughput distributed publish-subscribe message system for storage through a lightweight transmission program for forwarding and concentrating log data, and such log is called as a service original log. At present, in the process of performing data operation on such an original log, all jobs are entrusted to a system administrator for management, and the whole operation environment is also operated based on only one application of a resource manager, so if a situation that a service is unavailable occurs in a resource manager master node or the system administrator, all jobs of a whole cluster are affected, and normal operation is difficult.

In view of the above problems, no effective solution has been proposed.

Disclosure of Invention

At least some embodiments of the present invention provide a data processing method, an apparatus, a storage medium, and an electronic apparatus, so as to at least solve the technical problem of low efficiency when data in a related technology resource space is executed.

According to an embodiment of the present invention, there is provided a data processing method including: acquiring a target data packet and configuration parameters corresponding to the target data packet, wherein the target data packet is a data packet to be operated, and the configuration parameters are used for determining resources to be used for operating the target data packet; replacing the target placeholder in the preset layout file based on the configuration parameters to obtain a target layout file, wherein the field of the position of the target placeholder is a field to be adjusted; determining a target resource space based on the target layout file, wherein the target resource space is used for providing an independent running resource for the target data packet; and operating the target data packet based on the target resource space to obtain an operation result.

Optionally, the obtaining of the configuration parameter corresponding to the target data packet includes: calling a first interface to analyze a target data packet to obtain a target operation resource, wherein the target operation resource is used for expressing a resource required by operation of the target data packet; determining a first parameter based on the target computational resource, wherein the first parameter is the number of first components required for computing the target computational resource in the target resource space; acquiring a second parameter, wherein the second parameter is the number of second components for managing the target resource space; based on the first parameter and the second parameter, a configuration parameter is determined.

Optionally, the method further comprises: acquiring target keywords of the initial arrangement template, wherein the target keywords are fields to be adjusted in the initial arrangement template; adding a target underline at a preset position of the target keyword to obtain a target placeholder; and adjusting the initial layout template based on the target placeholder to obtain a preset layout file.

Optionally, the target resource space and the preset resource space share a log directory, and in the process of operating the target data packet by using the target resource space, the method further includes: collecting a log file of a target resource space based on a preset resource space and a log directory; and monitoring the process of operating the target data packet in the target resource space based on the log file to obtain a monitoring result, wherein the monitoring result is used for indicating whether the process of operating the target data packet in the target resource space is abnormal or not.

Optionally, the method further comprises: and responding to a target instruction of the preset resource space, and stopping monitoring the processing process of the target resource space based on the log file, wherein the target instruction is generated within a preset time period after the target resource space stops running.

Optionally, the obtaining the target data packet and the configuration parameter corresponding to the target data packet includes: responding to a touch instruction of a first control in a first interface in a target platform, and acquiring a target data packet, wherein the first interface is used for determining the target data packet from at least one data packet; and responding to a touch instruction of a second control of a second interface in the target platform, and uploading configuration information corresponding to the target data packet to the target platform, wherein the second interface is used for editing the configuration information corresponding to the target data packet.

Optionally, processing the target data packet by using the target resource space to obtain a processing result, including: acquiring first version information corresponding to a target resource space; determining a target image based on the first version information, wherein the target image is used for providing an environment for running a target resource space; and operating the target data packet based on the target mirror image and the target resource space to obtain an operation result.

Optionally, the obtaining of the first version information corresponding to the target resource space includes: and responding to a touch instruction of a third control in a third interface in the target platform, and acquiring first version information corresponding to the target resource space, wherein the third interface is used for editing the first version information corresponding to the target resource space.

Optionally, the method further comprises: responding to an upgrading instruction of the target resource space, and acquiring a second interface, wherein the second interface is associated with second version information, and the second version information is version information corresponding to the upgraded target resource space; the second interface is added to the third interface.

Optionally, before analyzing the target data packet based on the first interface to obtain the target computing resource, the method further includes: preprocessing a target data packet to obtain a preprocessing result, wherein the preprocessing result is used for indicating whether the target data packet can normally run in a target environment; and analyzing the target data packet based on the first interface to obtain the target operation resource under the condition that the target data packet can normally run in the target resource as a preprocessing result.

According to an embodiment of the present invention, there is also provided a data processing apparatus including: the device comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a target data packet and configuration parameters corresponding to the target data packet, the target data packet is a data packet to be operated, and the configuration parameters are used for determining resources to be used for operating the target data packet; the replacing module is used for replacing the target placeholder in the preset layout file based on the configuration parameters to obtain a target layout file, wherein the field of the position of the target placeholder is a field to be adjusted; the scheduling module is used for determining a target resource space based on the target scheduling file, wherein the target resource space is used for providing an independent running resource for the target data packet; and the processing module is used for operating the target data packet based on the target resource space to obtain an operation result.

According to an embodiment of the present invention, there is further provided a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to be executed by a processor to perform the data processing method described in any one of the above.

There is further provided, according to an embodiment of the present invention, an electronic apparatus including a memory and a processor, the memory storing a computer program therein, and the processor being configured to execute the computer program to perform the data processing method in any one of the above.

In at least some embodiments of the present invention, a target data packet and configuration parameters corresponding to the target data packet are obtained first, where the target data packet is a data packet to be operated, and the configuration parameters are used to determine resources to be used for operating the target data packet; then, replacing the target placeholder in the preset layout file based on the configuration parameters to obtain a target layout file, wherein the field of the position of the target placeholder is a field to be adjusted; determining a target resource space based on the target layout file, wherein the target resource space is used for providing an independent running resource for the target data packet; and finally, operating the target data packet based on the target resource space to obtain an operation result, thereby improving the operation efficiency of the target data packet. It is easy to notice that the target arrangement file corresponding to the target data packet can be quickly determined according to the configuration parameters and the preset arrangement file, and the target resource space corresponding to the target data packet is determined based on the target arrangement file, so that the operation process of the target data packet is isolated from the operation processes of other data packets through the target resource space, an independent operation resource is provided for the target data packet, the target data packet can be prevented from being influenced by other data packets, the operation efficiency of the target data packet can be improved, and the problem of low operation efficiency of the target data packet in the related technology can be solved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention and do not constitute a limitation of the invention. In the drawings:

fig. 1 is a block diagram of a hardware configuration of a mobile terminal of a data processing method according to an embodiment of the present invention;

FIG. 2 is a flow diagram of a data processing method according to one embodiment of the invention;

FIG. 3 is a schematic illustration of an operational state display according to one embodiment of the present invention;

FIG. 4 is a schematic diagram of a user submission interface, in accordance with one embodiment of the present invention;

FIG. 5 is a schematic diagram of an edit job interface display according to one embodiment of the present invention;

FIG. 6 is a schematic diagram of a display of a Job initiation interface according to one embodiment of the present invention;

FIG. 7 is a general block diagram according to one embodiment of the invention;

FIG. 8 is a block diagram of a data processing apparatus according to one embodiment of the present invention;

fig. 9 is a schematic diagram of an electronic device according to an embodiment of the invention.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Where a data processing method is provided according to one embodiment of the present invention, it is noted that the steps illustrated in the flow charts of the accompanying figures may be performed in a computer system such as a set of computer-executable instructions, and that while logical sequences are illustrated in the flow charts, in some cases, the steps shown or described may be performed in an order different than here.

The method embodiments may be performed in a mobile terminal, a computer terminal or a similar computing device. Taking the Mobile terminal as an example, the Mobile terminal may be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet device (MID for short), a PAD, a game console, and the like. Fig. 1 is a block diagram of a hardware structure of a mobile terminal of a data processing method according to an embodiment of the present invention. As shown in fig. 1, the mobile terminal may include one or more (only one shown in fig. 1) processors 102 (the processors 102 may include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Digital Signal Processing (DSP) chip, a Microprocessor (MCU), a programmable logic device (FPGA), a neural Network Processor (NPU), a Tensor Processor (TPU), an Artificial Intelligence (AI) type processor, etc.) and a memory 104 for storing data. Optionally, the mobile terminal may further include a transmission device 106, an input/output device 108, and a display device 110 for communication functions. It will be understood by those skilled in the art that the structure shown in fig. 1 is only an illustration, and does not limit the structure of the mobile terminal. For example, the mobile terminal may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.

The memory 104 can be used for storing computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the data processing method in the embodiment of the present invention, and the processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, implementing the data processing method described above. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory located remotely from the processor 102, which may be connected to the mobile terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The transmission device 106 is used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.

The inputs in the input output Device 108 may come from a plurality of Human Interface Devices (HIDs). For example: keyboard and mouse, game pad, other special game controller (such as steering wheel, fishing rod, dance mat, remote controller, etc.). Some human interface devices may provide output functions in addition to input functions, such as: force feedback and vibration of the gamepad, audio output of the controller, etc.

The display device 110 may be, for example, a head-up display (HUD), a touch screen type Liquid Crystal Display (LCD), and a touch display (also referred to as a "touch screen" or "touch display screen"). The liquid crystal display may enable a user to interact with a user interface of the mobile terminal. In some embodiments, the mobile terminal has a Graphical User Interface (GUI) with which a user can interact by touching finger contacts and/or gestures on a touch-sensitive surface, where the human interaction functionality optionally includes the following interactions: executable instructions for creating web pages, drawing, word processing, making electronic documents, games, video conferencing, instant messaging, emailing, call interfacing, playing digital video, playing digital music, and/or web browsing, etc., for performing the above-described human-computer interaction functions, are configured/stored in one or more processor-executable computer program products or readable storage media.

First, technical terms or technical terms appearing in the embodiments of the present invention are explained as follows:

flink: also known as Apache Flink, is a distributed processing engine and framework that performs stateful computations on bounded and unbounded data streams.

JobManager: the manager of the Flink system is responsible for receiving and processing the Flink job request, generating an execution plan for the job, scheduling the TaskManager to serve the job operation, and managing the status of the recorded job, which may be referred to as JM for short.

The executor of the TaskManager Flink system is the main component responsible for executing computing tasks, called TM for short, and is essentially JVM, i.e. Java Virtual Machine, which can also be called Java Virtual Machine.

CheckPoint: the fault-tolerant recovery mechanism of the Flink can define the execution frequency by a user, record the current running state for the operation at regular intervals and store the current running state in a database, and when the operation is failed and restarted, read the current stored state from the database and recover the operation to the current state

SavePoint: the Flink job recovery mechanism is a consistent mirror image of a stream job execution state created according to the Flink CheckPoint mechanism, and needs manual execution, creation, and deletion by a user, wherein the Flink CheckPoint is used for restarting the job by unexpected failure, and the SavePoint is generally used for restarting the job by planning.

FileBeat: the lightweight transmission program for forwarding and centralizing log data is mainly used for log collection and transmission to Kafka for storage in departments.

Kafka: the distributed publish-subscribe messaging system with high throughput can be used for storing original log information and providing the original log information for Flink consumption.

ZooKeeper: the distributed configuration service, the synchronization service and the naming registration of the open source can be provided for large-scale distributed computing, and the meta information of the operation of the job and the meta data required by the failure recovery of the job, referred to as ZK for short, can be stored in the Flink.

rocksDB: the framework engine for providing key value storage and read-write functions can be used in Flink to store operation running states in the process of checkPoint and SavePoint.

Kubernets: is a portable and extensible open source platform for managing containerized workload and services, and can promote declarative configuration and automation, abbreviated as K8S, and is used as a main tool for Flink resource regulation.

PersistentVolumeClaim: the persistent volume expresses the user's request for storage, PVC for short.

Persistentvolume: and the persistent volume is a block of storage in the cluster, and is called PV for short.

Container: is a container, is a portable, executable, lightweight image, contains the software therein and its associated dependencies.

Image: for mirroring, it is a stored container instance that can package a set of software needed for the application to run.

HDFS (Hadoop distributed File System): the Hadoop distributed File System, also called Hadoop distributed File System, is responsible for storing binary files generated during SavePoint and CheckPoint processes in Flink.

JFlink: the platform is a platform facing to the Flink user and can be used for managing the Flink cluster operation start and stop, the operation life cycle control, the operation error check and the version iteration of the program operation package. The platform can maintain the operation version, check and adjust the operation state and the like.

Prometheus: the Promiex is a system monitoring and warning kit developed in the form of open source software, and the Flink has a complete monitoring acquisition program and a reporting process, so that great help is provided for the operation and maintenance of the Flink.

Monitoring data: the data of the running state of the component can be reflected in real time and is often composed of time, monitoring indexes and monitoring results.

Cluster configuration: the basic attribute of a Flink cluster is shown, which is a necessary basis for stable operation of the Flink job, and a configuration table is made by a user according to the resource requirement of the self job.

And (3) operation planning: the returned result obtained by the Plan method in the official Flink provides a topological diagram of the Flink operation, results of the operation analysis of each operator of the Flink and the like, and potential problems of the Flink operation in the starting stage can be exposed in advance.

Arranging files: deployment of resource management and resource object choreography in the K8S cluster is addressed by declaring a style file, called a resource manifest file, that is used herein to create the necessary components needed for Flink to run on K8S.

Fig. 2 is a flowchart of a data processing method according to an embodiment of the present invention, in which a terminal device provides a graphical user interface, and content displayed on the graphical user interface includes a touch area, as shown in fig. 2, the method includes the following steps:

step S202, a target data packet and configuration parameters corresponding to the target data packet are obtained, wherein the target data packet is a data packet to be operated, and the configuration parameters are used for determining resources to be used for operating the target data packet.

The target data packet can be uploaded on the operation management platform by a user, and after the target data packet is uploaded, the target data packet can be configured, so that configuration parameters are obtained. Optionally, the target data packet may include relevant logs of login, payment, etc. for the game. After the target data packet is obtained, the target data packet can be cleaned and filtered, error data and repeated data are deleted, format conversion is carried out to be in accordance with a format adapted to the operation management platform, aggregation operation is carried out to divide the target data packet into different groups according to data types, the different groups are finally stored in a database, and then the target data packet can be called from the database and put on the operation management platform to run when the target data packet is needed.

The configuration parameters may include the memory size of the job management platform, the print level of the logger, and the like. Alternatively, jobs such as JobManager, TaskManager, etc. may be configured, and the number of parallel slots supported by each TaskManager may be determined, etc.

And S204, replacing the target placeholder in the preset layout file based on the configuration parameters to obtain the target layout file, wherein the field of the position of the target placeholder is a field to be adjusted.

The preset layout file may be a template of a layout file reserved on the server in advance, and the configuration file on the template may be filled with a placeholder in the key field for subsequent modification by the user.

The target placeholder can be used for temporarily replacing the position of the field to be adjusted, and the field of the placeholder can be replaced by the filled-in parameter during subsequent use, so that the filled-in parameter can successfully take effect, and the target arrangement file is further generated.

Since the target placeholder needs to be frequently modified by the user and the running of the job is closely related to the field, the target placeholder can be represented in the following manner, and underlining can be added before and after the placeholder field for distinguishing. In an alternative embodiment, assuming { { __ NAMESPACE __ } } is a template for the layout file, the target layout file can be derived from the template.

For example, the document layout template is composed of front manager, memory, process, SIZE, __ HEAP _ SIZE _ TM _ M __, and the parameter prefix and the end are underlined and are target placeholders. The user inputs parameters corresponding to the target placeholder fields through an API (Application Programming Interface), such as: __ HEAP _ SIZE _ TM _ M __: 2GB, the API fills the parameters into the template of the layout file, the template is replaced by the effect of taskmanager. If the user does not provide parameters, the API will default to using 2GB of memory as the parameter entry.

Optionally, the configuration parameters in the template for arranging files may include controller arranging files of JobManager and TaskManager, mounted disk files, domain name routing entry files (also referred to as Image files), Flink cluster configuration files, and scripts for starting and stopping the cluster.

Optionally, the keyword to be modified is processed through a placeholder, so that a preset layout file with the placeholder can be obtained, and different data packets can be conveniently used.

Step S206, determining a target resource space based on the target layout file, wherein the target resource space is used for providing an independent operating resource for the target data packet.

The target layout file may be generated from a preset layout file, and optionally, the placeholder of the preset layout file may be modified to fill in parameters, so as to generate the target layout file. The target layout file is the key for correctly starting the Flink cluster.

The target resource space may be used to provide an independent running resource to isolate different packet runs so that when a job JobManager of one job has a service unavailability, other jobs are not affected, and optionally, the target resource space may be a Flink cluster.

In an alternative embodiment, the NameSpace concept of K8S (also called NameSpace concept) may be used to generate an independent NameSpace for each job of the user, ensuring that the JobManager and TaskManager under the NameSpace serve only one job, i.e. when an a job is submitted to the cluster, a resource space named N _ a is applied for the a job in K8S, and the number of task managers and JM number required to run the job are only, where JM may be a model, and the job monopolizes N _ a resources and has the same resource space as other jobs: n _ B and N _ C do not interfere with each other, so that the operation can be isolated from the operation, and the operation mode of Per Job is achieved, namely each operation has a unique JobManager. That is, the target resource space can provide an independent running resource, so that different data packets for simultaneous jobs can be separated, and even if the service unavailability occurs in the JobManager of one job, the running of other jobs will not be affected.

And step S208, operating the target data packet based on the target resource space to obtain an operation result.

In an alternative embodiment, the target data packet may be run after the independent run resource has been determined, i.e. after the target resource space, to obtain a run result. Fig. 3 is a schematic view of an operation status display according to an embodiment of the present invention, and as shown in fig. 3, when the user presses the start button, the interface displays the operation status in the form of a progress bar, wherein the progress bar may be a long bar or a sector.

Through the steps, a target data packet and configuration parameters corresponding to the target data packet are obtained, wherein the target data packet is a data packet to be operated, and the configuration parameters are used for determining resources to be used for operating the target data packet; then, replacing the target placeholder in the preset layout file based on the configuration parameters to obtain a target layout file, wherein the field of the position of the target placeholder is a field to be adjusted; determining a target resource space based on the target layout file, wherein the target resource space is used for providing an independent running resource for the target data packet; and finally, operating the target data packet based on the target resource space to obtain an operation result, thereby realizing the improvement of the operation efficiency of the target data packet. It is easy to notice that the target arrangement file corresponding to the target data packet can be quickly determined according to the configuration parameters and the preset arrangement file, and the target resource space corresponding to the target data packet is determined based on the target arrangement file, so that the operation process of the target data packet is isolated from the operation processes of other data packets through the target resource space, an independent operation resource is provided for the target data packet, the target data packet can be prevented from being influenced by other data packets, the operation efficiency of the target data packet can be improved, and the problem of low operation efficiency of the target data packet in the related technology can be solved.

Optionally, the obtaining of the configuration parameter corresponding to the target data packet includes: calling a first interface to analyze a target data packet to obtain a target operation resource, wherein the target operation resource is used for expressing a resource required by operation of the target data packet; determining a first parameter based on the target computational resource, wherein the first parameter is the number of first components required for calculating the target computational resource in the target resource space; acquiring a second parameter, wherein the second parameter is the number of second components for managing the target resource space; based on the first parameter and the second parameter, a configuration parameter is determined.

The first interface may be an API interface, where the interface is configured to analyze the target data packet to obtain the target operation resource, and optionally, after the target data packet is analyzed to obtain the target operation resource, a request of a user is sent to an API daemon, and the first parameter is further determined based on the target operation resource.

The first parameter is the number of first components required for calculating the target computational resource in the target resource space, and optionally, the number of first components may be the number of taskmanagers.

The second parameter may be the number of second components for managing the target resource space, and optionally, the number of second components may be the number of jobmanagers.

In an optional embodiment, when determining the configuration parameters, the number of taskmanagers and the number of jobmanagers may be determined, and then the number of parallel slots supported by the taskmanagers, the memory sizes corresponding to the jobmanagers and the taskmanagers, the log printing level, and other relevant configuration parameters may be further determined.

Optionally, the method further comprises: acquiring a target keyword of the initial arrangement template, wherein the target keyword is a field to be adjusted in the initial arrangement template; adding a target underline at a preset position of the target keyword to obtain a target placeholder; and adjusting the initial layout template based on the target placeholder to obtain a preset layout file.

The preset position may be a front-back position of the field to be adjusted.

The target underline may be an underline added at a predetermined position.

In an optional embodiment, a target underline may be added at a position before and after a field to be adjusted of the initial layout template, and the target underline is determined as a target placeholder, so as to obtain a preset layout file with the target placeholder, and optionally, in a subsequent use process, a user only needs to fill a keyword for replacing the target placeholder in the target placeholder.

Taking TaskManager-deployment as an example, the comparison key field may be copy data, where the copy number may be the number of work components, that is, one work component may process one task flow at the same time. This field determines the number of taskmanagers, i.e. the number of major components of the computation task. Only when the number of the taskmanagers is enough to support the maximum operator number of a job, the job can be run, in the template, the key word { { __ SLOT _ TM __ } } can be used as a placeholder, and when the API analyzes the maximum operator number of the job in the user data packet, the placeholder is replaced by the placeholder, so as to ensure that enough taskmanagers are started in the cluster for job running.

The preset resource space may share the log directory with the target resource space, so as to monitor the target resource space, and optionally, the preset resource space may be a filebed cluster.

In an optional embodiment, when collecting the flight logs, two K8S containers (containers) of the flight and the filebed can share a Pod mode for processing, wherein the Pod mode is a sharing mode, and by using the sharing mode, the filebed and the flight can make log paths mutually visible in a directory sharing mode, the flight logs can be printed on the log directory, and the filebed collects log files on the log directory in real time, wherein the filebed can be used for a lightweight transmission program for forwarding and concentrating log data, and is mainly used for log collection and transmission to Kafka in departments to store logs so as to ensure that the logs are collected completely.

In another optional embodiment, a monitoring component of a monitoring data acquisition program (also called exporter) supported by the Flink may be used, and after the exporter is started in the configuration, the system may automatically acquire various states of a Java Virtual Machine (also called JVM) in the Flink running process and report to a preset address, where the preset address may be set by a user, and a Prometheus in the same K8S cluster may identify the preset address reported by the exporter, take data according to the preset address, store and aggregate the data, and finally draw the data on an open source data visualization tool (also called Grafana) table by an operation and maintenance person. Optionally, the monitoring data is reported to a preset address according to the monitoring state, and the monitoring data is processed based on the preset address, so as to obtain a target table corresponding to the monitoring data.

Optionally, the method further comprises: and responding to a target instruction of the preset resource space, and stopping monitoring the processing process of the target resource space based on the log file, wherein the target instruction is generated in a preset time period after the target resource space stops running.

The target instruction may be a computer instruction generated for a preset time period after the target resource space stops running, where the preset time period may be self-defined by a user. Alternatively, the target instruction may act on the target resource space, and in response to the target instruction, monitoring of the target resource space during processing based on the log file may be stopped.

In an alternative embodiment, because it is necessary to consider ensuring the integrity of log collection, the following instruction may be added to filebed:

furthermore, by adopting the instruction, the acquisition of Filebeat and other Flink can be stopped after the operation of the Filebeat and other Flink is stopped, so that the complete acquisition of the logs is ensured.

The target platform can be oriented to the Flink user, and can manage a platform for starting and stopping the Flink cluster operation, controlling the operation life cycle, checking the operation error and iterating the version of the program operation package. The user can maintain the job version, check and adjust the job state and the like through the platform. Alternatively, the target platform may be the JFlink platform.

Optionally, when the user needs to obtain the target data packet, the user may perform touch operation on the control to send a touch instruction, and the target data packet may be obtained based on the touch instruction.

Optionally, when the user needs to upload the configuration information corresponding to the target data packet to the target platform, the control may be touch-controlled, so as to issue a touch-control instruction, and based on the touch-control instruction, the configuration information corresponding to the target data packet may be uploaded to the target platform.

In an alternative embodiment, the JFlink platform may manage the data packets through the job template, which may cause multiple versions of the data packets to exist in the job template at the same time because the versions of the data packets may be frequently iterated. When the method is introduced into the job template, a user can check all versions of data packages after selecting the job template when starting the job, the data packages can be uploaded to a data package list under the job template through clicking uploading operation, and the data packages of corresponding versions are started according to self requirements, so that the data packages are uploaded to a target platform.

In another alternative embodiment, the job name, Flink job run entry, person in charge, startup parameters, and timed save point may be configured to implement an edit job. Wherein, the timing saving point can be used for returning to the previous state when the error writing of the statement is found when the last statement is found to be executed. Optionally, when there are several versions of the background job template, each flink version is different, the job defaults to starting the cluster according to the required version, and the main role is to determine that the target version principal is mainly used for receiving the alarm and sending the alarm when K8S has problems.

Fig. 4 is a schematic diagram of a user submission interface display according to an embodiment of the present invention, as shown in fig. 4, in an upload package interface, a user is required to fill in a template name, identification information, and a project name, and upload a data package after completion of the filling, the user may click an upload button, select a version suffix as no suffix, and fill in a version number, and click to determine all information after no error is found, and click to cancel if there is an error, and re-fill in the correct information.

The first version information may be version information of a target resource space before upgrading, where the version information may be a version number, a version name, and the like.

The target mirror image can be used for providing an environment for operating a target resource space, and it can be determined based on the target mirror image which version the K8S uses to start the Flink cluster, optionally, if the mirror image version is missing, operation and maintenance personnel can quickly package and make, and under the condition that no special environment requirement exists, the specified mirror image can be directly pulled or updated from a mirror image warehouse, that is, a docker pull Flink instruction is adopted to obtain the mirror image. If special environment requirements exist, the following steps can be carried out:

step 1: version, a docker pull flash.

And pulling the latest flash, wherein version is the version required by the user.

Step 2: docker images | grep flink.

The mirror ID just pulled down is viewed.

And step 3: docker run-it mirror ID/bin/bash.

And (4) entering the mirror image, verifying the lower version through the application, keeping the running state, and installing required basic software and creating the environment according to the requirements of the user after entering the mirror image.

And 4, step 4: docker ps obtains the container ID.

After the container ID is acquired, operations such as copying of a file can be performed.

And 5: and (5) copying the file.

docker cp, data container id: /opt/flink/lib/.

And 6: and logging in a mirror image center.

docker login url。

And 7: the mark is mirrored.

docker tag flink：version url：version。

And 8: push to mirror center.

docker push url：version。

Furthermore, a perfect mirror image can be universal for all users once being manufactured, and the effect of once and for all is achieved.

Fig. 5 is a schematic diagram of an edit job interface display according to an embodiment of the present invention, as shown in fig. 5, in a newly created job interface, a user may select a job template and fill in a job name, the job name may be controlled within 100 words, and a default startup cluster, a designated entry, and a master principal may be selected, wherein the principal may be configured to receive an alarm and issue an alarm when a problem occurs in K8S. And simultaneously, performing operation description, filling in starting parameters, and selecting the configuration of the saving points, wherein the configuration of the saving points can be selected to be not configured, configured once every hour, configured once every half day or configured once every day. Optionally, the background job template has several versions, each flink version is different, the job is to default to start the cluster according to the required version, and the target version can be determined by editing the configuration of the job interface.

The third control element may be a control element of a third interface, where the third control element may issue a touch instruction, and in response to the touch instruction, the system may obtain the first version information corresponding to the target resource space.

In an alternative embodiment, the user may manually select the packet version of the startup job, the startup cluster, and manually configure the cluster configuration required for running the job, where the parameters are all reserved fields in the orchestration file.

Fig. 6 is a schematic view of a display of a job starting interface according to an embodiment of the present invention, as shown in fig. 6, in the job starting interface, a job template, a current usage version, a start cluster, and a resource policy may be selected first, after the above information is selected, the number of task managers and the number of slots in each task manager need to be filled, and a task manager memory and a model memory may be determined by clicking, and if the information is incorrect, the task manager memory and the model memory may be cancelled by clicking and then refilled.

The second version information may be version information of the upgraded target resource space, where the version information may be a version number, a version name, and the like.

The second interface can be an interface newly built in the upgrading process.

In an optional embodiment, in the operation of upgrading the K8S cluster, a high-version K8S may be built, at this time, only a service of a JAVA application program interface needs to be built again in the cluster, the JFLink uses a relationship between a data version stored in the database and the API, and when a version is newly added, a corresponding version number and an API address corresponding to the version are inserted into the database, such as: when a flash 1.14.3 version is newly added, an insert inter table (flash, v1.14.3, API) is selected by a user to start a cluster interface, so that all flash versions stored in the current database can be checked, and after the user selects a required version, a correct flash version cluster can be pulled up according to the corresponding API, so that the user can easily complete the migration of the operation through the version switching.

By adopting the deployment mode, the operation is convenient and flexible, the configuration can be selected by a user, the resource utilization maximization is achieved, furthermore, the migration of the cluster is remarkably facilitated, and a Flink environment can be quickly established after an arbitrary K8S cluster is given.

The preprocessing may be a one-step operation performed on the target data packet before the target data packet is analyzed based on the first interface, and a preprocessing result may be obtained through the preprocessing, where the result may be used to indicate whether the target data packet is damaged or not, or may not be operated due to an environment failure. Optionally, the target packet may be analyzed in case the target packet is normally available, thereby reaching the target computational resource.

In an optional embodiment, after the target data packet and the configuration parameters corresponding to the target data packet are obtained, the job management platform may receive a request for data processing sent by a user, where the request may be sent to an API daemon, and in response to the request for data processing, the target data packet may be sent to a Flink client for preprocessing. Optionally, the operation may analyze the target packet of the user, so as to obtain the maximum number of taskmanagers required by the operation of the target packet, and may also expose the problem of incapability of operation due to damage of the target packet, environmental obstruction, and the like in advance. Finally, the analyzed target data packet which can be normally run and returned is written into the TaskManager and the JobManager by the API background program, and the editing files are used for sending a request to a cluster brain (namely, a manager in the cluster, which does not participate in execution and is also called K8S master), so that a Flink on K8S cluster which meets the operation of user jobs can be pulled up.

In an alternative embodiment, when the user has submitted a job, the user requests that parameters be rendered into the K8S layout file template via the API, and initiates the script template to pull up the cluster. After the cluster is started, the management address of the JobManager is mapped to the domain name + job ID through the ingress configuration file, and the management address can be exposed. After the configuration is started, the JobManager interface of the cluster can be accessed by a user and cannot be isolated by the virtual IP environment of K8S, and when the JFlink platform takes the management address of the JobManager, the data packet of the user can be uploaded to the cluster through the interface, and then the data packet is started by using the interface.

Fig. 7 is a general framework diagram according to an embodiment of the present invention, as shown in the figure, a job management platform may start and stop a Flink cluster, start and stop jobs, modify cluster/job configuration, and save points at regular time, and view job status. The interface part can analyze the data packet, configure the cluster and start and stop the cluster operation. The K8S layer may perform rendering configuration, routing domain name, resource scheduling, high availability guarantees, and system logging operations. The method can be used for operation starting and stopping on the Flink cluster, high availability guarantee, a PerJob mode, data processing and database reading and writing operation. Among them, what is done on the K8S layer and the Flink cluster may be called logging. When data calling is carried out in the database, ES data, My SQL data, TiDB data, RocksDB data and Kafka data can be called, and notification can be carried out, wherein monitoring and alarming operations can be carried out on an interface part, a K8S layer, a Flink cluster and the database. The user can also control the authority of the job management platform, the interface part, the K8S layer, the Flink cluster and the database, and the running environment of all the information can be the K8S server or an independent server.

Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention or portions thereof contributing to the prior art may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (which may be a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.

In this embodiment, a data processing apparatus is further provided, and the apparatus is used to implement the foregoing embodiments and preferred embodiments, and details are not repeated for what has been described. As used below, the terms "unit", "module" may implement a combination of software and/or hardware of predetermined functions. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.

Fig. 8 is a block diagram of a data processing apparatus according to an embodiment of the present invention, as shown in fig. 8, the apparatus including:

an obtaining module 802, configured to obtain a target data packet and configuration parameters corresponding to the target data packet, where the target data packet is a data packet to be operated, and the configuration parameters are used to determine resources to be used for operating the target data packet.

And a replacing module 804, configured to replace the target placeholder in the preset layout file based on the configuration parameter, so as to obtain the target layout file, where a field at a position where the target placeholder is located is a field to be adjusted.

And a retrieving module 806, configured to determine a target resource space based on the target layout file, where the target resource space is used to provide an independent operating resource for the target data packet.

And the processing module 808 is configured to run the target data packet based on the target resource space to obtain a running result.

Optionally, the obtaining module 802 includes: the calling unit is used for calling the first interface to analyze the target data packet to obtain a target operation resource, wherein the target operation resource is used for expressing a resource required by operation of the target data packet; a first determining unit, configured to determine a first parameter based on a target computational resource, where the first parameter is a number of first components required to calculate the target computational resource in a target resource space; a first obtaining unit, configured to obtain a second parameter, where the second parameter is a number of second components used for managing a target resource space; a second determining unit, configured to determine the configuration parameter based on the first parameter and the second parameter.

Optionally, the apparatus further comprises: the second acquisition module is used for acquiring target keywords of the initial arrangement template, wherein the target keywords are fields to be adjusted in the initial arrangement template; adding a target underline at a preset position of the target keyword to obtain a target placeholder; and adjusting the initial layout template based on the target placeholder to obtain a preset layout file.

Optionally, the processing module 808 includes: the acquisition unit is used for acquiring a log file of a target resource space based on a preset resource space and a log directory; and the monitoring unit is used for monitoring the process of operating the target data packet in the target resource space based on the log file to obtain a monitoring result, wherein the monitoring result is used for indicating whether the process of operating the target data packet in the target resource space is abnormal or not.

Optionally, the monitoring unit is further configured to stop monitoring the processing procedure of the target resource space based on the log file in response to a target instruction for the preset resource space, where the target instruction is generated at a preset time period after the target resource space stops running.

Optionally, the obtaining module 802 further includes: the second obtaining unit is used for responding to a touch instruction of a first control in a first interface in the target platform and obtaining a target data packet, wherein the first interface is used for determining the target data packet from at least one data packet; and the uploading unit is used for responding to a touch instruction of a second control of a second interface in the target platform and uploading the configuration information corresponding to the target data packet to the target platform, wherein the second interface is used for editing the configuration information corresponding to the target data packet.

Optionally, the upload unit: the method comprises the following steps: the acquisition subunit is used for acquiring first version information corresponding to the target resource space; a determining subunit, configured to determine a target image based on the first version information, where the target image is used to provide an environment for running a target resource space; and the operation subunit is used for operating the target data packet based on the target mirror image and the target resource space to obtain an operation result.

Optionally, the obtaining subunit is further configured to obtain, in response to a touch instruction for a third control in a third interface in the target platform, first version information corresponding to the target resource space, where the third interface is configured to edit the first version information corresponding to the target resource space.

Optionally, the obtaining subunit is further configured to obtain a second interface in response to the target resource space upgrading instruction, where the second interface is associated with second version information, and the second version information is version information corresponding to the upgraded target resource space; the second interface is added to the third interface.

Optionally, the retrieving module 806 includes: the preprocessing unit is used for preprocessing the target data packet to obtain a preprocessing result, wherein the preprocessing result is used for indicating whether the target data packet can normally run in a target environment or not; and the analysis unit is used for analyzing the target data packet based on the first interface to obtain the target operation resource under the condition that the preprocessing result is that the target data packet can normally run in the target resource.

It should be noted that, the above units and modules may be implemented by software or hardware, and for the latter, the following may be implemented, but not limited to: the units and the modules are all positioned in the same processor; alternatively, the units and modules may be located in different processors in any combination.

An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

Alternatively, in the present embodiment, the above-mentioned computer-readable storage medium may be configured to store a computer program for executing the steps of:

and S1, acquiring a target data packet and configuration parameters corresponding to the target data packet, wherein the target data packet is a data packet to be operated, and the configuration parameters are used for determining resources to be used for operating the target data packet.

S2, replacing the target placeholder in the preset layout file based on the configuration parameters to obtain a target layout file, wherein the field of the position of the target placeholder is a field to be adjusted.

And S3, determining a target resource space based on the target layout file, wherein the target resource space is used for providing an independent running resource for the target data packet.

And S4, operating the target data packet based on the target resource space to obtain an operation result.

Optionally, in this embodiment, the computer-readable storage medium may include, but is not limited to: various media capable of storing computer programs, such as a usb disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.

Optionally, in this embodiment, the computer-readable storage medium may be located in any one of a group of computer terminals in a computer network, or in any one of a group of mobile terminals.

Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps:

Optionally, the target resource space and the preset resource space share a log directory, and in the process of running the target data packet by using the target resource space, the method further includes: collecting a log file of a target resource space based on a preset resource space and a log directory; and monitoring the process of operating the target data packet in the target resource space based on the log file to obtain a monitoring result, wherein the monitoring result is used for indicating whether the process of operating the target data packet in the target resource space is abnormal or not.

Optionally, the obtaining of the target data packet and the configuration parameter corresponding to the target data packet includes: responding to a touch instruction of a first control in a first interface in a target platform, and acquiring a target data packet, wherein the first interface is used for determining the target data packet from at least one data packet; and responding to a touch instruction of a second control of a second interface in the target platform, and uploading configuration information corresponding to the target data packet to the target platform, wherein the second interface is used for editing the configuration information corresponding to the target data packet.

Optionally, processing the target data packet by using the target resource space to obtain a processing result, including: acquiring first version information corresponding to a target resource space; determining a target mirror image based on the first version information, wherein the target mirror image is used for providing an environment for operating a target resource space; and operating the target data packet based on the target mirror image and the target resource space to obtain an operation result.

In the computer-readable medium of this embodiment, a target data packet and a configuration parameter corresponding to the target data packet are obtained first, where the target data packet is a data packet to be operated, and the configuration parameter is used to determine a resource to be used for operating the target data packet; then, replacing the target placeholder in the preset layout file based on the configuration parameters to obtain a target layout file, wherein the field of the position of the target placeholder is a field to be adjusted; determining a target resource space based on the target layout file, wherein the target resource space is used for providing an independent running resource for the target data packet; and finally, operating the target data packet based on the target resource space to obtain an operation result. It is easy to think that, after the target placeholder in the preset layout file is replaced based on the configuration parameter to obtain the target layout file, the target resource space can be determined based on the target layout file, so that the efficiency and the accuracy of data processing are further realized, and the technical problem of low efficiency in data operation in the related technology resource space is solved.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, and may also be implemented by software in combination with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to make a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) execute the method according to the embodiment of the present invention.

In an exemplary embodiment of the present application, a computer readable storage medium has stored thereon a program product capable of implementing the above-described method of the present embodiment. In some possible implementations, various aspects of the embodiments of the present invention may also be implemented in the form of a program product including program code for causing a terminal device to perform the steps according to various exemplary implementations of the present invention described in the above section "exemplary method" of this embodiment, when the program product is run on the terminal device.

According to the program product for realizing the method, the portable compact disc read only memory (CD-ROM) can be adopted, the program code is included, and the program product can be operated on terminal equipment, such as a personal computer. However, the program product of the embodiments of the invention is not limited thereto, and in the embodiments of the invention, the computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The program product described above may employ any combination of one or more computer-readable media. The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

It should be noted that the program code embodied on the computer readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Embodiments of the present invention further provide an electronic device, comprising a memory in which a computer program is stored and a processor configured to execute the computer program to perform the steps in any of the above method embodiments.

Optionally, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.

Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:

Optionally, the processor may be further configured to execute the following steps by a computer program: calling a first interface to analyze a target data packet to obtain target operation resources, wherein the target operation resources are used for expressing resources required by operation of the target data packet; determining a first parameter based on the target computational resource, wherein the first parameter is the number of first components required for computing the target computational resource in the target resource space; acquiring a second parameter, wherein the second parameter is the number of second components for managing the target resource space; based on the first parameter and the second parameter, a configuration parameter is determined.

Optionally, the processor may be further configured to execute the following steps by a computer program: acquiring a target keyword of the initial arrangement template, wherein the target keyword is a field to be adjusted in the initial arrangement template; adding a target underline at a preset position of the target keyword to obtain a target placeholder; and adjusting the initial layout template based on the target placeholder to obtain a preset layout file.

Optionally, the processor may be further configured to execute the following steps by a computer program: collecting a log file of a target resource space based on a preset resource space and a log directory; and monitoring the process of operating the target data packet in the target resource space based on the log file to obtain a monitoring result, wherein the monitoring result is used for indicating whether the process of operating the target data packet in the target resource space is abnormal or not.

Optionally, the processor may be further configured to execute the following steps by a computer program: and responding to a target instruction of the preset resource space, and stopping monitoring the processing process of the target resource space based on the log file, wherein the target instruction is generated within a preset time period after the target resource space stops running.

Optionally, the processor may be further configured to execute the following steps by a computer program: responding to a touch instruction of a first control in a first interface in a target platform, and acquiring a target data packet, wherein the first interface is used for determining the target data packet from at least one data packet; and responding to a touch instruction of a second control of a second interface in the target platform, and uploading configuration information corresponding to the target data packet to the target platform, wherein the second interface is used for editing the configuration information corresponding to the target data packet.

Optionally, the processor may be further configured to execute the following steps by a computer program: acquiring first version information corresponding to a target resource space; determining a target image based on the first version information, wherein the target image is used for providing an environment for running a target resource space; and operating the target data packet based on the target mirror image and the target resource space to obtain an operation result.

Optionally, the processor may be further configured to execute the following steps by a computer program: and responding to a touch instruction of a third control in a third interface in the target platform, and acquiring first version information corresponding to the target resource space, wherein the third interface is used for editing the first version information corresponding to the target resource space.

Optionally, the processor may be further configured to execute the following steps by a computer program: responding to an upgrading instruction of the target resource space, and acquiring a second interface, wherein the second interface is associated with second version information, and the second version information is version information corresponding to the upgraded target resource space; the second interface is added to the third interface.

Optionally, the processor may be further configured to execute the following steps by a computer program: preprocessing a target data packet to obtain a preprocessing result, wherein the preprocessing result is used for indicating whether the target data packet can normally run in a target environment; and analyzing the target data packet based on the first interface to obtain the target operation resource under the condition that the target data packet can normally run in the target resource as a preprocessing result.

In the electronic device of the embodiment, a target data packet and configuration parameters corresponding to the target data packet are obtained first, wherein the target data packet is a data packet to be operated, and the configuration parameters are used for determining resources to be used for operating the target data packet; then, replacing the target placeholder in the preset layout file based on the configuration parameters to obtain a target layout file, wherein the field of the position of the target placeholder is a field to be adjusted; determining a target resource space based on the target layout file, wherein the target resource space is used for providing an independent running resource for the target data packet; and finally, operating the target data packet based on the target resource space to obtain an operation result. It is easy to think that, after the target placeholder in the preset layout file is replaced based on the configuration parameter to obtain the target layout file, the target resource space can be determined based on the target layout file, so that the efficiency and the accuracy of data processing are further realized, and the technical problem of low efficiency in data operation in the related technology resource space is solved.

Fig. 9 is a schematic diagram of an electronic device according to an embodiment of the invention. As shown in fig. 9, the electronic device 900 is only an example and should not bring any limitation to the functions and the scope of the application of the embodiments of the present invention.

As shown in fig. 9, the electronic apparatus 900 is embodied in the form of a general purpose computing device. The components of electronic device 900 may include, but are not limited to: the at least one processor 910, the at least one memory 920, the bus 930 connecting the various system components (including the memory 920 and the processor 910), and the display 940.

Wherein the above-mentioned memory 920 stores program codes, which can be executed by the processor 910, so that the processor 910 performs the steps according to various exemplary embodiments of the present invention described in the above-mentioned method section of the embodiments of the present application.

The memory 920 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)9201 and/or a cache memory unit 9202, may further include a read only memory unit (ROM)9203, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid state memory.

In some examples, memory 920 may also include program/utility 9204 having a set (at least one) of program modules 9205, such program modules 13205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment. The memory 920 may further include memory remotely located from the processor 910 and such remote memory may be coupled to the electronic device 900 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

Bus 930 can be any of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, and processor 910 or a local bus using any of a variety of bus architectures.

Display 940 may, for example, be a touch screen type Liquid Crystal Display (LCD) that may enable a user to interact with a user interface of electronic device 900.

Optionally, the electronic apparatus 900 may also communicate with one or more external devices 900 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic apparatus 900, and/or any device (e.g., router, modem, etc.) that enables the electronic apparatus 900 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interfaces 950. Also, the electronic device 900 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) via the network adapter 960. As shown in fig. 9, the network adapter 960 communicates with the other modules of the electronic device 900 via the bus 930. It should be appreciated that although not shown in FIG. 9, other hardware and/or software modules may be used in conjunction with electronic device 900, which may include but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

The electronic device 900 may further include: a keyboard, a cursor control device (e.g., a mouse), an input/output interface (I/O interface), a network interface, a power source, and/or a camera.

It will be understood by those skilled in the art that the structure shown in fig. 9 is only an illustration and is not intended to limit the structure of the electronic device. For example, electronic device 900 may also include more or fewer components than shown in FIG. 9, or have a different configuration than shown in FIG. 1. The memory 920 may be used for storing a computer program and corresponding data, such as a computer program and corresponding data corresponding to the data processing method in the embodiment of the present invention. The processor 910 executes various functional applications and data processing by running computer programs stored in the memory 920, that is, implements the data processing method described above.

The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.

In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described apparatus embodiments are merely illustrative, and for example, the division of the units may be a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or may not be executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.

The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.

Claims

1. A method of data processing, the method comprising:

acquiring a target data packet and configuration parameters corresponding to the target data packet, wherein the target data packet is a data packet to be operated, and the configuration parameters are used for determining resources to be used for operating the target data packet;

replacing a target placeholder in a preset layout file based on the configuration parameters to obtain a target layout file, wherein a field at the position of the target placeholder is a field to be adjusted;

determining a target resource space based on the target arrangement file, wherein the target resource space is used for providing an independent running resource for the target data packet;

and operating the target data packet based on the target resource space to obtain an operation result.

2. The method of claim 1, wherein obtaining the configuration parameters corresponding to the target packet comprises:

calling a first interface to analyze the target data packet to obtain a target operation resource, wherein the target operation resource is used for expressing a resource required by operation of the target data packet;

determining a first parameter based on the target computational resource, wherein the first parameter is a number of first components required to compute the target computational resource in the target resource space;

acquiring a second parameter, wherein the second parameter is the number of second components for managing the target resource space;

determining the configuration parameter based on the first parameter and the second parameter.

3. The method of claim 1, further comprising:

acquiring a target keyword of an initial arranging template, wherein the target keyword is a field to be adjusted in the initial arranging template;

adding a target underline at a preset position of the target keyword to obtain the target placeholder;

and adjusting the initial layout template based on the target placeholder to obtain the preset layout file.

4. The method of claim 1, wherein the target resource space shares a log directory with a preset resource space, and during the running of the target data packet by using the target resource space, the method further comprises:

collecting a log file of the target resource space based on the preset resource space and the log directory;

and monitoring the process of operating the target data packet in the target resource space based on the log file to obtain a monitoring result, wherein the monitoring result is used for indicating whether the process of operating the target data packet in the target resource space is abnormal or not.

5. The method of claim 4, further comprising:

and responding to a target instruction of the preset resource space, and stopping monitoring the processing process of the target resource space based on the log file, wherein the target instruction is generated in a preset time period after the target resource space stops running.

6. The method of claim 1, wherein obtaining a target data packet and configuration parameters corresponding to the target data packet comprises:

responding to a touch instruction of a first control in a first interface in a target platform, and acquiring the target data packet, wherein the first interface is used for determining the target data packet from at least one data packet;

and responding to a touch instruction of a second control of a second interface in the target platform, and uploading configuration information corresponding to the target data packet to the target platform, wherein the second interface is used for editing the configuration information corresponding to the target data packet.

7. The method of claim 6, wherein processing the target data packet using the target resource space to obtain a processing result comprises:

acquiring first version information corresponding to the target resource space;

determining a target image based on the first version information, wherein the target image is used for providing an environment for running the target resource space;

and operating the target data packet based on the target mirror image and the target resource space to obtain the operation result.

8. The method of claim 7, wherein obtaining the first version information corresponding to the target resource space comprises:

and responding to a touch instruction of a third control in a third interface in the target platform, and acquiring first version information corresponding to the target resource space, wherein the third interface is used for editing the first version information corresponding to the target resource space.

9. The method of claim 8, further comprising:

responding to the target resource space upgrading instruction, and acquiring a second interface, wherein the second interface is associated with second version information, and the second version information is version information corresponding to the upgraded target resource space;

adding the second interface to the third interface.

10. The method of claim 1, wherein before analyzing the target packet based on the first interface to obtain the target computational resource, the method further comprises:

preprocessing the target data packet to obtain a preprocessing result, wherein the preprocessing result is used for indicating whether the target data packet can normally run in a target environment;

and analyzing the target data packet based on the first interface to obtain the target operation resource under the condition that the preprocessing result is that the target data packet can normally run in the target resource.

11. A data processing apparatus, characterized in that the apparatus comprises:

the device comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a target data packet and configuration parameters corresponding to the target data packet, the target data packet is a data packet to be operated, and the configuration parameters are used for determining resources to be used for operating the target data packet;

the replacing module is used for replacing the target placeholder in the preset layout file based on the configuration parameters to obtain a target layout file, wherein the field of the position of the target placeholder is a field to be adjusted;

the calling module is used for determining a target resource space based on the target layout file, wherein the target resource space is used for providing an independent running resource for the target data packet;

and the processing module is used for operating the target data packet based on the target resource space to obtain an operation result.

12. A computer-readable storage medium, in which a computer program is stored, wherein the computer program is arranged to, when executed by a processor, perform a data processing method as claimed in any one of claims 1 to 10.

13. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and the processor is arranged to execute the computer program to perform the data processing method of any of claims 1 to 10.