CN112925788A - Data set management method, system, device, electronic equipment and storage medium - Google Patents

Data set management method, system, device, electronic equipment and storage medium Download PDF

Info

Publication number
CN112925788A
CN112925788A CN202110113181.9A CN202110113181A CN112925788A CN 112925788 A CN112925788 A CN 112925788A CN 202110113181 A CN202110113181 A CN 202110113181A CN 112925788 A CN112925788 A CN 112925788A
Authority
CN
China
Prior art keywords
data set
processing
message
storage directory
identifier
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110113181.9A
Other languages
Chinese (zh)
Inventor
沙晶成
徐伟伟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Sensetime Intelligent Technology Co Ltd
Original Assignee
Shanghai Sensetime Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Sensetime Intelligent Technology Co Ltd filed Critical Shanghai Sensetime Intelligent Technology Co Ltd
Priority to CN202110113181.9A priority Critical patent/CN112925788A/en
Publication of CN112925788A publication Critical patent/CN112925788A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5083—Techniques for rebalancing the load in a distributed system
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/546—Message passing systems or structures, e.g. queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Evolutionary Computation (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Artificial Intelligence (AREA)
  • Mathematical Physics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The present disclosure relates to a data set management method, a system, an apparatus, an electronic device, and a storage medium, the method comprising: receiving a data set processing request aiming at a service container deployed on a physical machine, wherein the data set processing request comprises an identifier of a data set used by the service container and a storage directory of the service container mounted in the local of the physical machine; in response to the data set processing request, performing processing operation on the data set corresponding to the identifier in the storage directory, where the processing operation includes: and downloading the data set corresponding to the identifier into the storage directory, or deleting the data set corresponding to the identifier from the storage directory. The embodiment of the disclosure can realize quick access of the data set.

Description

Data set management method, system, device, electronic equipment and storage medium
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a method, a system, an apparatus, an electronic device, and a storage medium for data set management.
Background
In the fields of artificial intelligence, data science, and the like, a large amount of data is generally processed. For example, in machine learning by artificial intelligence, machine learning is generally performed by using a large amount of data; in large data-related studies, a large amount of data is also analyzed. In the fields of artificial intelligence and data science, a collection of a large amount of data is generally called a data set.
Due to the large amount of data in a data set, in a container-based distributed server (for example, a distributed container management system kubernets developed by google corporation), the data set is usually stored in a shared storage, and a plurality of containers perform data processing by remotely accessing the shared storage, but if a certain container needs to perform operations such as modification and deletion on data in the shared storage, the use of other containers is affected.
Disclosure of Invention
The present disclosure proposes a technical solution for data set management.
According to an aspect of the present disclosure, there is provided a data set management method including:
receiving a data set processing request aiming at a service container deployed on a physical machine, wherein the data set processing request comprises an identifier of a data set used by the service container and a storage directory of the service container mounted in the local of the physical machine;
in response to the data set processing request, performing processing operation on the data set corresponding to the identifier in the storage directory, where the processing operation includes: and downloading the data set corresponding to the identifier into the storage directory, or deleting the data set corresponding to the identifier from the storage directory.
In one possible implementation, after the receiving the data set processing request for the service container, the method further includes:
determining a processing progress of the processing operation;
and returning the processing progress to the front end through message middleware and/or a webpage socket WebSocket protocol.
In one possible implementation manner, the data set management method is applied to a distributed server, and the distributed server comprises a request receiving module and a data set processing module;
the processing operation on the data set corresponding to the identifier in the storage directory in response to the data set processing request includes:
the request receiving module receives a data set processing request, generates a data set processing message supported by a message middleware according to the data set processing request, and pushes the data set processing message to a message queue corresponding to the physical machine in the message middleware;
and the data set processing module positioned in the physical machine acquires the data set processing message in the message queue, and processes the data set corresponding to the identifier in the storage directory according to the identifier and the storage directory in the data set processing message.
In a possible implementation manner, the distributed server further includes a processing progress pushing module and a Websocket server, and the returning the processing progress to the front end through the message middleware and the Websocket protocol includes:
the processing progress pushing module generates a processing progress message according to the processing progress and pushes the processing progress message to a message middleware;
and the Websocket server acquires the processing progress message from the message middleware and sends the processing progress in the processing progress message to a front Websocket client.
In a possible implementation manner, after the downloading the data set corresponding to the identifier into the storage directory, the method further includes:
receiving a service processing request aiming at the service container;
and responding to the business processing request, and utilizing the data set in the storage directory to perform business processing operation.
In one possible implementation, the method further includes:
and deleting the data set in the storage directory in response to the closing of the service container.
In one possible implementation, downloading the data set corresponding to the identifier to the storage directory includes:
and downloading the data set corresponding to the identification to the service container from a storage directory of a database or other containers, and mounting the service container in the local storage directory of the physical machine.
According to an aspect of the present disclosure, there is provided a data set management apparatus including:
a request receiving unit, configured to receive a data set processing request for a service container deployed on a physical machine, where the data set processing request includes an identifier of a data set used by the service container, and a storage directory in which the service container is mounted in a local area of the physical machine;
a data set processing unit, configured to, in response to the data set processing request, perform a processing operation on the data set corresponding to the identifier in the storage directory, where the processing operation includes: and downloading the data set corresponding to the identifier into the storage directory, or deleting the data set corresponding to the identifier from the storage directory.
In one possible implementation, the apparatus further includes:
a progress determination unit configured to determine a processing progress of the processing operation;
and the progress returning unit is used for returning the processing progress to the front end through message middleware and/or a webpage socket WebSocket protocol.
In one possible implementation manner, the device is applied to a distributed server, and the distributed server comprises a request receiving module and a data set processing module;
the data set processing unit comprises the request receiving module and the data set processing module, wherein the request receiving module receives a data set processing request, generates a data set processing message supported by a message middleware according to the data set processing request, and pushes the data set processing message to a message queue corresponding to the physical machine in the message middleware; and the data set processing module positioned in the physical machine acquires the data set processing message in the message queue, and processes the data set corresponding to the identifier in the storage directory according to the identifier and the storage directory in the data set processing message.
In a possible implementation manner, the distributed server further includes a processing progress pushing module and a Websocket server, and the progress returning unit includes the processing progress pushing module and the Websocket server, where the processing progress pushing module generates a processing progress message according to the processing progress and pushes the processing progress message to a message middleware; and the Websocket server acquires the processing progress message from the message middleware and sends the processing progress in the processing progress message to a front Websocket client.
In one possible implementation, the apparatus further includes:
a service processing request receiving unit, configured to receive a service processing request for the service container;
and the service processing unit is used for responding to the service processing request and utilizing the data set in the storage directory to perform service processing operation.
In one possible implementation, the apparatus further includes:
and the deleting unit is used for deleting the data set in the storage directory in response to the closing of the service container.
In a possible implementation manner, the data set processing unit is configured to download the data set corresponding to the identifier from a storage directory of a database or another container to the storage directory of the physical machine local mounted by the service container.
According to an aspect of the present disclosure, there is provided a data set management system including:
the request receiving module is used for receiving a data set processing request aiming at a service container deployed on a physical machine, wherein the data set processing request comprises an identifier of a data set used by the service container and a storage directory of the service container mounted in the local of the physical machine;
a data set processing module, configured to, in response to the data set processing request, perform a processing operation on the data set corresponding to the identifier in the storage directory, where the processing operation includes: and downloading the data set corresponding to the identifier into the storage directory, or deleting the data set corresponding to the identifier from the storage directory.
According to an aspect of the present disclosure, there is provided an electronic device including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the memory-stored instructions to perform the above-described method.
According to an aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the above-described method.
In this embodiment of the present disclosure, after a service container is created on a physical machine, when a data set processing request for the service container is received, according to an identifier of a data set used by the service container included in the data set processing request, under a storage directory where the service container is mounted in the local area of the physical machine, a processing operation is performed on the data set corresponding to the identifier, where the processing operation includes: and downloading the data set corresponding to the identifier into the storage directory, or deleting the data set corresponding to the identifier from the storage directory. Therefore, the data set is synchronized to the physical machine where the user container is located and is mounted in the storage directory where the container is local to the physical machine, so that the user can access the data set of the user as local data, and the data set is not influenced by network delay. In addition, the data set is stored in the storage catalog mounted on the container, so that the method has a good isolation effect, the operation on the data set between the containers is not influenced mutually, and the user can freely carry out the operation such as modification, deletion and the like on the data set.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.
FIG. 1 shows a flow diagram of a data set management method according to an embodiment of the present disclosure;
FIG. 2 illustrates an architecture diagram of a distributed server provided in accordance with an embodiment of the present disclosure;
FIG. 3 illustrates a block diagram of a data set management system according to an embodiment of the present disclosure;
FIG. 4 illustrates a schematic structural diagram of yet another data set management system according to an embodiment of the present disclosure;
FIG. 5 illustrates a schematic structural diagram of yet another data set management system according to an embodiment of the present disclosure;
FIG. 6 illustrates a schematic structural diagram of yet another data set management system according to an embodiment of the present disclosure;
FIG. 7 shows a block diagram of a data set management apparatus according to an embodiment of the present disclosure;
FIG. 8 shows a block diagram of an electronic device in accordance with an embodiment of the disclosure;
fig. 9 shows a block diagram of an electronic device in accordance with an embodiment of the disclosure.
Detailed Description
Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers can indicate functionally identical or similar elements. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.
The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.
The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.
Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements and circuits that are well known to those skilled in the art have not been described in detail so as not to obscure the present disclosure.
With the development of computer technology, programs have entered the era of virtualized deployment, and programs can be virtualized deployed through containers, and in a container-based distributed server (for example, a distributed container management system kubernets developed by google corporation), distributed processing of tasks by programs can be realized by deploying the same program in multiple containers, respectively.
In the field of artificial intelligence, an artificial intelligence platform can be deployed on a Kubernets cluster so that users can perform artificial intelligence-related learning, research and application on the platform, such as machine learning, data science, data processing and the like. The activities described above tend to process the data set, for example, using the data set for machine learning.
In large-scale container clusters, how to manage the data set is crucial. If the data set is stored in the shared storage, network delay may cause a performance bottleneck to occur in the process of using the data set by a user, so that the computational efficiency is greatly reduced.
In the embodiment of the disclosure, the data set is synchronized to the physical machine where the user container is located, and is mounted in the storage directory where the container is local to the physical machine, so that the user can access the data set of the user as local data, and the data set is not affected by network delay. In addition, the data set is stored in the storage catalog mounted on the container, so that the method has a good isolation effect, the operation on the data set between the containers is not influenced mutually, and the user can freely carry out the operation such as modification, deletion and the like on the data set.
In one possible implementation, the data set management method may be performed by an electronic device such as a terminal device or a server, the terminal device may be a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, or the like, and the method may be implemented by a processor calling computer readable instructions stored in a memory. Alternatively, the method may be executed by a server, for example, the method may be executed by a distributed server, and the following takes the execution subject as the distributed server as an example, and a possible implementation manner of the present disclosure is exemplified.
In the disclosed embodiments, the business container may be created on a physical machine. After receiving a service processing request sent by a user front end, a corresponding service container can be created, the service container is used for processing a service corresponding to the request, and a process for processing the service is loaded in the container. The service may be, for example, a machine learning task in the field of artificial intelligence, a data processing service in data science, and so on.
The service container is created on a certain physical machine, a plurality of service containers can be created on the same physical machine, after the service container is created, the address of the physical machine where the service container is located can be recorded in the service container, and a local storage directory of the physical machine can be mounted on the service container, and the storage directory can be used for the service container to perform operations such as data set storage, reading, modification, deletion and the like. The storage directories mounted by different service containers are not influenced mutually.
The service processing request may be sent by a user front end, where the user front end may be a browser, or may also be a client, that is, a user may send the service processing request by operating in a web page of the browser.
After the service container is created, a message that the creation of the service container is successful can be sent to the user front end, and in addition, a storage directory in which the service container is mounted in a physical machine local can be sent to the user front end.
After creating the service container, data set management may be performed on the service container, fig. 1 shows a flowchart of a data set management method according to an embodiment of the present disclosure, and as shown in fig. 1, the data set management method includes:
in step S11, a data set processing request for a service container deployed on a physical machine is received, where the data set processing request includes an identifier of a data set used by the service container, and a storage directory local to the physical machine where the service container is mounted.
The user front end displays the data set available for the user to select, and after the user selects the data set used by the service container, the user front end encapsulates the identifier of the data set selected by the user into a data set processing request, and in addition, the request can also encapsulate the storage catalog of the service container mounted in the physical machine local.
The user front end sends the encapsulated data set processing request to the back end server, wherein the back end server is an execution main body of the implementation mode.
The data set processing request can be a user request for downloading the data set into a local storage directory of the physical machine so as to facilitate subsequent use of the data set; alternatively, the request may be a user request to delete the data set from a storage directory local to the physical machine.
In the embodiment of the present disclosure, the data set may be a data set composed of data having a common theme and/or similar attributes, for example, a data set used for machine learning in artificial intelligence, or a data set used for data science.
In step S12, in response to the data set processing request, a processing operation is performed on the data set corresponding to the identifier in the storage directory, where the processing operation includes: and downloading the data set corresponding to the identifier into the storage directory, or deleting the data set corresponding to the identifier from the storage directory.
After receiving a data set processing request, downloading the data set corresponding to the identifier into a storage directory local to the physical machine under the condition that the request requests to download the data set into the storage directory. Specifically, according to the identifier, the data set corresponding to the identifier may be downloaded from the shared object store.
In the event that the request requests deletion of a data set from a storage directory local to the physical machine, the data set corresponding to the identification is deleted from the storage directory.
Referring to fig. 2, fig. 2 shows an architecture diagram of a distributed server provided by an embodiment of the present disclosure, as shown in fig. 2, the distributed server includes a plurality of physical machines, users 1 to N send user requests through a front end, and the user requests are distributed to a container of a certain physical machine through load balancing, a plurality of service containers may be deployed in one physical machine, each service container corresponds to one data set, and the data sets are isolated from each other and do not interfere with each other.
In this embodiment of the present disclosure, after a service container is created on a physical machine, when a data set processing request for the service container is received, according to an identifier of a data set used by the service container included in the data set processing request, under a storage directory where the service container is mounted in the local area of the physical machine, a processing operation is performed on the data set corresponding to the identifier, where the processing operation includes: and downloading the data set corresponding to the identifier into the storage directory, or deleting the data set corresponding to the identifier from the storage directory. Therefore, the data set is synchronized to the physical machine where the user container is located and is mounted in the storage directory where the container is local to the physical machine, so that the user can access the data set of the user as local data, and the data set is not influenced by network delay. In addition, the data set is stored in the storage catalog mounted on the container, so that the method has a good isolation effect, the operation on the data set between the containers is not influenced mutually, and the user can freely carry out the operation such as modification, deletion and the like on the data set.
In one possible implementation, downloading the data set corresponding to the identifier to the storage directory includes: and downloading the data set corresponding to the identification to the service container from a storage directory of a database or other containers, and mounting the service container in the local storage directory of the physical machine.
The database can be a storage space for storing the data set in a centralized manner, so that the data set can be downloaded from the database when being downloaded; in addition, if the data set corresponding to the identifier is also stored in the storage directory of another container, the data set can be downloaded from the storage directory of another container, so that the occupation of the database bandwidth is reduced, and the downloading efficiency of the data set is improved.
In a possible implementation manner, the data sets in the storage directories of other containers may be already edited data sets, and then downloading the edited data sets from the storage directories of other containers enables sharing of the edited data sets, thereby improving the efficiency of performing business processing on the data sets.
In a possible implementation manner, after the downloading the data set corresponding to the identifier into the storage directory, the method further includes: receiving a service processing request aiming at the service container; and responding to the business processing request, and utilizing the data set in the storage directory to perform business processing operation.
As mentioned above, the service container may be used to process the service requested by the service processing request, where the service may be, for example, a machine learning service in machine learning of artificial intelligence, or a data analysis service in big data related research, etc.
In the process of processing business, the data set in the storage directory is used for business processing operation, for example, training a neural network by using the data set, or performing user behavior analysis by using the data set, etc.
In the embodiment of the present disclosure, by using the data set in the storage directory to perform the service processing operation, the data set is stored in the storage directory where the containers are mounted, so that a better isolation effect is achieved, the operations on the data set between the containers do not affect each other, and a user can freely perform operations such as modification and deletion of the data set.
In one possible implementation, after the receiving the data set processing request for the service container, the method further includes: determining a processing progress of the processing operation; and returning the processing progress to the front end through message middleware and/or a webpage socket WebSocket protocol.
The processing progress here is, for example, the progress of downloading the data set, for example, the data set 01 has been downloaded completely, or the data set 02 has been downloaded 65%; the processing progress may also be a progress of deleting the data set, for example, the data set 03 has been deleted, or the data set 03 fails to delete, please retry.
In the embodiment of the disclosure, the processing progress can be returned to the front end through the message middleware and the WebSocket protocol. The message middleware can transmit the processing progress asynchronously through the message queue so as to improve the transmission efficiency of the processing progress among different components of the distributed server. In addition, the processing progress is pushed to the front end from the server through the WebSocket protocol, and compared with the situation that the user front end requests the processing progress to the server in a polling mode, the pressure of the server is reduced.
In one possible implementation manner, the data set management method is applied to a distributed server, and the distributed server comprises a request receiving module and a data set processing module; the processing operation on the data set corresponding to the identifier in the storage directory in response to the data set processing request includes: a request receiving module receives a data set processing request, generates a data set processing message supported by a message middleware according to the data set processing request, and pushes the data set processing message to a message queue corresponding to the physical machine in the message middleware; and the data set processing module positioned in the physical machine acquires the data set processing message in the message queue, and processes the data set corresponding to the identifier in the storage directory according to the identifier and the storage directory in the data set processing message.
In this implementation, the message middleware may operate with a publish-subscribe mode in which a message sender (producer) publishes a message and one or more message recipients (consumers) subscribe to the message. The message sender (producer) and the message receiver (consumer) are not directly coupled, the message sender sends the message to a distributed message queue in the message middleware, namely, the message receiver finishes processing the message, and the message receiver acquires the message from the distributed message queue and then performs subsequent processing.
In this implementation, after receiving the data set processing request, the request receiving module generates a message supported by the message middleware according to the data set processing request, and for convenience of description, the generated message is referred to as a data set processing message, and the data set processing message still includes an identifier of a data set used by the service container and a storage directory in which the service container is mounted in a physical machine local.
The request receiving module can be used as a producer in a 'publish-subscribe mode' to push the data set processing message to the message middleware, and the data set processing message can be put into a message queue corresponding to a physical machine where the service container is located.
The data set processing module in the physical machine where the service container is located can be used as a consumer in a 'publish-subscribe mode', acquire a data set processing message from a message queue corresponding to the physical machine, analyze the data set processing message to obtain an identifier of a data set contained in the data set processing message and a storage directory where the service container is mounted in the local of the physical machine, and then perform processing operation on the data set corresponding to the identifier under the storage directory.
In the embodiment of the disclosure, the identifier and the storage directory of the data set included in the data set processing request can be asynchronously transmitted to the data set processing module in the physical machine where the service container is located through the message queue in the message middleware, so that the transmission efficiency is high.
In a possible implementation manner, the distributed server further includes a processing progress pushing module and a Websocket server, and the returning of the processing progress to the front end through the message middleware and/or the Websocket protocol includes: the processing progress pushing module generates a processing progress message according to the processing progress and pushes the processing progress message to a message middleware; and the Websocket server acquires the processing progress message from the message middleware and sends the processing progress in the processing progress message to a front Websocket client.
In this implementation, the message middleware can still work in a publish-subscribe mode, and the processing progress pushing module can be used as a producer in the publish-subscribe mode to obtain the processing progress of data processing, and then encapsulate the processing progress into a message supported by the message middleware, that is, a processing progress message. The processing progress pushing module pushes the processing progress message to a message queue of the message middleware, and specifically puts the processing progress message into the message queue corresponding to the service container.
The Websocket server connected with the user front end can be used as a consumer in a 'publish-subscribe mode', obtains a processing progress message from a message queue corresponding to the service container, analyzes the processing progress message to obtain a processing progress, and then sends the processing progress to the Websocket client of the user front end. The Websocket server can package the processing progress into a data transmission format supported by a Websocket protocol, and then sends the data transmission format to the Websocket client. After receiving the processing progress, the Websocket client may show the processing progress to the user through the user front end, for example, the processing progress may be shown through a web page in a browser.
In the embodiment of the disclosure, the processing progress can be asynchronously transmitted to the Websocket server through the message queue in the message middleware, the transmission efficiency is high, and the processing progress is pushed to the front end through the Websocket server, so that the pressure of the server is reduced compared with the situation that the user front end requests the server for the processing progress in a polling manner.
In one possible implementation, the method further includes: and deleting the data set in the storage directory in response to the closing of the service container. Under the condition of service container destruction, the data set in the local storage directory of the physical machine mounted with the service container can be simultaneously destroyed, so that the storage space in the physical machine is saved.
In addition, the present disclosure also provides a data set management system 30, as shown in fig. 3, including:
a request receiving module 301, configured to receive a data set processing request for a service container deployed on a physical machine, where the data set processing request includes an identifier of a data set used by the service container and a storage directory in which the service container is mounted in a local area of the physical machine;
the data set processing module 303, in response to the data set processing request, performs a processing operation on the data set corresponding to the identifier in the storage directory, where the processing operation includes: and downloading the data set corresponding to the identifier into the storage directory, or deleting the data set corresponding to the identifier from the storage directory.
In this embodiment of the present disclosure, after a service container is created on a physical machine, when a data set processing request for the service container is received, according to an identifier of a data set used by the service container included in the data set processing request, under a storage directory where the service container is mounted in the local area of the physical machine, a processing operation is performed on the data set corresponding to the identifier, where the processing operation includes: and downloading the data set corresponding to the identifier into the storage directory, or deleting the data set corresponding to the identifier from the storage directory. Therefore, the data set is synchronized to the physical machine where the user container is located and is mounted in the storage directory where the container is local to the physical machine, so that the user can access the data set of the user as local data, and the data set is not influenced by network delay. In addition, the data set is stored in the storage catalog mounted on the container, so that the method has a good isolation effect, the operation on the data set between the containers is not influenced mutually, and the user can freely carry out the operation such as modification, deletion and the like on the data set.
In a possible implementation manner, the data set management system 30 further includes a message middleware 302, please refer to fig. 4, in which the request receiving module 301 receives a data set processing request for a service container, generates a data set processing message supported by the message middleware 302 according to the data set processing request, and pushes the data set processing message to the message middleware 302, where the data set processing request includes an identifier of a data set used by the service container and a storage directory of the physical machine local mounted by the service container;
the message middleware 302 places the data set processing message in a message queue;
and the data set processing module 303 acquires the data set processing message in the message queue 302, and performs processing operation on the data set corresponding to the identifier under the storage directory.
In the disclosed embodiment, the Message middleware 302 may be a rabbitmq (Message queue, MQ), or may also be an Active QM, Kafka, socket MQ, Zero MQ, etc., which is not limited by the disclosed embodiment.
The message middleware 302 may include a switch (Exchange), the Exchange may distribute a message sent by a producer, and during a process of distributing a message, distribution modes are also different according to different message types, for example, the following four modes may be included: direct (direct) mode, broadcast (fanout) mode, topic (topic) mode, and headers (headers) mode.
In the process of placing the data set processing message into the message queue, the message middleware 302 may use a topic mode, and in this mode, the data set processing message may be placed into the message queue corresponding to the physical machine where the service container is located according to an identifier of the physical machine included in a routing-key (routing-key) of the data set processing message. The data set processing module 303 on the physical machine acquires the data set processing message in the message queue, and performs processing operation on the data set corresponding to the data set identifier under the storage directory.
In the embodiment of the disclosure, the data set is synchronized to the physical machine where the user container is located, and is mounted in the storage directory where the container is local to the physical machine, so that the user can access the data set of the user as local data, and the data set is not affected by network delay. In addition, the data set is stored in the storage catalog mounted on the container, so that the method has a good isolation effect, the operation on the data set between the containers is not influenced mutually, and the user can freely carry out the operation such as modification, deletion and the like on the data set. In addition, the identification and the storage directory of the data set contained in the data set processing request are asynchronously transmitted to the data set processing module in the physical machine where the service container is located through the message queue in the message middleware, so that the transmission efficiency is high.
In a possible implementation manner, the system 30 further includes a processing progress pushing module 304 and a Websocket server 305, as shown in fig. 5, where:
the processing progress pushing module 304 is configured to generate a processing progress message according to the processing progress of the processing operation, and push the processing progress message to the message middleware 302;
the message middleware 302, which puts the processing progress message into a message queue;
the Websocket server 305 obtains the processing progress message from the message queue of the message middleware 302, and sends the processing progress in the processing progress message to the Websocket client at the front end.
Message middleware 302 may use fanout mode in which a message queue is bound to a switch during the process of placing a process progress message into the message queue, and the process progress message sent to the switch is forwarded to all message queues bound to the switch. Then, the Websocket server can obtain the processing progress message from the message queue of the message middleware 302.
In the embodiment of the disclosure, the processing progress can be asynchronously transmitted to the Websocket server through the message queue in the message middleware, the transmission efficiency is high, and the processing progress is pushed to the front end through the Websocket server, so that the pressure of the server is reduced compared with the situation that the user front end requests the server for the processing progress in a polling manner.
Please refer to fig. 6, which is a schematic structural diagram of a data set management system according to the present application, the system includes: the system comprises a request receiving module, a message middleware, a data set processing module, a processing progress pushing module and a Websocket server, wherein the message middleware comprises a switch and a message queue.
After a user sends a data set processing request through a front end, the data set processing request is distributed to a request receiving module through load balancing, the request receiving module generates a data set processing message supported by a message middleware according to the data set processing request and pushes the data set processing message to the message middleware, and the data set processing request comprises an identifier of a data set used by a service container and a storage directory of the physical machine local mounted by the service container.
The switch of the message middleware may adopt a topic mode, and place the data set processing message into a message queue corresponding to a physical machine where the service container is located according to an identifier of the physical machine included in a routing key (routingkey) of the data set processing message. The data set processing module on the physical machine acquires the data set processing message in the message queue, and performs processing operation on the data set corresponding to the data set identifier under the storage directory.
The processing progress pushing module is used for generating a processing progress message according to the processing progress of the processing operation and pushing the processing progress message to the message middleware;
the switch of the message middleware adopts fanout mode to bind the message queue to the switch, and the processing progress message sent to the switch is forwarded to all the message queues bound to the switch. And the Websocket server acquires the processing progress message from the message queue of the message middleware and sends the processing progress in the processing progress message to the Websocket client at the front end.
In the embodiment of the disclosure, the data set is synchronized to the physical machine where the user container is located, and is mounted in the storage directory where the container is local to the physical machine, so that the user can access the data set of the user as local data, and the data set is not affected by network delay. In addition, the data set is stored in the storage catalog mounted on the container, so that the method has a good isolation effect, the operation on the data set between the containers is not influenced mutually, and the user can freely carry out the operation such as modification, deletion and the like on the data set. In addition, the identification and the storage directory of the data set contained in the data set processing request are asynchronously transmitted to the data set processing module in the physical machine where the service container is located through the message queue in the message middleware, so that the transmission efficiency is high.
In the embodiment of the disclosure, the processing progress can be asynchronously transmitted to the Websocket server through the message queue in the message middleware, the transmission efficiency is high, and the processing progress is pushed to the front end through the Websocket server, so that the pressure of the server is reduced compared with the situation that the user front end requests the server for the processing progress in a polling manner.
For the implementation of the specific functions of each module, please refer to the related discussion in this disclosure, which is not described herein again.
It is understood that the above-mentioned method embodiments of the present disclosure can be combined with each other to form a combined embodiment without departing from the logic of the principle, which is limited by the space, and the detailed description of the present disclosure is omitted. Those skilled in the art will appreciate that in the above methods of the specific embodiments, the specific order of execution of the steps should be determined by their function and possibly their inherent logic.
In addition, the present disclosure also provides a data set management apparatus, an electronic device, a computer-readable storage medium, and a program, which can be used to implement any data set management method provided by the present disclosure, and the corresponding technical solutions and descriptions and corresponding descriptions in the method section are not repeated.
Fig. 7 shows a block diagram of a data set management apparatus according to an embodiment of the present disclosure, as shown in fig. 7, the apparatus 70 includes:
a request receiving unit 71, configured to receive a data set processing request for a service container deployed on a physical machine, where the data set processing request includes an identifier of a data set used by the service container, and a storage directory in which the service container is mounted in a local area of the physical machine;
a data set processing unit 72, configured to, in response to the data set processing request, perform a processing operation on the data set corresponding to the identifier in the storage directory, where the processing operation includes: and downloading the data set corresponding to the identifier into the storage directory, or deleting the data set corresponding to the identifier from the storage directory.
In one possible implementation, the apparatus further includes:
a progress determination unit configured to determine a processing progress of the processing operation;
and the progress returning unit is used for returning the processing progress to the front end through message middleware and/or a webpage socket WebSocket protocol.
In one possible implementation manner, the device is applied to a distributed server, and the distributed server comprises a request receiving module and a data set processing module;
the data set processing unit 72 includes the request receiving module and the data set processing module, where the request receiving module receives a data set processing request, generates a data set processing message supported by a message middleware according to the data set processing request, and pushes the data set processing message to a message queue corresponding to the physical machine in the message middleware; and the data set processing module positioned in the physical machine acquires the data set processing message in the message queue, and processes the data set corresponding to the identifier in the storage directory according to the identifier and the storage directory in the data set processing message.
In a possible implementation manner, the distributed server further includes a processing progress pushing module and a Websocket server, and the progress returning unit includes the processing progress pushing module and the Websocket server, where the processing progress pushing module generates a processing progress message according to the processing progress and pushes the processing progress message to a message middleware; and the Websocket server acquires the processing progress message from the message middleware and sends the processing progress in the processing progress message to a front Websocket client.
In one possible implementation, the apparatus further includes:
a service processing request receiving unit, configured to receive a service processing request for the service container;
and the service processing unit is used for responding to the service processing request and utilizing the data set in the storage directory to perform service processing operation.
In one possible implementation, the apparatus further includes:
and the deleting unit is used for deleting the data set in the storage directory in response to the closing of the service container.
In a possible implementation manner, the data set processing unit 72 is configured to download the data set corresponding to the identifier from a storage directory of a database or other container to the storage directory of the physical machine local mounted with the service container.
In some embodiments, functions of or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the method described in the above method embodiments, and specific implementation thereof may refer to the description of the above method embodiments, and for brevity, will not be described again here.
Embodiments of the present disclosure also provide a computer-readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the above-mentioned method. The computer readable storage medium may be a non-volatile computer readable storage medium.
An embodiment of the present disclosure further provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the memory-stored instructions to perform the above-described method.
The disclosed embodiments also provide a computer program product comprising computer readable code which, when run on a device, a processor in the device executes instructions for implementing a data set management method as provided in any of the above embodiments.
The disclosed embodiments also provide another computer program product for storing computer readable instructions, which when executed, cause a computer to perform the operations of the data set management method provided by any of the above embodiments.
The electronic device may be provided as a terminal, server, or other form of device.
Fig. 8 illustrates a block diagram of an electronic device 800 in accordance with an embodiment of the disclosure. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like terminal.
Referring to fig. 8, electronic device 800 may include one or more of the following components: processing component 802, memory 804, power component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814, and communication component 816.
The processing component 802 generally controls overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 802 may include one or more processors 820 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 802 can include one or more modules that facilitate interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.
The memory 804 is configured to store various types of data to support operations at the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 804 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.
The power supply component 806 provides power to the various components of the electronic device 800. The power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.
The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the electronic device 800 is in an operation mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.
The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.
The sensor assembly 814 includes one or more sensors for providing various aspects of state assessment for the electronic device 800. For example, the sensor assembly 814 may detect an open/closed state of the electronic device 800, the relative positioning of components, such as a display and keypad of the electronic device 800, the sensor assembly 814 may also detect a change in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, orientation or acceleration/deceleration of the electronic device 800, and a change in the temperature of the electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include a light sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 may access a wireless network based on a communication standard, such as a wireless network (WiFi), a second generation mobile communication technology (2G) or a third generation mobile communication technology (3G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the electronic device 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.
In an exemplary embodiment, a non-transitory computer-readable storage medium, such as the memory 804, is also provided that includes computer program instructions executable by the processor 820 of the electronic device 800 to perform the above-described methods.
Fig. 9 illustrates a block diagram of an electronic device 1900 in accordance with an embodiment of the disclosure. For example, the electronic device 1900 may be provided as a server. Referring to fig. 9, electronic device 1900 includes a processing component 1922 further including one or more processors and memory resources, represented by memory 1932, for storing instructions, e.g., applications, executable by processing component 1922. The application programs stored in memory 1932 may include one or more modules that each correspond to a set of instructions. Further, the processing component 1922 is configured to execute instructions to perform the above-described method.
The electronic device 1900 may also include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input/output (I/O) interface 1958. The electronic device 1900 may operate based on an operating system, such as the Microsoft Server operating system (Windows Server), stored in the memory 1932TM) Apple Inc. of the present application based on the graphic user interface operating System (Mac OS X)TM) Multi-user, multi-process computer operating system (Unix)TM) Free and open native code Unix-like operating System (Linux)TM) Open native code Unix-like operating System (FreeBSD)TM) Or the like.
In an exemplary embodiment, a non-transitory computer readable storage medium, such as the memory 1932, is also provided that includes computer program instructions executable by the processing component 1922 of the electronic device 1900 to perform the above-described methods.
The present disclosure may be systems, methods, and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied thereon for causing a processor to implement various aspects of the present disclosure.
The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.
The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.
The computer program instructions for carrying out operations of the present disclosure may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, the electronic circuitry that can execute the computer-readable program instructions implements aspects of the present disclosure by utilizing the state information of the computer-readable program instructions to personalize the electronic circuitry, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA).
Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The computer program product may be embodied in hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied in a computer storage medium, and in another alternative embodiment, the computer program product is embodied in a Software product, such as a Software Development Kit (SDK), or the like.
Having described embodiments of the present disclosure, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims (11)

1. A data set management method, comprising:
receiving a data set processing request aiming at a service container deployed on a physical machine, wherein the data set processing request comprises an identifier of a data set used by the service container and a storage directory of the service container mounted in the local of the physical machine;
in response to the data set processing request, performing processing operation on the data set corresponding to the identifier in the storage directory, where the processing operation includes: and downloading the data set corresponding to the identifier into the storage directory, or deleting the data set corresponding to the identifier from the storage directory.
2. The method of claim 1, wherein after the receiving a data set processing request for the service container, the method further comprises:
determining a processing progress of the processing operation;
and returning the processing progress to the front end through message middleware and/or a webpage socket WebSocket protocol.
3. The method according to any one of claims 1-2, wherein the data set management method is applied to a distributed server, and the distributed server comprises a request receiving module and a data set processing module;
the processing operation on the data set corresponding to the identifier in the storage directory in response to the data set processing request includes:
the request receiving module receives a data set processing request, generates a data set processing message supported by a message middleware according to the data set processing request, and pushes the data set processing message to a message queue corresponding to the physical machine in the message middleware;
and the data set processing module positioned in the physical machine acquires the data set processing message in the message queue, and processes the data set corresponding to the identifier in the storage directory according to the identifier and the storage directory in the data set processing message.
4. The method as claimed in any one of claims 2 to 3, wherein the distributed server further includes a processing progress pushing module and a Websocket server, and the returning the processing progress to the front end through the message middleware and the Websocket protocol includes:
the processing progress pushing module generates a processing progress message according to the processing progress and pushes the processing progress message to a message middleware;
and the Websocket server acquires the processing progress message from the message middleware and sends the processing progress in the processing progress message to a front Websocket client.
5. The method according to any one of claims 1-4, wherein after said downloading the data set corresponding to the identifier into the storage directory, the method further comprises:
receiving a service processing request aiming at the service container;
and responding to the business processing request, and utilizing the data set in the storage directory to perform business processing operation.
6. The method of any of claims 1-5, further comprising:
and deleting the data set in the storage directory in response to the closing of the service container.
7. The method according to any one of claims 1-6, wherein downloading the data set corresponding to the identifier into the storage directory comprises:
and downloading the data set corresponding to the identification to the service container from a storage directory of a database or other containers, and mounting the service container in the local storage directory of the physical machine.
8. A data set management system, comprising:
the request receiving module is used for receiving a data set processing request aiming at a service container deployed on a physical machine, wherein the data set processing request comprises an identifier of a data set used by the service container and a storage directory of the service container mounted in the local of the physical machine;
a data set processing module, configured to, in response to the data set processing request, perform a processing operation on the data set corresponding to the identifier in the storage directory, where the processing operation includes: and downloading the data set corresponding to the identifier into the storage directory, or deleting the data set corresponding to the identifier from the storage directory.
9. A data set management apparatus, comprising:
a request receiving unit, configured to receive a data set processing request for a service container deployed on a physical machine, where the data set processing request includes an identifier of a data set used by the service container, and a storage directory in which the service container is mounted in a local area of the physical machine;
a data set processing unit, configured to, in response to the data set processing request, perform a processing operation on the data set corresponding to the identifier in the storage directory, where the processing operation includes: and downloading the data set corresponding to the identifier into the storage directory, or deleting the data set corresponding to the identifier from the storage directory.
10. An electronic device, comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to invoke the memory-stored instructions to perform the method of any of claims 1 to 7.
11. A computer readable storage medium having computer program instructions stored thereon, which when executed by a processor implement the method of any one of claims 1 to 7.
CN202110113181.9A 2021-01-27 2021-01-27 Data set management method, system, device, electronic equipment and storage medium Pending CN112925788A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110113181.9A CN112925788A (en) 2021-01-27 2021-01-27 Data set management method, system, device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110113181.9A CN112925788A (en) 2021-01-27 2021-01-27 Data set management method, system, device, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN112925788A true CN112925788A (en) 2021-06-08

Family

ID=76167257

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110113181.9A Pending CN112925788A (en) 2021-01-27 2021-01-27 Data set management method, system, device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN112925788A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114827017A (en) * 2022-03-31 2022-07-29 北京声智科技有限公司 Kafka cluster communication method and device, electronic equipment and storage medium
CN116028247A (en) * 2023-03-27 2023-04-28 中新宽维传媒科技有限公司 File acquisition system based on message middleware

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114827017A (en) * 2022-03-31 2022-07-29 北京声智科技有限公司 Kafka cluster communication method and device, electronic equipment and storage medium
CN114827017B (en) * 2022-03-31 2024-01-30 北京声智科技有限公司 Communication method and device of Kafka cluster, electronic equipment and storage medium
CN116028247A (en) * 2023-03-27 2023-04-28 中新宽维传媒科技有限公司 File acquisition system based on message middleware

Similar Documents

Publication Publication Date Title
CN114629865B (en) Instant messaging system for interacting with dynamically expanding applications
CN110990075B (en) Method, device, equipment and storage medium for starting fast application
CN113259161B (en) Application management method and device, electronic equipment and storage medium
CN113377465A (en) Service-based SDK calling method and device, electronic equipment and storage medium
CN111790148B (en) Information interaction method and device in game scene and computer readable medium
CN113542902B (en) Video processing method and device, electronic equipment and storage medium
CN114217707B (en) Sharing method, sharing device, electronic equipment and storage medium
CN106991018B (en) Interface skin changing method and device
WO2015176352A1 (en) Android system-based method and device for information exchange between applications
CN112925788A (en) Data set management method, system, device, electronic equipment and storage medium
CN111596991A (en) Interactive operation execution method and device and electronic equipment
CN112420217A (en) Message pushing method, device, equipment and storage medium
CN111459364A (en) Icon updating method and device and electronic equipment
CN113177093A (en) Data synchronization method and device, electronic equipment and storage medium
CN114417782A (en) Display method and device and electronic equipment
CN113312103A (en) Software definition method and device for intelligent camera, electronic equipment and storage medium
CN115113787B (en) Message processing method, device, equipment and medium
CN116089099A (en) Communication method, device, electronic equipment and storage medium
CN115576458A (en) Application window display method, device, equipment and medium
JP6441385B2 (en) Information input method, apparatus, program, and recording medium
CN113691526A (en) Pushing method, device, system, equipment and medium based on WebRTC
CN114265714A (en) Drive control method and device based on cloud mobile phone and storage medium
CN112966147A (en) File access method and device, electronic equipment and storage medium
EP4328726A1 (en) Video generation method and apparatus, and electronic device and storage medium
CN111158824A (en) Interface display method, device, equipment and medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination