CN116061189B - Robot operation data processing system, method, device, equipment and medium - Google Patents
Robot operation data processing system, method, device, equipment and medium Download PDFInfo
- Publication number
- CN116061189B CN116061189B CN202310214848.3A CN202310214848A CN116061189B CN 116061189 B CN116061189 B CN 116061189B CN 202310214848 A CN202310214848 A CN 202310214848A CN 116061189 B CN116061189 B CN 116061189B
- Authority
- CN
- China
- Prior art keywords
- data
- robot
- robot operation
- operation data
- original
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1602—Programme controls characterised by the control system, structure, architecture
- B25J9/161—Hardware, e.g. neural networks, fuzzy logic, interfaces, processor
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1628—Programme controls characterised by the control loop
- B25J9/163—Programme controls characterised by the control loop learning, adaptive, model based, rule based expert control
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
- B25J9/1661—Programme controls characterised by programming, planning systems for manipulators characterised by task planning, object-oriented languages
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Landscapes
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Automation & Control Theory (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Numerical Control (AREA)
Abstract
The application discloses a robot operation data processing system, a method, a device, equipment and a medium, which are applied to the technical field of robots and specifically comprise the following steps: acquiring historical robot operation data of each first robot in a first time period and marking the historical robot operation data; training the initial data cleaning model based on historical robot operation data of each first robot in a first time period and labeling robot operation data of the historical robot operation data to obtain a target data cleaning model; based on the target data cleaning model, data cleaning is carried out on the original robot operation data of each second robot in the second time period to obtain the standard robot operation data of each second robot in the second time period, so that big data analysis and accurate data cleaning are realized by combining a machine learning technology, and further data cleaning efficiency and accuracy are improved.
Description
Technical Field
The present disclosure relates to the field of robots, and in particular, to a system, a method, an apparatus, a device, and a medium for processing robot job data.
Background
At present, intelligent automation has penetrated into aspects of daily life and industrial production, robots replace manpower to engage in heavy work in many occasions, such as industrial robots like live working robots and household robots like sweeping robots, thereby bringing great convenience to aspects of daily life, industrial production and the like, and improving quality and efficiency of daily life and industrial production.
As robots continue to operate, the scale of robot operation data becomes huge, and the overlapping of the data volume of the robot operation data causes that a large amount of non-clean data such as repeated data, useless data, abnormal data, missing data, inconsistent data and the like exist in the robot operation data, so that inconvenience is brought to the application of the robot operation data, and therefore cleaning the robot operation data is a vital step in the technical field of robots, however, the existing data cleaning mode mostly adopts a manual mode, so that the data cleaning efficiency is lower and the effect is poor.
Disclosure of Invention
The application provides a robot operation data processing system, a method, a device, equipment and a medium, and specifically, the technical scheme provided by the application is as follows:
In one aspect, the application provides a robot job data processing system, including a data acquisition system, a model training system, and a data cleaning system; the data acquisition system is respectively in communication connection with the model training system and the data cleaning system, and the model training system and the data cleaning system are in communication connection;
the data acquisition system is used for acquiring historical robot operation data of each first robot in a first time period and labeling robot operation data of the historical robot operation data;
the model training system is used for training the initial data cleaning model based on the historical robot operation data of each first robot in the first time period and the labeling robot operation data of the historical robot operation data to obtain a target data cleaning model;
and the data cleaning system is used for cleaning the data of the original robot operation data of each second robot in the second time period based on the target data cleaning model to obtain the standard robot operation data of each second robot in the second time period.
In another aspect, the present application provides a method for processing robot job data, including:
acquiring historical robot operation data of each first robot in a first time period and marking the historical robot operation data;
Training the initial data cleaning model based on historical robot operation data of each first robot in a first time period and labeling robot operation data of the historical robot operation data to obtain a target data cleaning model;
and based on the target data cleaning model, performing data cleaning on the original robot operation data of each second robot in the second time period to obtain standard robot operation data of each second robot in the second time period.
On the other hand, the application also provides a robot job data processing device, which comprises:
a data acquisition unit for acquiring historical robot operation data of each first robot in a first time period and labeling robot operation data of the historical robot operation data;
the model training unit is used for training the initial data cleaning model based on the historical robot operation data of each first robot in the first time period and the labeling robot operation data of the historical robot operation data to obtain a target data cleaning model;
and the model application unit is used for carrying out data cleaning on the original robot operation data of each second robot in the second time period based on the target data cleaning model to obtain the standard robot operation data of each second robot in the second time period.
On the other hand, the application also provides electronic equipment, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the computer program to realize the robot job data processing method.
In another aspect, the present application also provides a computer readable storage medium storing computer instructions that when executed by a processor implement the above-described robot job data processing method.
The beneficial effects of this application are as follows:
according to the robot operation data processing method and device, the robot operation data of different robots are used as different data sources to be fused together and combined with the machine learning technology, so that big data analysis and accurate data cleaning can be achieved, and the data cleaning efficiency and accuracy can be improved.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application. The objectives and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:
FIG. 1 is a schematic diagram of a robot job data processing system according to an embodiment of the present application;
FIG. 2 is a flow chart of a method for processing robot job data according to an embodiment of the present disclosure;
FIG. 3 is another flow chart of a method for processing robot job data according to an embodiment of the present disclosure;
FIG. 4 is a schematic diagram of a data cleansing process based on an initial data cleansing model according to an embodiment of the present application;
FIG. 5 is a schematic diagram of a data cleansing process based on a target data cleansing model according to an embodiment of the present application;
FIG. 6 is a schematic functional structure of a robot job data processing device according to an embodiment of the present application;
fig. 7 is a schematic diagram of a hardware structure of an electronic device in an embodiment of the application.
Detailed Description
In order to make the objects, technical solutions and advantageous effects of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments, but not all embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
In order to facilitate a better understanding of the present application, technical terms related to the present application will be briefly described below.
Machine learning is a method of designing, learning, adjusting, and optimizing a system based on sample data. According to the method and the device, machine learning can be conducted on the robot operation data cleaning scheme based on historical robot operation data of each robot and marking robot operation data of the historical robot operation data, and the robot operation data cleaning scheme of the initial data cleaning model is optimized and adjusted in the machine learning process so as to obtain the target data cleaning model with higher performance and accuracy.
Data mining is a process of algorithmically searching for information hidden in a large amount of data. The present application may search for a robot job data cleaning solution hidden therein from a large amount of historical robot job data and labeling robot job data during a machine learning process.
A distributed file system is a file system that stores large amounts of data on multiple devices to reduce the cost and complexity of storing large amounts of data. The method and the device can store historical robot operation data and marking robot operation data used in a machine learning process and cleaned standard robot operation data in a data cleaning process through a distributed file system.
ETL (Extract Transform Load ) is the process of extracting, transforming, and loading data from a source to a data warehouse. The application can convert the original robot operation data into standard robot operation data suitable for use by extracting the original robot operation data and cleaning and/or enriching rules, and load the standard robot operation data into a mysql, redis, mongo and other suitable databases for use by a data application system.
Flink, a distributed processing engine that operates in a clustered environment to perform stateful computation on unbounded and bounded data streams. The robot operation data can be subjected to machine learning, data mining, data cleaning, data storage and the like through the Flink.
Kafka is a distributed log system for web/nginx logs, access logs, message services, etc. based on zookeeper coordination, and may also be used as an MQ (Message Queue) system. The application can carry out data transmission, log recording and the like of robot operation data through Kafka.
HDFS is a distributed file system with high fault tolerance and high throughput. The method and the device can buffer and reuse the cleaned standard robot operation data through the HDFS.
It should be noted that references to "first," "second," etc. in this application are for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that such terms are interchangeable under appropriate circumstances such that the embodiments described herein are capable of operation in other sequences than those illustrated or otherwise described herein. Furthermore, references to "and/or" in this application describe association relationships of associated objects, meaning that there may be three relationships, e.g., a and/or B, may represent: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.
After technical terms related to the present application are introduced, a detailed description is given next to the technical solution provided in the present application.
The embodiment of the application provides a robot job data processing system, referring to fig. 1, a robot job data processing system 100 provided in the embodiment of the application at least includes a data acquisition system 101, a model training system 102 and a data cleaning system 103; the data acquisition system 101 is respectively in communication connection with the model training system 102 and the data cleaning system 103, and the model training system 102 and the data cleaning system 103 are in communication connection;
A data acquisition system 101 for acquiring historical robot operation data of each first robot in a first period and labeling robot operation data of the historical robot operation data;
the model training system 102 is configured to train the initial data cleaning model based on historical robot operation data of each first robot in a first period and labeling robot operation data of the historical robot operation data to obtain a target data cleaning model;
and the data cleaning system 103 is used for cleaning the data of the original robot operation data of each second robot in the second time period based on the target data cleaning model to obtain the standard robot operation data of each second robot in the second time period.
In one possible implementation, the robotic job data processing system 100 provided in the embodiments of the present application further includes a data storage system 104; the data storage system 104 is in communication with the data cleansing system 103;
the data cleaning system 103 is further configured to send standard robot operation data of each second robot in the second time period to the data storage system;
and the data storage system 104 is used for storing the standard robot operation data of each second robot in the second time period into the target database according to a standard data format.
In one possible implementation, the robotic job data processing system 100 provided in the embodiments of the present application further includes a data storage system 104 and a data verification system 105; the data verification system 105 is respectively in communication with the data cleaning system 103 and the data storage system 104;
the data cleaning system 103 is further configured to send standard robot operation data of each second robot in the second time period to the data verification system 105;
a data verification system 105, configured to verify whether the standard robot operation data of each second robot in the second time period meets the standard condition; when the standard robot operation data of each second robot in the second time period is determined to meet the standard conditions, the standard robot operation data of each second robot in the second time period is sent to the data storage system 104; when the standard robot operation data of each second robot in the second time period is determined to be not in accordance with the standard conditions, determining the standard robot operation data of each second robot in the second time period as original robot operation data, and sending the original robot operation data to the data cleaning system 103;
and the data storage system 104 is used for storing the standard robot operation data of each second robot in the second time period into the target database according to a standard data format.
In one possible implementation manner, the data cleaning system 103 is configured to analyze the raw robot operation data based on the target data cleaning model, determine a data cleaning scheme of the raw robot operation data based on the data type of the raw robot operation data after obtaining the data type of the raw robot operation data, perform quality detection on the raw robot operation data, obtain unclean data of the raw robot operation data, then locate the unclean data of the raw robot operation data to obtain the unclean data type of the raw robot operation data, and perform data cleaning on the raw robot operation data to obtain standard robot operation data of the raw robot operation data based on the data cleaning scheme corresponding to the unclean data type in the data cleaning scheme of the raw robot operation data.
Based on the robot job data processing system 100 provided in the embodiment of the present application, the embodiment of the present application provides a method for processing robot job data, as shown in fig. 2, the historical robot job data of each first robot in the first period of time and the original robot job data of each second robot in the second period of time may all be accessed through MQTT (Message Queuing Telemetry Transport, message queue telemetry transmission) and other technologies and processed through basic devices such as Flink, kafka and the like, and then after being processed through the robot job data processing system 100, the historical robot job data of each first robot in the first period of time and the original robot job data of each second robot in the second period of time are stored in HDFS and the like for cache reuse or mysql, redis, mongo and the like for data application. Specifically, referring to fig. 3, an overview flow of the method for processing robot job data provided in the embodiment of the present application is as follows:
Step 301: and acquiring historical robot operation data of each first robot in a first time period and labeling robot operation data of the historical robot operation data.
In a specific implementation, in one possible implementation manner, the original robot operation data of each first robot in the first time period may be used as historical robot operation data, and a labeling manner such as manual labeling and/or machine labeling is adopted to label a data cleaning scheme of non-clean data such as repeated data, useless data and abnormal data in the historical robot operation data of each first robot in the first time period, so as to obtain labeled robot operation data corresponding to the historical robot operation data of each first robot in the first time period. In another possible embodiment, the standard robot operation data of each first robot in the first time period may be obtained as the labeling robot operation data, and the non-clean data such as the repetition data, the useless data, the abnormal data and the like may be randomly added to the standard robot operation data of each first robot in the first time period, so as to obtain the historical robot operation data of each first robot in the first time period.
Step 302: and training the initial data cleaning model based on the historical robot operation data of each first robot in the first time period and the labeling robot operation data of the historical robot operation data to obtain a target data cleaning model.
In specific implementation, the historical robot operation data and the labeling robot operation data of the historical robot operation data of each first robot in the first time period are used as training sample sets, iterative training is carried out on the initial data cleaning model based on the training sample sets until the iteration termination condition is met (for example, the iteration training times reach the set times, and if the current loss value is not greater than the set threshold value, and the like), and a target data cleaning model is obtained based on each model parameter of the initial data cleaning model obtained in the last iteration training; wherein each iterative training comprises: selecting a target training sample from the training sample set, inputting historical robot operation data in the target training sample into an initial data cleaning model to obtain cleaned target robot operation data, obtaining a current loss value based on the target robot operation data and labeling robot operation data in the target training sample, and updating each model parameter of the initial data cleaning model based on the current loss value. Specifically, in each iterative training process, referring to fig. 4, the data cleaning process based on the initial data cleaning model includes, but is not limited to, the following steps:
Step one, a preparation step, including but not limited to demand analysis, big data category analysis, task definition, small category method definition, basic setting, obtaining a data cleaning scheme based on the above steps, and the like; the requirement analysis can identify the data cleaning requirement of the robot operation data; the big data category analysis can classify big data by analyzing robot category, robot use scene, robot job type, robot job task, robot job subtask, etc. in order to analyze the same kind of data; the task definition may determine a data cleansing task goal; the small-class method defines a data cleaning scheme capable of determining adaptation of various types of non-cleaning data; the basic configuration can complete the configuration of data interfaces and the like; based on the above steps, a complete data cleansing scheme can be obtained and archived.
Step two, the detection step comprises, but is not limited to, preprocessing data between the historical robot operation data and the historical robot operation data, detecting similar and/or repeated data, detecting incomplete data and/or logic, detecting error data and/or logic, detecting redundancy and/or abnormal data and the like, and carrying out statistics on detection results to obtain data quality information and tidying and archiving.
And step three, a positioning step, which comprises but is not limited to carrying out data quality positioning on historical robot operation data based on data quality information to obtain unclean data positioning information, analyzing unclean data and influence of the unclean data on knowledge representation after data tracking and data quality evaluation, analyzing root causes of the unclean data, determining positions and types of the unclean data and a data cleaning scheme corresponding to the types, and sorting and archiving. It should be noted that, in practical applications, it may also be necessary to return to the detection step to further locate the data location to be cleaned according to the location analysis condition.
And step four, a correction step, which includes but is not limited to performing data cleaning on the unclean data based on the type, the position and the data cleaning scheme of the unclean data, for example performing data cleaning operations such as unclean data marking, incomplete data filling, unavailable data deleting, error data modifying, repeated data merging, redundant data deleting and the like on the unclean data to obtain cleaned target robot operation data, and performing storage management.
And fifthly, a verification step, including but not limited to comparing and verifying the cleaned target robot operation data with the task-defined data cleaning task target, if the cleaned target robot operation data does not accord with the data cleaning task target, performing further positioning analysis and data cleaning, and even returning to the preparation step to adjust the corresponding preparation operation.
Step 303: and based on the target data cleaning model, performing data cleaning on the original robot operation data of each second robot in the second time period to obtain standard robot operation data of each second robot in the second time period.
In this embodiment of the present application, the data cleaning may be performed on the original robot job data of each second robot in the second time period based on the target data cleaning model, but is not limited to the following steps:
firstly, analyzing the original robot operation data to obtain the data type of the original robot operation data, and then determining a data cleaning scheme of the original robot operation data based on the data type of the original robot operation data.
And then, performing quality detection on the original robot operation data to obtain unclean data of the original robot operation data, and positioning the unclean data of the original robot operation data to obtain unclean data types of the original robot operation data.
And finally, based on a data cleaning scheme corresponding to the non-cleaning data type in the data cleaning scheme of the original robot operation data, performing data cleaning on the original robot operation data to obtain standard robot operation data of the original robot operation data.
In particular implementations, the data cleansing process based on the target data cleansing model includes, but is not limited to, the following steps:
step one, preprocessing, including but not limited to obtaining description information such as field interpretation, data sources, code tables and the like from metadata of original robot operation data, determining a data type of the original robot operation data based on the description information, and then determining a data cleaning scheme of the original robot operation data based on the data type of the original robot operation data; performing similar and/or repeated data check, incomplete data and/or logic check, error data and/or logic check, redundancy and/or abnormal data check and the like on the original robot operation data based on the data check rule to obtain data quality information of the original robot operation data; and based on the data quality information, carrying out data quality positioning and unclean data generation reason analysis on the original robot operation data to obtain the position and type of unclean data in the original robot operation data and a data cleaning scheme corresponding to the type. In practical application, the data checking rules include, but are not limited to, non-empty checking, primary key repetition checking, illegal checking (ETL can be adopted), data format checking, record number checking and the like; the non-empty checking is to check the data in the field under the condition that the required field is non-empty; the main key is repeated to check the uniqueness of the main key when the same kind of data in a plurality of robots are uniformly stored after being cleaned; the illegal checking comprises illegal code checking and illegal value checking, the illegal code cleaning comprises checking illegal codes and codes which do not accord with the data standard, and the illegal value cleaning comprises checking data such as value error, format error, redundant field symbol, messy code and the like; the data format checking is to check whether the formats of various attribute values such as time, messy codes, redundant characters and the like are correct or not; the record number checking is to check the total number of data among the related data of each robot or the fluctuation of the daily data quantity.
Step two, a data cleaning step, which includes but is not limited to a data cleaning step based on the position and type of non-cleaning data in the original robot operation data, such as missing value cleaning, format content cleaning, logic error cleaning and the like of a data cleaning scheme corresponding to the type; wherein:
the missing value cleaning is a process of filling and complementing data for re-acquiring lost important data through operations such as removing weight, removing dryness, missing value processing and the like, including but not limited to the following cases: 1) Determining a missing value range, namely respectively calculating a missing value proportion for each field of the original robot operation data, and filling and completing according to the missing value proportion and missing fields (for example, fields according to id, name and the like); 2) Removing unnecessary fields, namely directly deleting the unnecessary fields, wherein the step needs to be backed up once every time of execution, or processing the whole data after the small-scale data is successfully processed, so as to avoid the loss caused by deleting the data by mistake; 3) Filling missing content, namely, filling missing values can be estimated according to business knowledge or experience, the missing values can be filled according to calculation results (such as equal, median, mode and the like) of the same index, and the missing values can be filled according to calculation results of different indexes; 4) Counting the occurrence number of each element when the data quantity is large, and repeatedly selecting when the data with the occurrence number more than 1 is selected;
Format content cleansing, which is a process of cleansing a data format, is as small as time, date, value, as large as whether characters (such as null, blank, etc.) which are not present or not in the value content exist, and the like, and can be conducted, including but not limited to the following cases: 1) Data with inconsistent display formats such as time, date, numerical value and the like, which are related to different data sources accessed by the MQ system, can be subjected to the data format problem when the robot operation data of multiple data sources are integrated, for example, the received text format data can be converted into the needed JSON format data; 2) The data content has unnecessary characters (such as null, blank, etc.), and the data may only include a part of the content, for example, the device id of the robot is a number+letter, the received device id has blank, chinese son, etc., the device id of the robot is a Chinese son, the received device id has the content of a number, letter, etc., and the data needs to be deleted after the unnecessary characters such as null, blank, etc. are positioned; 3) The data content has characters which are not matched with the content of the field, such as gender in the name field, job number in the job type field, and the like, and the data cannot be simply cleaned by deleting, because the cause is possible to be manual filling errors, the front-end equipment is not checked, and the problem that columns are not aligned when the data is imported is also possible to be partially or completely existed, so that the corresponding cleaning treatment is needed after the problem type is identified in detail.
Logical error cleaning, which is a process of cleaning data such as duplicate data, abnormal data which does not conform to common sense, attribute-dependent conflict error data, and the like, includes, but is not limited to, performing corresponding data cleaning and conversion operations according to different forms of different non-clean data. It should be noted that when the original robot operation data is subjected to data cleaning, the robot operation data should be backed up so as to prevent the cleaning operation from being required to be canceled. Moreover, to facilitate handling data quality issues with single, multiple, and single and other data sources, it is generally necessary to perform data conversion operations on each data source, and when values are extracted (split) from the attribute fields of the original data source, the attributes of the original data source may contain a lot of information, which sometimes needs to be refined into multiple attributes, facilitating subsequent cleaning of duplicate records. In addition, input and spelling errors are confirmed and corrected, and this step is automated as much as possible, which is advantageous if spelling errors are queried based on a dictionary. In addition, to facilitate record data matching and merging, attribute values should be converted into a uniform format.
Further, in the embodiment of the present application, after data cleaning is performed on the original robot operation data of each second robot in the second time period to obtain the standard robot operation data of each second robot in the second time period, in a possible implementation manner, the standard robot operation data of each second robot in the second time period may be directly stored in the target database according to the standard data format; in another possible implementation manner, whether the standard robot operation data of each second robot in the second time period accords with the standard condition or not (for example, format verification, data relevance verification and the like) can be verified, when the standard robot operation data of each second robot in the second time period does not accord with the standard condition, the standard robot operation data of each second robot in the second time period is determined to be the original robot operation data, data cleaning is carried out again, when the standard robot operation data of each second robot in the second time period accords with the standard condition, the standard robot operation data of each second robot in the second time period is determined, the standard robot operation data of each second robot in the second time period is stored in the target database, so that the quality of the robot operation data can be improved, repeated cleaning processing is avoided after the same robot operation data are extracted again, and the standard robot operation data stored in the target database can be guaranteed to have accuracy, consistency, completeness, effectiveness and uniqueness through data cleaning of the target data cleaning model, so that the availability and stability of the standard robot operation data can be guaranteed.
In this embodiment, referring to fig. 5, the original robot operation data is received through the MQ system, the position and type of the unclean data in the original robot operation data are obtained by analyzing the original robot operation data through the target data cleaning model, after the data cleaning scheme corresponding to the type of the unclean data in the original robot operation data is further determined, when the original robot operation data is determined to be unnecessary for data cleaning, the original robot operation data is directly determined to be the standard robot operation data, when the original robot operation data is determined to be required for data cleaning and the original robot operation data is not dirty data (i.e., unnecessary data), the data cleaning scheme corresponding to the type of the unclean data is based on the position of the unclean data in the original robot operation data, the data cleaning is performed on the original robot operation data to obtain the standard robot operation data, and when the original robot operation data is determined to be dirty data (i.e., unnecessary data), the original robot operation data is directly filtered and removed, so that the clean standard robot operation data can be obtained and the usability and stability of the standard robot can be improved.
Based on the foregoing embodiments, the present embodiment further provides a robotic job data processing device, and referring to fig. 6, the robotic job data processing device 600 provided in the embodiment of the present application at least includes:
a data acquisition unit 601, configured to acquire historical robot operation data of each first robot in a first period and labeling robot operation data of the historical robot operation data;
the model training unit 602 is configured to train the initial data cleaning model based on the historical robot operation data of each first robot in the first period and the labeling robot operation data of the historical robot operation data to obtain a target data cleaning model;
the model application unit 603 is configured to perform data cleaning on the original robot operation data of each second robot in the second time period based on the target data cleaning model, so as to obtain standard robot operation data of each second robot in the second time period.
In one possible implementation manner, the robotic job data processing device 600 provided in the embodiments of the present application further includes:
a data storage unit 604, configured to store standard robot job data of each second robot in the second time period to the target database according to a standard data format;
In one possible implementation manner, the robotic job data processing device 600 provided in the embodiments of the present application further includes:
a data verification unit 605 for verifying whether the standard robot job data of each second robot in the second period meets the standard conditions; when the standard robot operation data of each second robot in the second time period meets the standard conditions, storing the standard robot operation data of each second robot in the second time period into a target database; and when the standard robot operation data of each second robot in the second time period is determined to be not in accordance with the standard conditions, determining the standard robot operation data of each second robot in the second time period as the original robot operation data, and carrying out data cleaning again.
In one possible implementation manner, based on the target data cleaning model, when performing data cleaning on the original robot operation data of each second robot in the second time period to obtain the standard robot operation data of each second robot in the second time period, the model application unit 603 is specifically configured to:
analyzing the original robot operation data based on the target data cleaning model, and determining a data cleaning scheme of the original robot operation data based on the data type of the original robot operation data after obtaining the data type of the original robot operation data;
Performing quality detection on the original robot operation data to obtain unclean data of the original robot operation data, and then positioning the unclean data of the original robot operation data to obtain unclean data types of the original robot operation data;
and carrying out data cleaning on the original robot operation data based on a data cleaning scheme corresponding to the non-cleaning data type in the data cleaning scheme of the original robot operation data to obtain standard robot operation data of the original robot operation data.
It should be noted that, the principle of solving the technical problem of the robotic job data processing device 600 provided in the embodiment of the present application is similar to that of the robotic job data processing method provided in the embodiment of the present application, so that the implementation of the robotic job data processing device 600 provided in the embodiment of the present application can refer to the implementation of the robotic job data processing method provided in the embodiment of the present application, and the repetition is omitted.
After the system, the method and the device for processing the robot job data provided by the embodiment of the application are introduced, the electronic equipment provided by the embodiment of the application is briefly introduced.
Referring to fig. 7, an electronic device 700 provided in the embodiment of the present application at least includes a processor 701, a memory 702, and a computer program stored in the memory 702 and capable of running on the processor 701, where the processor 701 implements the method for processing robot job data provided in the embodiment of the present application when executing the computer program.
The electronic device 700 provided by the embodiments of the present application may also include a bus 703 that connects the different components, including the processor 701 and the memory 702. Bus 703 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and so forth.
The Memory 702 may include readable media in the form of volatile Memory, such as RAM (Random Access Memory ) 7021 and/or cache Memory 7022, and may further include ROM (Read Only Memory) 7023. The memory 702 may also include a program tool 7025 having a set (at least one) of program modules 7024, the program modules 7024 including, but not limited to: an operating subsystem, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
The electronic device 700 may also communicate with one or more external devices 704 (e.g., keyboard, remote control, etc.), with one or more devices that enable a user to interact with the electronic device 700 (e.g., cell phone, computer, etc.), and/or with any device that enables the electronic device 700 to communicate with one or more other electronic devices 700 (e.g., router, modem, etc.). Such communication may occur through an I/O (Input/Output) interface 705. Also, the electronic device 700 may communicate with one or more networks (e.g., LAN (Local Area Network, local area network), WAN (Wide Area Network ) and/or public network, such as the Internet) through the network adapter 706. As shown in fig. 7, the network adapter 706 communicates with other modules of the electronic device 700 via the bus 703. It should be appreciated that although not shown in fig. 7, other hardware and/or software modules may be used in connection with electronic device 700, including, but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (Redundant Arrays of Independent Disks, disk array) subsystems, tape drives, data backup storage subsystems, and the like.
It should be noted that, the electronic device 700 shown in fig. 7 is only an example, and should not impose any limitation on the functions and application scope of the embodiments of the present application.
In addition, the embodiment of the application also provides a computer readable storage medium, and the computer readable storage medium stores computer instructions which are executed by a processor to realize the robot job data processing method provided by the embodiment of the application. Specifically, the computer instruction may be built into or installed in the electronic device, so that the electronic device may implement the robot job data processing method provided in the embodiment of the present application by executing the built-in or installed computer instruction.
In addition, the method for processing robot job data provided in the embodiments of the present application may also be implemented as a program product, where the program product includes program code, and when the program code runs on a processor, the method for processing robot job data provided in the embodiments of the present application is implemented.
The program product provided by the embodiments of the present application may employ any combination of one or more readable media, where the readable media may be a readable signal medium or a readable storage medium, and the readable storage medium may be, but is not limited to, a system, apparatus, or device that is an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination of the above, and more specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, RAM, ROM, EPROM (Erasable Programmable Read Only Memory, erasable programmable read-Only Memory), an optical fiber, a CD-ROM (Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The program product provided by the embodiment of the application can adopt a CD-ROM and comprises program codes, and can also run on the electronic device. However, the program product provided by the embodiments of the present application is not limited thereto, and in the embodiments of the present application, the readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such a division is merely exemplary and not mandatory. Indeed, the features and functions of two or more of the elements described above may be embodied in one element in accordance with embodiments of the present application. Conversely, the features and functions of one unit described above may be further divided into a plurality of units to be embodied.
Furthermore, although the operations of the methods of the present application are depicted in the drawings in a particular order, this is not required to or suggested that these operations must be performed in this particular order or that all of the illustrated operations must be performed in order to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform.
While preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the application.
It will be apparent to those skilled in the art that various modifications and variations can be made to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if such modifications and variations of the embodiments of the present application fall within the scope of the claims and the equivalents thereof, the present application is intended to encompass such modifications and variations.
Claims (8)
1. The robot operation data processing system is characterized by comprising a data acquisition system, a model training system and a data cleaning system; the data acquisition system is respectively in communication connection with the model training system and the data cleaning system, and the model training system is in communication connection with the data cleaning system;
the data acquisition system is used for acquiring historical robot operation data of each first robot in a first time period and labeling robot operation data of the historical robot operation data;
The model training system is used for training an initial data cleaning model based on the historical robot operation data of each first robot in the first time period and the labeling robot operation data of the historical robot operation data to obtain a target data cleaning model;
the data cleaning system is used for cleaning data of original robot operation data of each second robot in a second time period based on the target data cleaning model to obtain standard robot operation data of each second robot in the second time period;
the data cleaning system is specifically configured to analyze the original robot operation data based on the target data cleaning model to obtain a data type of the original robot operation data, determine a data cleaning scheme of the original robot operation data based on the data type of the original robot operation data, perform quality detection on the original robot operation data to obtain unclean data of the original robot operation data, then position the unclean data of the original robot operation data to obtain an unclean data type of the original robot operation data, and perform data cleaning on the original robot operation data to obtain standard robot operation data of the original robot operation data based on the data cleaning scheme corresponding to the unclean data type in the data cleaning scheme of the original robot operation data.
2. The robotic job data processing system according to claim 1, further comprising a data storage system; the data storage system is in communication connection with the data cleaning system;
the data cleaning system is further used for sending standard robot operation data of each second robot in a second time period to the data storage system;
and the data storage system is used for storing the standard robot operation data of each second robot in the second time period into a target database according to a standard data format.
3. The robotic job data processing system according to claim 1, further comprising a data verification system and a data storage system; the data verification system is respectively in communication connection with the data cleaning system and the data storage system;
the data cleaning system is further used for sending standard robot operation data of each second robot in a second time period to the data verification system;
the data verification system is used for verifying whether the standard robot operation data of each second robot in the second time period meets standard conditions or not; when the standard robot operation data of each second robot in the second time period accords with the standard conditions, the standard robot operation data of each second robot in the second time period is sent to the data storage system; when the standard robot operation data of each second robot in the second time period is determined to be not in accordance with the standard conditions, determining the standard robot operation data of each second robot in the second time period as original robot operation data, and sending the original robot operation data to the data cleaning system;
And the data storage system is used for storing the standard robot operation data of each second robot in the second time period into a target database according to a standard data format.
4. A robot job data processing method, comprising:
acquiring historical robot operation data of each first robot in a first time period and labeling robot operation data of the historical robot operation data;
training an initial data cleaning model based on the historical robot operation data of each first robot in the first time period and the labeling robot operation data of the historical robot operation data to obtain a target data cleaning model;
performing data cleaning on the original robot operation data of each second robot in the second time period based on the target data cleaning model to obtain standard robot operation data of each second robot in the second time period;
based on the target data cleaning model, performing data cleaning on original robot operation data of each second robot in a second time period to obtain standard robot operation data of each second robot in the second time period, wherein the method comprises the following steps:
Analyzing the original robot operation data based on the target data cleaning model, and determining a data cleaning scheme of the original robot operation data based on the data type of the original robot operation data after obtaining the data type of the original robot operation data;
performing quality detection on the original robot operation data to obtain unclean data of the original robot operation data, and then positioning the unclean data of the original robot operation data to obtain unclean data types of the original robot operation data;
and carrying out data cleaning on the original robot operation data based on a data cleaning scheme corresponding to the unclean data type in the data cleaning scheme of the original robot operation data to obtain standard robot operation data of the original robot operation data.
5. The method for processing robot job data according to claim 4, wherein performing data cleaning on the original robot job data of each second robot in the second time period to obtain standard robot job data of each second robot in the second time period, further comprises:
Storing the standard robot operation data of each second robot in a second time period into a target database according to a standard data format;
or alternatively;
verifying whether the standard robot operation data of each second robot in the second time period meets standard conditions or not; when the standard robot operation data of each second robot in the second time period accords with the standard conditions, storing the standard robot operation data of each second robot in the second time period into a target database; and when the standard robot operation data of each second robot in the second time period is determined to be not in accordance with the standard conditions, determining the standard robot operation data of each second robot in the second time period as the original robot operation data, and carrying out data cleaning again.
6. A robot job data processing apparatus, comprising:
a data acquisition unit, configured to acquire historical robot operation data of each first robot in a first period of time and labeling robot operation data of the historical robot operation data;
the model training unit is used for training the initial data cleaning model based on the historical robot operation data of each first robot in the first time period and the labeling robot operation data of the historical robot operation data to obtain a target data cleaning model;
The model application unit is used for carrying out data cleaning on the original robot operation data of each second robot in the second time period based on the target data cleaning model to obtain standard robot operation data of each second robot in the second time period;
the model application unit is specifically configured to analyze the original robot operation data based on the target data cleaning model, and determine a data cleaning scheme of the original robot operation data based on the data type of the original robot operation data after obtaining the data type of the original robot operation data; performing quality detection on the original robot operation data to obtain unclean data of the original robot operation data, and then positioning the unclean data of the original robot operation data to obtain unclean data types of the original robot operation data; and carrying out data cleaning on the original robot operation data based on a data cleaning scheme corresponding to the unclean data type in the data cleaning scheme of the original robot operation data to obtain standard robot operation data of the original robot operation data.
7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the robotic job data processing method according to any one of claims 4-5 when executing the computer program.
8. A computer-readable storage medium storing computer instructions which, when executed by a processor, implement the robotic job data processing method of any one of claims 4-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310214848.3A CN116061189B (en) | 2023-03-08 | 2023-03-08 | Robot operation data processing system, method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310214848.3A CN116061189B (en) | 2023-03-08 | 2023-03-08 | Robot operation data processing system, method, device, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116061189A CN116061189A (en) | 2023-05-05 |
| CN116061189B true CN116061189B (en) | 2023-06-27 |
Family
ID=86169922
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310214848.3A Active CN116061189B (en) | 2023-03-08 | 2023-03-08 | Robot operation data processing system, method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116061189B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021151299A1 (en) * | 2020-05-29 | 2021-08-05 | 平安科技（深圳）有限公司 | Artificial intelligence-based data enhancement method, apparatus, electronic device, and medium |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10558627B2 (en) * | 2016-04-21 | 2020-02-11 | Leantaas, Inc. | Method and system for cleansing and de-duplicating data |
| CN108734330A (en) * | 2017-04-24 | 2018-11-02 | 北京京东尚科信息技术有限公司 | Data processing method and device |
| CN111797078A (en) * | 2019-04-09 | 2020-10-20 | Oppo广东移动通信有限公司 | Data cleaning method, model training method, device, storage medium and equipment |
| CN113568899A (en) * | 2021-02-06 | 2021-10-29 | 高云 | Data optimization method based on big data and cloud server |
| US11513886B2 (en) * | 2021-03-11 | 2022-11-29 | UiPath, Inc. | System and computer-implemented method for managing robotic process automation (RPA) robots |
| CN112925785A (en) * | 2021-03-29 | 2021-06-08 | 中国建设银行股份有限公司 | Data cleaning method and device |
-
2023
- 2023-03-08 CN CN202310214848.3A patent/CN116061189B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021151299A1 (en) * | 2020-05-29 | 2021-08-05 | 平安科技（深圳）有限公司 | Artificial intelligence-based data enhancement method, apparatus, electronic device, and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116061189A (en) | 2023-05-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109684352B (en) | Data analysis system, data analysis method, storage medium, and electronic device | |
| CN111027615B (en) | Middleware fault early warning method and system based on machine learning | |
| EP3123362A1 (en) | Method and system for large scale data curation | |
| CN103593352A (en) | Method and device for cleaning mass data | |
| CN112000656A (en) | Intelligent data cleaning method and device based on metadata | |
| CN112445775B (en) | Fault analysis method, device, equipment and storage medium of photoetching machine | |
| CN113626241B (en) | Abnormality processing method, device, equipment and storage medium for application program | |
| CN112905685A (en) | Framework management and control system and equipment for informatization construction | |
| EP4134877A1 (en) | Training method and apparatus for fault recognition model, fault recognition method and apparatus, and electronic device | |
| CN114968727B (en) | Database through infrastructure fault positioning method based on artificial intelligence operation and maintenance | |
| US20130030852A1 (en) | Associative Memory-Based Project Management System | |
| CA2793400C (en) | Associative memory-based project management system | |
| CN113360521A (en) | Log query method, device, equipment and storage medium | |
| CN112181490A (en) | Method, device, equipment and medium for identifying function category in function point evaluation method | |
| CN115689334A (en) | Efficiency analysis method and system of warehouse management system and computer equipment | |
| CN111865673A (en) | Automatic fault management method, device and system | |
| CN113138906A (en) | Call chain data acquisition method, device, equipment and storage medium | |
| CN117827923A (en) | Query demand processing method and device, computer equipment and storage medium | |
| CN116061189B (en) | Robot operation data processing system, method, device, equipment and medium | |
| CN112836150A (en) | Identification method, system, equipment and medium for tracing code of medicine | |
| CN116452212A (en) | Intelligent customer service commodity knowledge base information management method and system | |
| CN115757090A (en) | Software testing system, method, device and medium | |
| CN115936748A (en) | Business big data analysis method and system | |
| CN112800219B (en) | Method and system for feeding back customer service log to return database | |
| CN117501275A (en) | Method, computer program product and computer system for analyzing data consisting of a large number of individual messages |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |