CN111531581A - Industrial robot fault action detection method and system based on vision - Google Patents
Industrial robot fault action detection method and system based on vision Download PDFInfo
- Publication number
- CN111531581A CN111531581A CN202010342989.XA CN202010342989A CN111531581A CN 111531581 A CN111531581 A CN 111531581A CN 202010342989 A CN202010342989 A CN 202010342989A CN 111531581 A CN111531581 A CN 111531581A
- Authority
- CN
- China
- Prior art keywords
- industrial robot
- image
- video frame
- frame sequence
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000009471 action Effects 0.000 title claims abstract description 69
- 238000001514 detection method Methods 0.000 title claims abstract description 37
- 238000000034 method Methods 0.000 claims abstract description 28
- 238000003709 image segmentation Methods 0.000 claims description 30
- 230000007257 malfunction Effects 0.000 claims description 12
- 239000004576 sand Substances 0.000 claims description 12
- 238000000605 extraction Methods 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 4
- 239000000284 extract Substances 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 11
- 238000012545 processing Methods 0.000 description 7
- 230000009286 beneficial effect Effects 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 238000003745 diagnosis Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000005065 mining Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 208000012661 Dyskinesia Diseases 0.000 description 1
- 208000027418 Wounds and injury Diseases 0.000 description 1
- 230000032683 aging Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000006378 damage Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 238000001125 extrusion Methods 0.000 description 1
- 208000014674 injury Diseases 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
- B25J9/1697—Vision controlled systems
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J19/00—Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators
- B25J19/0095—Means or methods for testing manipulators
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
- B25J9/1661—Programme controls characterised by programming, planning systems for manipulators characterised by task planning, object-oriented languages
Landscapes
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Manipulator (AREA)
Abstract
The invention provides a method and a system for detecting fault actions of an industrial robot based on vision, wherein the method for detecting the fault actions of the industrial robot based on the vision comprises the following steps of S1: collecting standard operation videos of the industrial robot, and establishing a standard operation mode video frame sequence of the industrial robot; s2: acquiring an industrial robot operation image in real time, and acquiring an industrial robot real-time action image; s3: matching the real-time motion image of the industrial robot with the standard operation mode video frame sequence of the industrial robot, judging whether an image approximately matched with the real-time motion image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot, if so, executing S2, and if not, executing S4; s4: and controlling the industrial robot to suddenly stop. The invention has the advantages that the sudden failure of the industrial robot body is discovered in a non-contact mode, the safety accident that the robot hurts people is avoided when the robot cooperates with the human machine, and the detection process is simple and accurate.
Description
Technical Field
The invention relates to the technical field of intelligent manufacturing, in particular to a method and a system for detecting fault actions of an industrial robot based on vision.
Background
An industrial robot is a complex system which integrates hardware such as automation, machinery, embedding, hydraulic pressure, electricity and the like and control software thereof. It can replace workers in some dangerous and complicated repetitive labor. Industrial robots have been widely used in manufacturing due to their high precision and lack of rest. However, with the large number of applications of industrial robots, an industrial robot injury event occurs at times. The main reasons for safety accidents of industrial robots are human factors and self-failure of the robots. The safety accidents caused by the self misoperation of the robot account for more than half of the safety accidents. Human factors can be controlled through enhanced management and training, and the safety problem caused by self misoperation of the robot needs to be solved through technical means. Due to various reasons such as signal interference, device aging, metal fatigue and the like, misoperation of the robot exists in a large amount in the operation process of the robot. The misoperation of the robot causes the robot to have the dyskinesia, which causes the extrusion and collision accidents, and the life safety of the nearby personnel is threatened. Especially in a human-computer cooperation scene, the safety problem of the robot is very important.
The Chinese patent with the publication number of CN106625724B discloses a safety control method for an industrial robot body facing a cloud control platform, and the method comprises the following steps of firstly, downloading safety protection logics of corresponding levels from the cloud control platform to a safety protection module according to the field condition of the industrial robot; secondly, calculating and analyzing real-time state information of each shaft and the tail end of the industrial robot through safety protection logic, and sending alarm information and controlling the robot to stop moving when an abnormal state occurs; and finally, analyzing the control command sent by the cloud control platform by using the safety protection logic, judging whether the position and the posture of the industrial robot exceed the safety protection range or not, and finally judging whether the control command is isolated or executed.
Chinese patent publication No. CN101509839 discloses a cluster industrial robot fault diagnosis method based on outlier mining, which comprises the following steps:
1) acquiring running state data of the cluster industrial robot by adopting a multi-input channel data acquisition card; the operating state data includes: the total consumed power, the vibration of the base, the power and the working current of each motor, the angular speed of a rotary joint and a task execution result;
2) sorting and classifying the obtained running state data according to a uniform format, distinguishing data sources and data types by adding data identifiers, and then transmitting the data sources and the data types to a system database for storage;
3) clustering analysis is carried out on the running state data of the clustered industrial robots, the outlier mining method is utilized to calculate the outlier factor of each industrial robot to obtain the outlier degree of each industrial robot, the outlier is separated according to the outlier degree, whether the individual industrial robot represented by the outlier breaks down or not is further determined, the specific part of the robot where the fault occurs is judged according to the types of abnormal running parameters, and a fault diagnosis result is obtained;
4) and storing information including the operating state data and the fault diagnosis result of the industrial robot into a system database, and directly displaying the data through a special display port as a basis for managing, maintaining and updating the industrial robot.
The state information of the industrial robot needs to be acquired by the data acquisition devices in the prior art, and the state information of the industrial robot acquired by the data acquisition devices is processed to judge whether the state of the industrial robot is abnormal or not, so that the detection process is complex and the cost is high.
Disclosure of Invention
The invention aims to overcome the defects in the prior art and provides a visual-based industrial robot fault action detection method and system, which have the advantages that the sudden fault of an industrial robot body is found in a non-contact mode, the safety accident that the robot hurts people during man-machine cooperation is avoided, and the detection process is simple and accurate.
The purpose of the invention is realized by the following technical scheme: a method for detecting fault actions of an industrial robot based on vision comprises the following steps,
s1: collecting standard operation videos of the industrial robot, and establishing a standard operation mode video frame sequence of the industrial robot;
s2: acquiring an industrial robot operation image in real time, and acquiring an industrial robot real-time action image;
s3: matching the real-time motion image of the industrial robot with the standard operation mode video frame sequence of the industrial robot, judging whether an image approximately matched with the real-time motion image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot, if so, executing S2, and if not, executing S4;
s4: and controlling the industrial robot to suddenly stop.
The method has the advantages that the method collects the working image of the industrial robot in real time in a non-contact mode, matches the real-time action image of the industrial robot with the standard working mode video frame sequence of the industrial robot, judges whether the image approximately matched with the real-time action image of the industrial robot exists in the standard working mode video frame sequence of the industrial robot or not, and judges that the working posture of the industrial robot is normal if the image approximately matched with the real-time action image of the industrial robot exists in the standard working mode video frame sequence; if not, judge that industrial robot working attitude is unusual and control industrial robot scram, need not data acquisition device and gather industrial robot each axle and terminal real-time status information or industrial robot's running state data, the testing process is simple accurate and the cost is lower.
Further, the step S1 specifically includes the following steps,
s11: collecting standard operation video of an industrial robot;
s12: carrying out T video frame extraction on the standard operation video of the industrial robot to form a video frame sequence;
s13: extracting frames containing motion images of one period of the industrial robot in the video frame sequence, and establishing an industrial robot operation mode video frame sequence;
s14: and carrying out image segmentation on the industrial robot operation mode video frame sequence, separating the industrial robot image, and establishing the industrial robot standard operation mode video frame sequence.
The beneficial effect of adopting the above further scheme is that the standard operation video of the industrial robot is collected, and the T video frame extraction is carried out on the standard operation video of the industrial robot to form a video frame sequence. In order to collect the video, the video frame sequence does not only comprise an industrial robot image of a period operation, so that the video frame sequence is required to be cut, frames comprising a motion image of the industrial robot in a period are extracted from the video frame sequence, and the industrial robot operation mode video frame sequence is established. In order to increase the accuracy of motion detection of the processing robot, image segmentation needs to be performed on an industrial robot working mode video frame sequence, an industrial robot image is separated, and an industrial robot standard working mode video frame sequence is established.
Further, the step S13 specifically includes the following steps,
s131: the sequence of video frames is<I1,I2,…In>，IkK ∈ N are image frames, each image frame containing a working movement of the industrial robot, marking the start frame I of a working cycle of the working robotsAnd an end frame Ie；
S132: extracting image frames of a working cycle of the working robot to generate a video frame sequence of an industrial robot working mode<Is,Is+1,…Ie>。
The method has the advantages that for the convenience of video acquisition, when the standard operation video of the industrial robot is acquired, the acquisition is not strictly required from the beginning of one period, so that the video frame sequence does not only comprise the image of the industrial robot operating in one period, and therefore the video frame sequence needs to be cut, frames containing the motion image of the industrial robot operating in one period in the video frame sequence are extracted, and the industrial robot operation mode video frame sequence is established. By marking the starting frame I of one working cycle of the working robotsAnd an end frame IeExtracting image frames of a working cycle of the working robot to generate a video frame sequence of an industrial robot working mode<Is,Is+1,…Ie>。
Further, the image segmentation in S14 specifically includes,
s141: determining color C of an industrial robotr；
S142: i is a working image containing an industrial robot, P is any pixel in I, and whether the color value of P is in C or not is judgedrIn the area of the center, if yes, execute S143, if no, execute S144;
s143: setting the color value of P to black;
s144: the color value of P is set to white.
The beneficial effect who adopts above-mentioned further scheme is that, in order to avoid when industrial robot real-time action image matches with industrial robot standard operation mode video frame sequence, the background image causes the erroneous judgement, need extract the industrial robot image of each frame in industrial robot standard operation mode video frame sequence, matches with the industrial robot image in the industrial robot real-time action image, improves the accuracy that industrial robot trouble action detected.
Further, the step S3 specifically includes the following steps.
S31: sequence number variable q for initializing real-time motion image0＝-1；
S32: searching whether an image approximately matched with the real-time motion image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot, if so, executing S33, otherwise, executing S4;
s33: recording serial number q of image approximately matched with real-time motion image of industrial robot in standard operation mode video frame sequence of industrial robot1；
S34: if q is1＝-1Vq1＝q0+1, then let q0＝q1Where V is an operation symbol indicating an or operation, S2 is executed.
Further, the S32 specifically includes,
s321: image frame sequence number q in video frame sequence of standard operation mode of initial industrial robot1，q1＝s；
S322: calculating the difference value between the real-time motion image of the industrial robot and an image frame Iq1 in a standard operation mode video frame sequence of the industrial robot, wherein the image difference value calculation method comprises the following steps:
wherein d (I)1,I2) Representing an image I1And image I2The difference between m × n represents the resolution of the image, I1(i, j) tableDisplay image I1Of the ith row and the jth column of pixels, I2(I, j) represents an image I2The color value of the ith row and the jth column of the pixel;
s323: judging whether the difference value is smaller than a threshold value D, if so, executing S325, and if not, executing S324;
s324: let q be1＝q1+1, go to S322;
s325: judging that the serial number in a video frame sequence of a standard working mode of an industrial robot is q1Is an approximate matching image of the real-time motion image of the industrial robot.
A vision-based industrial robot fault action detection system comprises,
the image acquisition device is used for acquiring standard operation videos of the industrial robot and acquiring real-time action images of the industrial robot in real time;
the fault detection device is used for receiving the standard operation video of the industrial robot acquired by the image acquisition device to establish a standard operation mode video frame sequence of the industrial robot, receiving the real-time action image of the industrial robot acquired by the image acquisition device in real time, matching the real-time action image of the industrial robot with the standard operation mode video frame sequence of the industrial robot, judging whether an image approximately matched with the real-time action image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot or not, and sending an emergency stop control signal after judging that the image approximately matched with the real-time action image of the industrial robot does not exist in the standard operation mode video frame sequence of the industrial robot;
and the controller is used for sending an emergency stop control signal to the fault detection device and controlling the industrial robot to stop working.
The system has the advantages that the working images of the industrial robot are collected in real time in a non-contact mode, the real-time action images of the industrial robot are matched with the standard working mode video frame sequence of the industrial robot through the fault detection device, whether the images approximately matched with the real-time action images of the industrial robot exist in the standard working mode video frame sequence of the industrial robot or not is judged, and if yes, the working posture of the industrial robot is judged to be normal; if not, judge industrial robot working attitude unusual to through controller control industrial robot scram, need not data acquisition device and gather industrial robot each axle and terminal real-time state information or industrial robot's running state data, the testing process is simple accurate and the cost is lower.
Further, the fault detection device comprises an industrial robot standard operation video establishing unit, an image segmentation unit and an image matching unit,
the industrial robot standard operation video establishing unit is used for extracting a T video frame from an industrial robot standard operation video to form a video frame sequence, extracting frames containing action images of one period of the industrial robot in the video frame sequence and establishing an industrial robot operation mode video frame sequence, wherein each frame in the industrial robot operation mode video frame sequence contains an operation action of the industrial robot;
the image segmentation unit is used for extracting an industrial robot image in each frame of image of the robot industrial robot working mode video frame sequence and sending the image to the industrial robot standard working video establishing unit;
the industrial robot standard operation video establishing unit is used for receiving the industrial robot image of each frame extracted by the image segmentation unit and generating an industrial robot standard operation video;
the image segmentation unit is also used for extracting an industrial robot image in the real-time action image of the industrial robot;
the image matching unit is used for matching the real-time action image of the industrial robot with the image in the standard operation mode video frame sequence of the industrial robot, judging whether the image approximately matched with the real-time action image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot or not, and sending an emergency stop control signal after judging that the image approximately matched with the real-time action image of the industrial robot does not exist in the standard operation mode video frame sequence of the industrial robot.
The beneficial effect of adopting above-mentioned further scheme is that, image acquisition device gathers industrial robot standard operation video, and industrial robot standard operation video establishes the unit and carries out T video frame extraction to industrial robot standard operation video, forms the video frame sequence. In order to collect the video, the video frame sequence does not only comprise an industrial robot image of a period operation, so that the video frame sequence is required to be cut, frames comprising a motion image of the industrial robot in a period are extracted from the video frame sequence, and the industrial robot operation mode video frame sequence is established. In order to increase the accuracy of motion detection of the processing robot, an image segmentation unit is required to perform image segmentation on the industrial robot working mode video frame sequence and separate the industrial robot images, and an industrial robot standard working video establishment unit establishes the industrial robot standard working mode video frame sequence according to the images processed by the image segmentation unit.
Further, the image segmentation unit extracting the image of the industrial robot comprises the steps of,
s141: determining color C of an industrial robotr；
S142: i is a working image containing an industrial robot, P is any pixel in I, and whether the color value of P is in C or not is judgedrIn the area of the center, if yes, execute S143, if no, execute S144;
s143: setting the color value of P to black;
s144: the color value of P is set to white.
The beneficial effect of adopting above-mentioned further scheme is that, when avoiding the image matching unit to match industrial robot real-time action image and industrial robot standard operation mode video frame sequence, the background image causes erroneous judgement, need extract the industrial robot image of each frame in industrial robot standard operation mode video frame sequence, match with the industrial robot image in the industrial robot real-time action image, improve industrial robot fault action detection's accuracy.
Further, the image matching unit is also used for recording the industrial robot after judging that the image approximately matched with the real-time action image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robotSequence number q of image approximately matched with real-time motion image of industrial robot in robot standard operation mode video frame sequence1。
The image matching unit is also used for recording the serial number q of the image approximately matched with the real-time action image of the industrial robot in the standard operation mode video frame sequence of the industrial robot after judging that the image approximately matched with the real-time action image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot1And the operator can conveniently record the sequence number q according to the image1And the industrial robot works in real time, and the accuracy of the image matching unit is judged.
Drawings
Fig. 1 is a schematic diagram of a vision-based industrial robot malfunction detection system according to embodiment 1 of the present invention;
FIG. 2 is a schematic flow chart illustrating the detection of malfunction of an industrial robot according to the present invention;
FIG. 3 is a schematic diagram illustrating a process for establishing a standard working mode video frame sequence of an industrial robot according to the present invention;
fig. 4 is a schematic diagram for showing the process of the approximate matching of the real-time motion image of the industrial robot according to the invention.
Detailed Description
The technical solutions of the present invention are further described in detail below with reference to the accompanying drawings, but the scope of the present invention is not limited to the following.
Example 1
Referring to fig. 1 and 2, a vision-based fault motion detection system for an industrial robot includes,
the image acquisition device is used for acquiring standard operation videos of the industrial robot and acquiring real-time action images of the industrial robot in real time; it should be noted that, in this embodiment, the image acquisition device is a high-definition camera;
the fault detection device is used for receiving the standard operation video of the industrial robot acquired by the image acquisition device to establish a standard operation mode video frame sequence of the industrial robot, receiving the real-time action image of the industrial robot acquired by the image acquisition device in real time, matching the real-time action image of the industrial robot with the standard operation mode video frame sequence of the industrial robot, judging whether an image approximately matched with the real-time action image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot or not, and sending an emergency stop control signal after judging that the image approximately matched with the real-time action image of the industrial robot does not exist in the standard operation mode video frame sequence of the industrial robot;
and the controller is used for sending an emergency stop control signal to the fault detection device and controlling the industrial robot to stop working.
Specifically, the system collects an industrial robot working image in real time in a non-contact mode, matches the industrial robot real-time action image with an industrial robot standard working mode video frame sequence through a fault detection device, judges whether an image approximately matched with the industrial robot real-time action image exists in the industrial robot standard working mode video frame sequence, and judges that the working posture of the industrial robot is normal if the image approximately matched with the industrial robot real-time action image exists in the industrial robot standard working mode video frame sequence; if not, judge industrial robot working attitude unusual to through controller control industrial robot scram, need not data acquisition device and gather industrial robot each axle and terminal real-time state information or industrial robot's running state data, the testing process is simple accurate and the cost is lower.
Referring to fig. 1, it is worth explaining that the fault detection apparatus includes an industrial robot standard work video creation unit, an image segmentation unit, and an image matching unit. The three units are explained in turn below.
Referring to fig. 3, the standard working video creation unit of the industrial robot is configured to perform T video frame extraction on a standard working video of the industrial robot, and it should be noted that the T video frame extraction means that a separation time between two adjacent frames in a video frame sequence is T to form the video frame sequence,<I1,I2,…In>，Ikand k ∈ N is an image frame, the standard work video establishing unit of the industrial robot is also used for extracting the frames containing the action images of one period of the industrial robot in the video frame sequence and establishing the industrial robotSequence of operation mode video frames<Is,Is+1,…Ie>Wherein, IsAnd IeInitial frame I of one working cycle of working robot respectivelysAnd an end frame Ie，Each frame of the industrial robot work mode video frame sequence comprises a work action of the industrial robot.
It should be noted that, in this embodiment, the initial frame I of one working cycle of the working robot is determined by using an artificial labeling methodsAnd an end frame Ie. In another embodiment, the start frame I of a working robot for a working cycle can be determined in another waysAnd an end frame IeFor example, the starting frame I is determined as the starting image of one working cycle of the working robotsDetermining the start frame IsThe images separated by N periods of one period are end frames Ie. In another embodiment, the start frame I of a work robot work cycle can be determined in another waysAnd an end frame IeFor example, the starting frame I is determined as the starting image of one working cycle of the working robotsFind the start frame IsSeparated by a time greater than a time threshold and spaced from the start frame IsThe image with the image similarity larger than the similarity threshold is the ending frame Ie。
The image segmentation unit is used for extracting an industrial robot image in each frame of image of the robot working mode video frame sequence and sending the image to the industrial robot standard working video establishing unit.
It is worth mentioning that the image segmentation unit for extracting the image of the industrial robot comprises the following steps,
s141: determining color C of an industrial robotr；
S142: i is a working image containing an industrial robot, P is any pixel in I, and whether the color value of P is in C or not is judgedrIn the area of the center, if yes, execute S143, if no, execute S144;
s143: setting the color value of P to black;
s144: the color value of P is set to white.
Therefore, binarization processing of the operation image of the industrial robot is realized, the industrial robot is separated from the background, and the accuracy of fault action detection of the industrial robot is improved.
The standard work video establishing unit of the industrial robot is used for receiving the industrial robot image of each frame extracted by the image segmentation unit and generating a standard work video of the industrial robot.
The image segmentation unit is also used for extracting an industrial robot image in the real-time motion image of the industrial robot.
Referring to fig. 4, the image matching unit is configured to match the real-time motion image of the industrial robot with an image in a standard operation mode video frame sequence of the industrial robot, determine whether an image approximately matching the real-time motion image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot, and send an emergency stop control signal after determining that an image approximately matching the real-time motion image of the industrial robot does not exist in the standard operation mode video frame sequence of the industrial robot. The image matching unit is also used for recording the serial number q of the image approximately matched with the real-time motion image of the industrial robot in the standard work mode video frame sequence of the industrial robot after judging that the image approximately matched with the real-time motion image of the industrial robot exists in the standard work mode video frame sequence of the industrial robot1。
It is worth mentioning that the image matching unit for matching the real-time motion image of the industrial robot with the image in the standard working mode video frame sequence of the industrial robot specifically comprises the following steps,
s31: sequence number variable q for initializing real-time motion image0＝-1；
S32: searching whether an image approximately matched with the real-time motion image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot, if so, executing S33, and if not, sending an emergency stop control signal to the controller;
s33: recording serial number q of image approximately matched with real-time motion image of industrial robot in standard operation mode video frame sequence of industrial robot1；
S34: if q is1＝-1Vq1＝q0+1, then let q0＝q1And V is an operation symbol, represents or operation and is used for matching the next frame of real-time motion image of the industrial robot.
It is further noted that S32 specifically includes,
s321: image frame sequence number q in video frame sequence of standard operation mode of initial industrial robot1，q1＝s；
S322: calculating the difference value between the real-time motion image of the industrial robot and an image frame Iq1 in a standard operation mode video frame sequence of the industrial robot, wherein the image difference value calculation method comprises the following steps:
wherein d (I)1,I2) Representing an image I1And image I2The difference between m × n represents the resolution of the image, I1(I, j) represents an image I1Of the ith row and the jth column of pixels, I2(I, j) represents an image I2The color value of the ith row and the jth column of the pixel;
s323: judging whether the difference value is smaller than a threshold value D, if so, executing S325, and if not, executing S324;
s324: let q be1＝q1+1, go to S322;
s325: judging that the serial number in a video frame sequence of a standard working mode of an industrial robot is q1Is an approximate matching image of the real-time motion image of the industrial robot, S33 is performed.
It is also worth mentioning that the serial number q is recorded from the image for the convenience of the operator1And the real-time working state of the industrial robot, and the accuracy of an image matching unit is judged, wherein the image matching unit records the serial number q of an image approximately matched with the real-time action image of the industrial robot in the standard working mode video frame sequence of the industrial robot1The operator can query the sequence by external human-computer interaction device (such as display screen, mouse and keyboard)Column number q1And judging by combining the real-time operation action of the industrial robot.
It is worth to be noted that in the embodiment, the controller is in emergency stop by communicating with the industrial robot control cabinet; in another embodiment, the controller can be directly connected with an electric control switch in a power-on line of the industrial robot, and the industrial robot is controlled to stop suddenly by controlling the disconnection of the electric control switch.
It should be noted that, in this embodiment, the hardware device of the fault detection apparatus may include a Central Processing Unit (CPU), and may further include other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The hardware device of the fault detection device further comprises a memory. The memory may be an internal storage unit of the processor, such as a hard disk or a memory of the processor. The memory may also be an external storage device of the processor, such as a plug-in hard disk provided on the processor, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like. The memory may also include both internal and external storage for the processor. The memory is used for storing computer programs and other programs and data required by the processor. The memory may also be used to temporarily store data that has been output or is to be output.
It should be noted that, in this embodiment, the hardware device of the controller may include a Central Processing Unit (CPU), and may further include other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The hardware device of the controller also includes a memory. The memory may be an internal storage unit of the processor, such as a hard disk or a memory of the processor. The memory may also be an external storage device of the processor, such as a plug-in hard disk provided on the processor, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like. The memory may also include both internal and external storage for the processor. The memory is used for storing computer programs and other programs and data required by the processor. The memory may also be used to temporarily store data that has been output or is to be output.
Example 2
Referring to fig. 2, a vision-based method for detecting malfunction of an industrial robot includes the following steps,
s1: collecting standard operation videos of the industrial robot, and establishing a standard operation mode video frame sequence of the industrial robot;
s2: acquiring an industrial robot operation image in real time, and acquiring an industrial robot real-time action image;
s3: matching the real-time motion image of the industrial robot with the standard operation mode video frame sequence of the industrial robot, judging whether an image approximately matched with the real-time motion image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot, if so, executing S2, and if not, executing S4;
s4: and controlling the industrial robot to suddenly stop.
Specifically, the method comprises the steps of acquiring an industrial robot working image in real time in a non-contact mode, matching the industrial robot real-time action image with an industrial robot standard working mode video frame sequence, judging whether an image approximately matched with the industrial robot real-time action image exists in the industrial robot standard working mode video frame sequence, and judging that the working posture of the industrial robot is normal if the image approximately matched with the industrial robot real-time action image exists in the industrial robot standard working mode video frame sequence; if not, judge that industrial robot working attitude is unusual and control industrial robot scram, need not data acquisition device and gather industrial robot each axle and terminal real-time status information or industrial robot's running state data, the testing process is simple accurate and the cost is lower.
Each step is described in detail below.
Referring to fig. 3, S1 specifically includes the following steps,
s11: collecting standard operation video of an industrial robot;
s12: carrying out T video frame extraction on the standard operation video of the industrial robot to form a video frame sequence; it is worth to be noted that T video frame extraction means that the interval time between two adjacent frames in a video frame sequence is T;
s13: extracting frames containing motion images of one period of the industrial robot in the video frame sequence, and establishing an industrial robot operation mode video frame sequence;
s14: and performing image segmentation on the industrial robot operation mode video frame sequence, separating the industrial robot image, and establishing the industrial robot standard operation mode video frame sequence.
Specifically, an industrial robot standard operation video is collected, and T video frame extraction is carried out on the industrial robot standard operation video to form a video frame sequence. In order to collect the video, the video frame sequence does not only comprise an industrial robot image of a period operation, so that the video frame sequence is required to be cut, frames comprising a motion image of the industrial robot in a period are extracted from the video frame sequence, and the industrial robot operation mode video frame sequence is established. In order to increase the accuracy of motion detection of the processing robot, image segmentation needs to be performed on an industrial robot working mode video frame sequence, an industrial robot image is separated, and an industrial robot standard working mode video frame sequence is established.
It is noted that S13 specifically includes the following steps,
s131: the sequence of video frames is<I1,I2,…In>，IkK ∈ N are image frames, each image frame containing a working motion of an industrial robotMarking the initial frame I of one working period of the working robotsAnd an end frame Ie；
S132: extracting image frames of a working cycle of the working robot to generate a video frame sequence of an industrial robot working mode<Is,Is+1,…Ie>。
It should be noted that, in this embodiment, the initial frame I of one working cycle of the working robot is determined by means of manual labelingsAnd an end frame Ie. In another embodiment, the start frame I of a working robot for a working cycle can be determined in another waysAnd an end frame IeFor example, the starting frame I is determined as the starting image of one working cycle of the working robotsDetermining the start frame IsThe images separated by N periods of one period are end frames Ie. In another embodiment, the start frame I of a work robot work cycle can be determined in another waysAnd an end frame IeFor example, the starting frame I is determined as the starting image of one working cycle of the working robotsFind the start frame IsSeparated by a time greater than a time threshold and spaced from the start frame IsThe image with the image similarity larger than the similarity threshold is the ending frame Ie。
It is further noted that the image segmentation in S14 specifically includes,
s141: determining color C of an industrial robotr；
S142: i is a working image containing an industrial robot, P is any pixel in I, and whether the color value of P is in C or not is judgedrIn the area of the center, if yes, execute S143, if no, execute S144;
s143: setting the color value of P to black;
s144: the color value of P is set to white.
Therefore, binarization processing of the operation image of the industrial robot is realized, and the industrial robot is separated from the background.
The S2 may specifically include the following,
the method comprises the steps of collecting an industrial robot operation image in real time, carrying out image segmentation on the industrial robot operation image, and obtaining the real-time action image of the industrial robot.
It is noted that the image segmentation in S2 includes the steps of,
s21: determining color C of an industrial robotr；
S22: i is a working image containing an industrial robot, P is any pixel in I, and whether the color value of P is in C or not is judgedrIn the area of the center, if yes, execute S143, if no, execute S144;
s23: setting the color value of P to black;
s24: the color value of P is set to white.
Through the image segmentation step in step S14 and the image segmentation step in S2, first, the influence of the background image in the image matching process in S3 is reduced; secondly, the influence of the difference between the color of the industrial robot and the color of the industrial robot in the standard operation video of the industrial robot is reduced, the accuracy of image matching in S3 is improved, and meanwhile the universality of the method is improved.
Referring to fig. 4, it is worth explaining that S3 specifically includes the following steps.
S31: sequence number variable q for initializing real-time motion image0＝-1；
S32: searching whether an image approximately matched with the real-time motion image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot, if so, executing S33, otherwise, executing S4;
s33: recording serial number q of image approximately matched with real-time motion image of industrial robot in standard operation mode video frame sequence of industrial robot1；
S34: if q is1＝-1Vq1＝q0+1, then let q0＝q1Where V is an operation symbol indicating an or operation, S2 is executed.
It is further noted that S32 specifically includes,
s321: in initial industrial robot standard operation mode video frame sequenceImage frame number q1，q1＝s；
S322: calculating the difference value between the real-time motion image of the industrial robot and an image frame Iq1 in a standard operation mode video frame sequence of the industrial robot, wherein the image difference value calculation method comprises the following steps:
wherein d (I)1,I2) Representing an image I1And image I2The difference between m × n represents the resolution of the image, I1(I, j) represents an image I1Of the ith row and the jth column of pixels, I2(I, j) represents an image I2The color value of the ith row and the jth column of the pixel;
s323: judging whether the difference value is smaller than a threshold value D, if so, executing S325, and if not, executing S324;
s324: let q be1＝q1+1, go to S322;
s325: judging that the serial number in a video frame sequence of a standard working mode of an industrial robot is q1Is an approximate matching image of the real-time motion image of the industrial robot, S33 is performed.
Specifically, the serial number q is recorded according to the image for the convenience of an operator1And the real-time working state of the industrial robot, the accuracy of the method is judged, and the serial number q of the image approximately matched with the real-time action image of the industrial robot in the standard working mode video frame sequence of the industrial robot is recorded1The operator can inquire the serial number q1And judging by combining the real-time operation action of the industrial robot.
The foregoing is merely a preferred embodiment of the invention, it is to be understood that the invention is not limited to the forms disclosed herein, but is not intended to be exhaustive or to limit the invention to other embodiments, and to various other combinations, modifications, and environments and may be modified within the scope of the inventive concept as expressed herein, by the teachings or the skill or knowledge of the relevant art. And that modifications and variations may be effected by those skilled in the art without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A method for detecting fault actions of an industrial robot based on vision is characterized by comprising the following steps,
s1: collecting standard operation videos of the industrial robot, and establishing a standard operation mode video frame sequence of the industrial robot;
s2: acquiring an industrial robot operation image in real time, and acquiring an industrial robot real-time action image;
s3: matching the real-time motion image of the industrial robot with the standard operation mode video frame sequence of the industrial robot, judging whether an image approximately matched with the real-time motion image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot, if so, executing S2, and if not, executing S4;
s4: and controlling the industrial robot to suddenly stop.
2. The vision based industrial robot malfunction motion detection method according to claim 1, wherein said S1 specifically includes the steps of,
s11: collecting standard operation video of an industrial robot;
s12: carrying out T video frame extraction on the standard operation video of the industrial robot to form a video frame sequence;
s13: extracting frames containing motion images of one period of the industrial robot in the video frame sequence, and establishing an industrial robot operation mode video frame sequence;
s14: and carrying out image segmentation on the industrial robot operation mode video frame sequence, separating the industrial robot image, and establishing the industrial robot standard operation mode video frame sequence.
3. The vision based industrial robot malfunction motion detection method according to claim 2, wherein said S13 specifically includes the steps of,
s131: the sequence of video frames is<I1，I2，...In>，IkK ∈ N are image frames, each image frame containing a working movement of the industrial robot, marking the start frame I of a working cycle of the working robotsAnd an end frame Ie；
S132: extracting image frames of a working cycle of the working robot to generate a video frame sequence of an industrial robot working mode<Is，Is+1，...Ie>。
4. The vision-based industrial robot malfunction detection method according to claim 2, wherein said image segmentation in S14 specifically includes,
s141: determining color C of an industrial robotr；
S142: i is a working image containing an industrial robot, P is any pixel in I, and whether the color value of P is in C or not is judgedrIn the area of the center, if yes, execute S143, if no, execute S144;
s143: setting the color value of P to black;
s144: the color value of P is set to white.
5. The vision-based industrial robot malfunction motion detection method according to any one of claims 1 to 4, wherein the step S3 specifically comprises the steps of:
s31: sequence number variable q for initializing real-time motion image0＝-1；
S32: searching whether an image approximately matched with the real-time motion image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot, if so, executing S33, otherwise, executing S4;
s33: recording serial number q of image approximately matched with real-time motion image of industrial robot in standard operation mode video frame sequence of industrial robot1；
S34: if q is1＝-1∨q1＝q0+1, then let q0＝q1Where V is an operation symbol indicating an or operation, S2 is executed.
6. The vision-based industrial robot malfunction motion detection method according to claim 5, wherein said S32 specifically includes,
s321: image frame sequence number q in video frame sequence of standard operation mode of initial industrial robot1，q1＝s；
S322: calculating the difference value between the real-time motion image of the industrial robot and an image frame Iq1 in a standard operation mode video frame sequence of the industrial robot, wherein the image difference value calculation method comprises the following steps:
wherein d (I)1，I2) Representing an image I1And image I2The difference between m × n represents the resolution of the image, I1(I, j) represents an image I1Of the ith row and the jth column of pixels, I2(I, j) represents an image I2The color value of the ith row and the jth column of the pixel;
s323: judging whether the difference value is smaller than a threshold value D, if so, executing S325, and if not, executing S324;
s324: let q be1＝q1+1, go to S322;
s325: judging that the serial number in a video frame sequence of a standard working mode of an industrial robot is q1Is an approximate matching image of the real-time motion image of the industrial robot, S33 is performed.
7. A vision-based industrial robot fault action detection system is characterized by comprising,
the image acquisition device is used for acquiring standard operation videos of the industrial robot and acquiring real-time action images of the industrial robot in real time;
the fault detection device is used for receiving the standard operation video of the industrial robot acquired by the image acquisition device to establish a standard operation mode video frame sequence of the industrial robot, receiving the real-time action image of the industrial robot acquired by the image acquisition device in real time, matching the real-time action image of the industrial robot with the standard operation mode video frame sequence of the industrial robot, judging whether an image approximately matched with the real-time action image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot or not, and sending an emergency stop control signal after judging that the image approximately matched with the real-time action image of the industrial robot does not exist in the standard operation mode video frame sequence of the industrial robot;
and the controller is used for sending an emergency stop control signal to the fault detection device and controlling the industrial robot to stop working.
8. The vision-based industrial robot malfunction detection system according to claim 7, wherein the malfunction detection apparatus includes an industrial robot standard work video creation unit, an image segmentation unit, and an image matching unit,
the industrial robot standard operation video establishing unit is used for extracting a T video frame from an industrial robot standard operation video to form a video frame sequence, extracting frames containing action images of one period of the industrial robot in the video frame sequence and establishing an industrial robot operation mode video frame sequence, wherein each frame in the industrial robot operation mode video frame sequence contains an operation action of the industrial robot;
the image segmentation unit is used for extracting an industrial robot image in each frame of image of the robot industrial robot working mode video frame sequence and sending the image to the industrial robot standard working video establishing unit;
the industrial robot standard operation video establishing unit is used for receiving the industrial robot image of each frame extracted by the image segmentation unit and generating an industrial robot standard operation video;
the image segmentation unit is also used for extracting an industrial robot image in the real-time action image of the industrial robot;
the image matching unit is used for matching the real-time action image of the industrial robot with the image in the standard operation mode video frame sequence of the industrial robot, judging whether the image approximately matched with the real-time action image of the industrial robot exists in the standard operation mode video frame sequence of the industrial robot or not, and sending an emergency stop control signal after judging that the image approximately matched with the real-time action image of the industrial robot does not exist in the standard operation mode video frame sequence of the industrial robot.
9. The vision based industrial robot malfunction detection system of claim 8, wherein the image segmentation unit extracts an industrial robot image comprising the steps of,
s141: determining color C of an industrial robotr；
S142: i is a working image containing an industrial robot, P is any pixel in I, and whether the color value of P is in C or not is judgedrIn the area of the center, if yes, execute S143, if no, execute S144;
s143: setting the color value of P to black;
s144: the color value of P is set to white.
10. The vision-based industrial robot malfunction detection system according to claim 8, wherein the image matching unit is further configured to record a serial number q of an image in the standard working mode video frame sequence of the industrial robot that approximately matches the real-time motion image of the industrial robot after determining that the image in the standard working mode video frame sequence of the industrial robot approximately matches the real-time motion image of the industrial robot1。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010342989.XA CN111531581B (en) | 2020-04-27 | 2020-04-27 | Industrial robot fault action detection method and system based on vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010342989.XA CN111531581B (en) | 2020-04-27 | 2020-04-27 | Industrial robot fault action detection method and system based on vision |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111531581A true CN111531581A (en) | 2020-08-14 |
| CN111531581B CN111531581B (en) | 2023-02-03 |
Family
ID=71970285
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010342989.XA Active CN111531581B (en) | 2020-04-27 | 2020-04-27 | Industrial robot fault action detection method and system based on vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111531581B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115958609A (en) * | 2023-03-16 | 2023-04-14 | 山东卓朗检测股份有限公司 | Instruction data safety early warning method based on intelligent robot automatic control system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6356671B1 (en) * | 1991-07-05 | 2002-03-12 | Fanuc Ltd. | Image processing method for an industrial visual sensor |
| JP2013258542A (en) * | 2012-06-12 | 2013-12-26 | Idec Corp | Robot control system, moving image transfer method, and moving image transfer program |
| CN104050771A (en) * | 2013-03-13 | 2014-09-17 | 霍尼韦尔国际公司 | System and method of anomaly detection |
| JP2017121685A (en) * | 2016-01-07 | 2017-07-13 | 富士通株式会社 | Method for determination of abnormality when manufacturing, program for abnormality determination when manufacturing, and manufacturing apparatus |
| CN108972593A (en) * | 2018-09-07 | 2018-12-11 | 顺德职业技术学院 | Control method and system under a kind of industrial robot system |
-
2020
- 2020-04-27 CN CN202010342989.XA patent/CN111531581B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6356671B1 (en) * | 1991-07-05 | 2002-03-12 | Fanuc Ltd. | Image processing method for an industrial visual sensor |
| JP2013258542A (en) * | 2012-06-12 | 2013-12-26 | Idec Corp | Robot control system, moving image transfer method, and moving image transfer program |
| CN104050771A (en) * | 2013-03-13 | 2014-09-17 | 霍尼韦尔国际公司 | System and method of anomaly detection |
| JP2017121685A (en) * | 2016-01-07 | 2017-07-13 | 富士通株式会社 | Method for determination of abnormality when manufacturing, program for abnormality determination when manufacturing, and manufacturing apparatus |
| CN108972593A (en) * | 2018-09-07 | 2018-12-11 | 顺德职业技术学院 | Control method and system under a kind of industrial robot system |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115958609A (en) * | 2023-03-16 | 2023-04-14 | 山东卓朗检测股份有限公司 | Instruction data safety early warning method based on intelligent robot automatic control system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111531581B (en) | 2023-02-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110826538B (en) | Abnormal off-duty identification system for electric power business hall | |
| CN110751022A (en) | Urban pet activity track monitoring method based on image recognition and related equipment | |
| US8494223B2 (en) | Real-time annotation of images in a human assistive environment | |
| CN103517042A (en) | Nursing home old man dangerous act monitoring method | |
| CN111104820A (en) | Gesture recognition method based on deep learning | |
| CN110490171B (en) | Dangerous posture recognition method and device, computer equipment and storage medium | |
| CN111552269B (en) | Multi-robot safety detection method and system based on attitude estimation | |
| CN110889339A (en) | Head and shoulder detection-based dangerous area grading early warning method and system | |
| CN111149129A (en) | Abnormality detection device and abnormality detection method | |
| CN111047824B (en) | Indoor child nursing linkage control early warning method and system | |
| CN111531580B (en) | Vision-based multi-task robot fault detection method and system | |
| CN113419298B (en) | Multi-parameter hydrological meteorological data acquisition device | |
| TWI752638B (en) | Method and system for detection of driving anomaly | |
| CN110738289A (en) | Multi-dimensional linkage comprehensive studying and judging device for electric power operation standardization and using method thereof | |
| CN112070000A (en) | Intelligent recognition algorithm training method and device, terminal server and storage medium | |
| CN111531581B (en) | Industrial robot fault action detection method and system based on vision | |
| CN112949457A (en) | Maintenance method, device and system based on augmented reality technology | |
| CN110597165B (en) | Steel piling monitoring system and steel piling monitoring method | |
| US20220163942A1 (en) | Distributed System and Data Transmission Method | |
| CN117114420B (en) | Image recognition-based industrial and trade safety accident risk management and control system and method | |
| US11321656B2 (en) | Difference extracting device | |
| CN111531582B (en) | Industrial robot fault detection method and system based on vision | |
| CN114565870A (en) | Production line control method, device and system based on vision, and electronic equipment | |
| CN112631272A (en) | Method and equipment for remotely recovering power of robot | |
| CN111427037A (en) | Obstacle detection method and device, electronic equipment and vehicle-end equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20231008 Address after: No. 28 Manao 5th Road, Zhashan Street, Caidian District, Wuhan City, Hubei Province, 430000 Patentee after: Wuhan Zhongshe Robot Technology Co.,Ltd. Address before: 430000 No.206 Guanggu 1st Road, Donghu New Technology Development Zone, Wuhan, Hubei Province Patentee before: WUHAN INSTITUTE OF TECHNOLOGY |
|
| TR01 | Transfer of patent right | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: A Vision based Fault Action Detection Method and System for Industrial Robots Granted publication date: 20230203 Pledgee: Wuhan Rural Commercial Bank Co.,Ltd. Caidian sub branch Pledgor: Wuhan Zhongshe Robot Technology Co.,Ltd. Registration number: Y2020420000011 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right |