CN103271784B - Man-machine interactive manipulator control system and method based on binocular vision - Google Patents
Man-machine interactive manipulator control system and method based on binocular vision Download PDFInfo
- Publication number
- CN103271784B CN103271784B CN201310223530.8A CN201310223530A CN103271784B CN 103271784 B CN103271784 B CN 103271784B CN 201310223530 A CN201310223530 A CN 201310223530A CN 103271784 B CN103271784 B CN 103271784B
- Authority
- CN
- China
- Prior art keywords
- target
- hand
- coordinate
- joint
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Abstract
The invention discloses a man-machine interactive manipulator control system and method based on binocular vision. The man-machine interactive manipulator control system is composed of a real-time image collecting device, a laser guiding device, a programmable controller and a driving device. The programmable controller is composed of a binocular three-dimensional vision module, a three-dimensional coordinate system transformation module, an inverse manipulator joint angle module and a control module. Color characteristics in a binocular image are extracted through the real-time image collecting device to be used as a signal source for controlling a manipulator, and three-dimensional information of red characteristic laser points in a view real-time image is obtained through transformation and calculation of the binocular three-dimensional vision system and a three-dimensional coordinate system and used for controlling the manipulator to conduct man-machine interactive object tracking operation. The control system and method can effectively conduct real-time tracking and extracting of a moving target object and is wide in application fields such as intelligent artificial limb installing, explosive-handling robots, manipulators helping the old and the disabled and the like.
Description
Technical field
The present invention relates to the manipulator control technology in the fields such as rehabilitation assistive device, robotics, particularly relate to the control technology that mechanical hand adopts man-machine interactive, is a kind of Based Intelligent Control based on Machine Vision Recognition location and high efficiency method.
Background technology
Along with the progress of science and technology, the application of mechanical hand extends gradually.Mechanical hand is not only widely used in traditional industry, as the field such as building, mining, metallurgy, oil, chemical industry, boats and ships, and expanded the high-tech areas such as nuclear energy, Aero-Space, biochemistry to, also start to be applied to family health care simultaneously, the service trade fields such as medical treatment and nursing, as artificial limb, handicapped-resisting mechanical arm etc. of helping the elderly in intelligence.Current, how convenient, flexible manipulation mechanical hand is a very important problem content in manipulator control research field.Nowadays the biomimetic control signal in the world for controlling mechanical hand mainly contains the eye relevant to human body self and moves signal, EEG signals, voice signal, dynamoelectric signal and nerve signal etc., wherein to the control mode that mechanical hand adopts, have a lot of weak point, as low in control accuracy, complex operation is complicated, dehumanization application etc.
What is called generally refers to the control mode to robot to the control mode of mechanical hand.At present, according to the difference of robot target state, be divided into structured work environment and destructuring working environment two kinds of patterns.
Be operated in the robot in structured work environment, because the position of its target and attitude are changeless, its job content is also changeless, so be generally all control robot according to the program woven in advance, complete the operation element of some sequencing, as being operated in the robot etc. on welding production line.The technology relevant with the present invention is the robot being operated in unstructured moving grids.
Be operated in the robot of unstructured moving grids, the position of its target and attitude change often at any time, and at present, the common operative scenario of this robot has following 3 kinds:
1. the operating position of robot is changeless, at this moment can pass through certain robotic vision system (as binocular vision system).Come the position of real-time testing target and the change of attitude opposed robots coordinate system, then according to the position of target and the different of attitude, adjust movement position and the attitude of robot hand, and according to the operating path planned or final goal position, object is operated, personnel do not participate in its work process set, as being operated in the robot etc. on Food Packaging Production Line.
2. though robot operating position is change, but can detect in advance the position of target and attitude, in its work process, the position relationship of robot and target is relatively stable constant, at this moment, only need make some parameter of robot controlling and adjusting accordingly, after determining relevant parameter, robot just can control robot according to the program woven in advance, complete the operation element of those sequencing, personnel do not participate in its work process set, as concrete jetting robot or problem of rock-drilling robot etc.
3. robot operating position is change, and accurately cannot detect the position of its target and attitude again, the master-slave mode mode of operation that adopts controls robot more at present, namely, by the operation to " main hands " handle or button, adjust the movement position of " from hands " (i.e. mechanical hand of practical operation work) hand at any time, robot is operated realization of goal, personnel participate in its operating process closely, as the anti-riot robot etc. of master-slave mode operation.
In the above-mentioned the 3rd kind of mode of operation, how to adjust " main hands " state, to make " from hands " accurately arrive target location, entirely control with the range estimation of operator and experience, such control mode has following deficiency:
(1). action is gone in all joints that operator must manipulate " main hands ", and to make " from hands " make corresponding action, whole control procedure is complicated and tediously long.
(2). due to operator range estimation error and " main hands " operation amplitude is not easily accurately controlled, so in adjustment " main hands " process, often make " from hands " movement " not in place " or " moving past position ", need repeatedly to adjust and " from hands " just can be made to reach requirement position, very inconvenient.
(3). this mode of operation not easily makes hand walk along a certain random walk or complete a class activity of playing chess.
In order to solve the difficult problem that mechanical hand exists in the 3rd kind of mode of operation, personnel can be made again at any time in Attended Operation process, content of the present invention can be proposed.
Present invention applicant had once applied for a kind of the method for solving of joint of artificial limb spatial parameter " on the multiple degrees of freedom " (publication number CN101953727A, application number 201010280499.8), the method can in the position, working place only knowing target, and under not knowing required hand gestures situation, to certain feasible solution of joint space gain of parameter of artificial limb on multiple degrees of freedom or robot, so that the operation task of specifying can be completed.
Summary of the invention
The object of the invention is for the deficiency existing for the operation of existing " master-slave mode " mechanical hand, in order to make mechanical hand manipulator convenient flexibly, efficient and the operation of hommization, realize in conjunction with existing multi-freedom artificial limb to carry out joint of artificial limb spatial parameter on sterically defined method and multiple degrees of freedom method for solving to random targets, propose that a kind of personnel's property of participation is strong, man-machine interaction effect good, can move down in the guiding of laser instrument and start a kind of new mechanical arm control system of position and control method.
For achieving the above object, manipulator control system of the present invention is made up of following four parts: real time image collection device, laser guide, programmable controller and driving device; Wherein:
Described real time image collection device is made up of two wireless camera machines or USB camera, form the so-called virtual binocular of mechanical hand thus---" eyes ", this pair " eyes " for synchronous Real-time Collection mechanical hand control left and right two two field picture of scope, it has trackability, follows the tracks of the visual field for man-machine Interactive control mechanical hand provides.
Described laser guide wears frame by one and be located at the mini laser instrument worn on frame and form, and this device is for the colour recognition module in binocular stereo vision module provides red characteristic point and set up.Laser guide by real time image collection module provide the target object in image to mark out, then laser spots carries out corresponding servo-actuated action to the target object of motion; The red laser point that laser instrument is got in the region of two wireless camera common identification, otherwise must can affect the normal operation of Binocular Stereo Vision System.
Described Programmable Logic Controller is made up of binocular stereo vision module, three-dimensional system of coordinate conversion module, reverse anti-solution manipulator joint Corner Block List Representation and control module; Wherein:
Binocular stereo vision module is used for beating laser instrument carrying out detection identification and location at the red laser point in the wireless camera machine visual field.Laser spots is as the characteristic point on a frame two field picture in wireless camera video flowing, carries out red Feature point recognition, and then carry out three-dimensional coordinate measurement to the laser spots identified for the Binocular Stereo Vision System developed based on OpenCV function library.
The three-dimensional system of coordinate conversion module relation be used between the video camera photocentre coordinate system that Binocular Stereo Vision System module recorded and laser point coordinates system is converted into the relation between the base coordinate system of mechanical hand top O point and target laser point world coordinate system.
Reverse anti-solution joint of artificial limb Corner Block List Representation is an artificial nerve network model trained, its Main Function is, after arm-and-hand system obtains the three dimensional space coordinate value of random work target in base coordinate system, these 3 coordinate figures and certain are owed the input of compensatory attitude description as network model of definition, come fast with this, obtain concurrently certain of each joint of mechanical hand feasible solve value, according to these values of solving to rotate the angle in each joint, just the hand center of mechanical hand can be placed into laser spots instruction position certain height near.Its operation principle is as described in existing patent of invention the method for solving of joint of artificial limb spatial parameter " on the multiple degrees of freedom " (publication number CN101953727A, application number 201010280499.8).It can thus be appreciated that after the three-dimensional information of target object imports into by three-dimensional coordinate transformation module, institute's established model just reverse anti-solution can obtain the feasible joint angles in each joint of mechanical hand at once, finally does evil through another person and is just arrived target object position smoothly.
Control module is used for sending trace command, captures target instruction target word, unclamps target instruction target word, hand lifting instruction, hand decline instruction and teaching target instruction target word.
Described driving device comprises the motor of driver and driving device hands; When driver receives the action command that control module sends, can at once drive corresponding motor rotation, each joint angle of mechanical hand be regulated and controled, finally completes the control to mechanical hand.
The control method of control system of the present invention is as follows:
The first step: when operator need to operate certain target, opening control brings into operation, and image collecting device starts real-time image acquisition data;
Second step: controller sends trace command, operator open laser instrument, on the target object operated to needs by laser beam projects artificially, position, lay a good groundwork for Manipulator Hand accurately arrives target object location to target object; If target object there occurs motion, now, operator can swing laser guide artificially, and laser spots is beaten on target object always, servo-actuated with target, so achieves man-machine interaction tracking target object;
3rd step: binocular vision identification locating module carries out colour recognition and three-dimensional information location to the red feature laser spots that laser instrument is beaten on target object, namely three-dimensional coordinate measurement is carried out to the red feature laser spots identified, calculate the three-dimensional coordinate of the laser spots on target object relative to left video camera photocentre point, and three-dimensional coordinate information is fed back to three-dimensional system of coordinate conversion module;
4th step: the three-dimensional coordinate that feedback is come is converted to the three bit space coordinate figures of random work target in base coordinate system by three-dimensional system of coordinate conversion module, and this three bit spaces coordinate figure is fed back to control module and reverse anti-solution joint of artificial limb Corner Block List Representation;
5th step: the random work target that reverse anti-solution joint of artificial limb Corner Block List Representation obtains trying to achieve from three-dimensional system of coordinate conversion module is when the three-dimensional information of working place, these 3 coordinate figures and certain are owed the input of compensatory attitude description as network model of definition, come fast with this, obtain concurrently certain of each joint of mechanical hand feasible solve value, each joint motor of driver drives adjusts to corresponding angle, makes arm-and-hand system hand to the target location at laser spots place;
6th step: system operators sends action command by controller, by control module, instruction is sent to driver module, by driver module complete capture target, unclamp target, lifting of doing evil through another person, decline of doing evil through another person, teaching target location associative operation.
Above-mentioned manipulator control system and method adopt extracts the signal source of the color character in binocular image as control mechanical hand, drawn the three-dimensional information of red feature laser spots in the real time imaging of the visual field by binocular vision system and three-dimensional system of coordinate conversion Calculation, control the operation that mechanical hand carries out the tracking target of man-machine interactive.When sending trace command and sending, the hand of mechanical hand just starts to follow the movement of laser spots and moves, when send capture target instruction target word time, mechanical hand captures target; When send unclamp target instruction target word time, mechanical hand unclamps target; When sending hand lifting instruction, the height of the vertical lifting of Manipulator Hand one setting; When sending hand decline instruction, Manipulator Hand vertically declines a height set; When sending teaching target instruction target word, Manipulator Hand is automatically moved to the good target location of certain teaching.
This control method can make up the deficiency of aforesaid robot the 3rd kind of mode of operation be operated in unstructured moving grids, the control tensity of personnel is reduced greatly, the wish of operator more easily describes and realizes, by the hot spot that Binocular Stereo Vision System identification positioning laser sends, mechanical hand can realize moving along arbitrary path easily, in conjunction with other control instruction of mechanical hand, easily control mechanical hand complete play chess, drink water, the action such as to have a meal.This control method is obviously different from again the 1st kind and the 2nd kind of mode of operation, it enhances operator's property of participation at any time, and is not only an onlooker.This control system and method system structure is simple, efficiency is high, precision is applicable, cost is low, it is convenient to control, wear hommization, and effectively can carry out real-time tracking crawl to the target object in motion, application is very broad, as artificial limb, explosive-removal robot, handicapped-resisting mechanical arm etc. of helping the elderly in intelligence.
Accompanying drawing explanation
Fig. 1 is the theory diagram of manipulator control system;
Fig. 2 is real time image collection device and laser guide schematic diagram, for deformed limb patient in figure;
Fig. 3 is ordinate transform and measuring principle schematic diagram;
Fig. 4 is control module figure;
Fig. 5 is the flow chart of method for controlling robot embodiment.
In figure: 1-video camera, 2-laser instrument, 3-artificial limb, 4-cup, 5-desk.
Detailed description of the invention
With reference to the accompanying drawings enforcement of the present invention is described further below.
Fig. 1 is present system theory diagram, as can be seen from the figure four major parts are comprised, respectively: real time image collection device, laser guide, the Programmable Logic Controller be made up of binocular stereo vision module, three-dimensional system of coordinate conversion module, reverse anti-solution manipulator joint Corner Block List Representation and control module, and driver module.Wherein:
Described real time image collection module embodiments as shown in Figure 2, it is the binocular camera 1 at the front A point place being placed on deformed limb patient, effect one is for man-machine Interactive control Robot actions provides live video stream, and the person that facilitates manipulator control obtains the field of view information in its front by video camera; Two is compositions binocular stereo vision module " eyes ", for the identification location of system provides real-time frame image.At this, assuming that real time image collection module is actionless at A place relative to breast abdomen, these two video cameras are obtained experimental data after demarcating, and gets that the suitableeest distance is parallel to be put together, and forms the eyes of binocular stereo vision module;
As shown in Figure 2, it is the laser instrument 2 at the left ear E point place being placed in people to described laser guide embodiment; Its major function is the cup 4 that locating and tracking is placed on desk 5, for binocular stereo vision module provides required feature hot spot.When carrying out man-machine interactive manipulation mechanical hand, when operator finds target object, open laser instrument, allow laser be mapped on target object, now laser instrument located target, lays a good groundwork for Manipulator Hand and artificial limb 3 hand accurately arrive target object location; If target object there occurs motion, now, operator can swing laser aiming module artificially, and laser spots is beaten on target object always, servo-actuated with target, so achieves man-machine interaction tracking target object.Mechanical hand is placed in the right arm place of people with disability, i.e. OGH section;
Described binocular stereo vision module, is use OPENCV function library, is programmed, in conjunction with the software that MFC does by VC.By real time image collection device recited above, obtain a certain amount of left images of camera system real-time synchronization shooting, then through software, left and right cameras individually demarcated, after binocular calibration, obtain inner parameter and the external parameter of left and right cameras; Moreover after the operations such as software successively carries out Image semantic classification to real time imaging, feature extraction, image Stereo matching, 3-d recovery, the degree of depth are determined, binocular stereo vision module identifies characteristic area, feature color, follow the tracks of, locates.Described binocular stereo vision module and the functional realiey of three-dimensional system of coordinate conversion module, we can in systems soft ware, see the world coordinates exporting target object relative shoulder joint node base coordinate system in real time, namely in fig. 2, the world three dimensional coordinate of the relative O point of F point.This binocular vision system module adopts carries out color recognition to the red laser facula of feature, follow the tracks of, after the laser instrument of laser aiming module recited above is to target object position and lock, the while of in the visual field of the left and right cameras of real time image collection device, red color visible laser spots F point (as shown in Figure 2) is in the frame figure that left and right is synchronous, after control system sends trace command, binocular vision system module identifies the same laser spots F point in the region of video camera shooting immediately, follow the tracks of, location, this module output red laser facula F point world three dimensional coordinate figure that is reference frame relative to video camera photocentre D point (as Fig. 3) left in real time image collection module at once simultaneously, when target object moves and laser spots is also followed, the relative world three dimensional coordinate figure that binocular vision system software continuous Output of laser point F is real-time, now, the D coordinates value of F point constantly changes, it reflects the movement locus of the relatively left video camera photocentre of laser spots.Finally, the three-dimensional world coordinate value of F point that Tracking Recognition goes out by Binocular Stereo Vision System feeds back to the three-dimensional system of coordinate conversion module that man-machine interactive controls in arm-and-hand system.
Described three-dimensional system of coordinate conversion module, Main Function be laser spots F point on the three-dimensional system of coordinate of the left video camera photocentre D point that Binocular Stereo Vision System module is obtained and target object three-dimensional system of coordinate between position orientation relation be converted to position orientation relation between base coordinate system and laser spots F point three-dimensional system of coordinate, finally draw the three-dimensional coordinate of laser facula F point comparative basis coordinate origin O point.As shown in Figure 3, known by description above, the pose that binocular vision system module has tried to achieve relative D point coordinates system of F point coordinates system describes, and obtains the D coordinates value of the relative D point of F point.CD and OB is parallel for left and right cameras photocentre line, at this, ignores some errors, then OBDC can regard the constant coplanar tetragon of size as.If the coordinate system direction vector of O, B, C, D in tetragon tetra-points is consistent, then the constant pure translation transformation of attitude has just done relative to O point coordinates system in D point coordinates system, and the just D point coordinates system initial point namely changed is relative to the change of reference frame O point coordinates system.Supposing that the position state of F point relative D point coordinates system describes pass is matrix T 1, the pass of D point coordinates system comparative basis coordinate system O point coordinates system is transformation matrix T2, then the pose of F point coordinates system relative O point coordinates system describes pass and is: T=T1 × T2, the three-dimensional information of the laser spots F relative D point coordinates system of then trying to achieve in conjunction with binocular vision module, easily tries to achieve the three-dimensional information of F point comparative basis coordinate system O point.Finally, the three-dimensional information of target object F point is fed back to reverse anti-solution manipulator joint Corner Block List Representation and control module by three-dimensional system of coordinate conversion module simultaneously, so that control system knows the three-dimensional information of target object in real time.
Described reverse anti-solution manipulator joint Corner Block List Representation, utilize neutral net to the nonlinear characteristic of the Parameter Mapping ability of Incomplete information uniqueness, fast parallel information processing capability and Multiinputoutput, set up a kind of solving model of special Multi-freedom-degreemanipulator manipulator joint space parameter.Its operation principle is as described in patent of invention the method for solving of joint of artificial limb spatial parameter " on the multiple degrees of freedom " (application number 201010280499.8) declared.When arm-and-hand system obtains three-dimensional information in working place of the random work target of trying to achieve from three-dimensional system of coordinate conversion module, these 3 coordinate figures and certain are owed the input of compensatory attitude description as network model of definition, come fast with this, obtain concurrently certain of each joint of mechanical hand feasible solve value, allow arm-and-hand system hand arrive the position at target object place.When control system module send tracking, capture, after the instruction such as to raise one's hand, mechanical hand respectively closes festival-gathering and will expand by the feasible value of the joint angle of trying to achieve or reduce angle, finally realizes the action such as tracking, crawl of mechanical hand to target.
As shown in Figure 4, described control module mainly contains tracking, captures target, unclamps target, hand lifting, hand decline, teaching target, stop seven control instructions.Recognition instruction mainly completes the identification to control instruction, after confirming instruction, sends the instruction identified to driver module.When manipulator's man-machine interactive operates machine hand system, the operational order needed for being sent by control module, arm-and-hand system modules carries out work according to corresponding instruction at once, has carried out the control to mechanical hand.
1). when control module sends trace command, binocular vision system module and three-dimensional system of coordinate conversion module will identify that the three-dimensional information Real-time Feedback of the laser spots oriented is to control system, driver module is then according to the feasible solution value that reverse anti-solution manipulator joint Corner Block List Representation returns, the angle in continuous each joint of adjustment mechanical hand, constantly position is adjusted to make the hand center of mechanical hand, realize following the tracks of the target object that laser instrument is specified, when the target object of the instruction of laser facula ceaselessly moves, the hand of mechanical hand will be followed the movement of laser spots and move.
2). when control module sends crawl target instruction target word, driver module can carry out the closed of joint by the hand motor of driving device hands, carries out grasping movement to target object.
3). when control module send unclamp target instruction target word time, driver module can carry out opening of joint by the hand motor of driving device hands, unclamps action to target object.
4). when control module sends hand lifting instruction, driver module then by the height of driving device hands hand lifting one setting, has carried out hand lifting action.
5). when control module sends hand decline instruction, driver module can be operated by the joint motor that driving device hands is relevant, the height of the vertical setting that declines.
6). when control module sends teaching target instruction target word, driver module can driving device hands to be correlated with joint motor running, makes Manipulator Hand move to the good target location of certain teaching.At this, the setting that teaching position can be random, its number is not unique, and this mainly meets some specific actions.
7). when control module sends halt instruction, driver module just immediately each joint motor of driving device hands shut down.And halt instruction has precedence over the identification of other instructions.
Described driver module comprises the motor of driver and driving device hands; When driver receives the action command that control module sends, can at once drive corresponding motor rotation, each joint angle of mechanical hand be regulated and controled, finally completes the control to mechanical hand.
The control method embodiment flow chart of the manipulator control system of the man-machine interactive tracking target of this binocular vision, as shown in Figure 5, it comprises following step: control system brings into operation; Real-time image acquisition; Target following order detects; Laser designation target; Binocular identification locating laser point; Target world coordinate system converts; The anti-Xie Jixieshouguanjiejiao of neutral net; Man-machine interactive controls mechanical hand; Control system quits work, and detailed step is as follows:.
First, step 1, control system brings into operation.Control people starts the control system that this man-machine interactive controls mechanical hand after finding that the random targets object in environment also confirms to carry out control operation.
Step 2, real-time image acquisition.After control system is opened, the binocular camera synchronous acquisition realtime image data in real time image collection module, both provides the field of view information of real-time target object place environment for controlling people, again for the instant operation of binocular vision system is laid a solid foundation.This step, when control system is run, needs to carry out always.
Step 3, target following order detects.When this man-machine interactive controls the system cloud gray model of mechanical hand, system time is in the detected state whether control module being sent to target following instruction.If do not receive the target following instruction that control system sends, then mechanical hand keeps certain original state, continues to wait for trace command information.If receive target following instruction, then enter step 4.
Step 4, laser designation target.When after system acceptance to target following instruction, the manipulation people carrying out human-computer interactive control mechanical hand needs manually to be opened by the laser instrument of laser aiming module, and laser is beaten on the target object that will follow the tracks of in twin camera visual field, for guiding function is played in the binocular identification location of following step 5.
Step 5, binocular identification locating laser point.After step 4 completes, Binocular Stereo Vision System module carries out color recognition and three-dimensional information location to the red feature laser spots on target object immediately, and at this, the three-dimensional information surveyed is the relatively left video camera photocentre of laser spots.Binocular vision system sends the three-dimensional information of output to target world coordinate system modular converter, namely carry out step 6.
Step 6, target world coordinate system converts.The D coordinates value of the relatively left video camera photocentre of the laser spots that binocular vision system imports into is the D coordinates value (specifically describing existing above) of laser spots comparative basis coordinate origin through matrixing formula scales.Then, the three-dimensional information of target is fed back to control system and reverse anti-solution manipulator joint Corner Block List Representation by this coordinate system transformation module.At this, be in order to operator better understands the movement locus of institute's tracking target object by the real-time three-dimensional information feed back of target to control system, simultaneously, carry out the operation of step 7 in real time.
Step 7, the anti-Xie Jixieshouguanjiejiao of neutral net.After the D coordinates value of the tracking target comparative basis coordinate origin drawn when step 6 feeds back to reverse anti-solution manipulator joint Corner Block List Representation, this module obtains the feasible value of each joint angle of mechanical hand immediately, then corresponding angle is adjusted in the running of each joint motor of driver drives, makes Manipulator Hand move to the target location at laser spots place.And arm end hand followed by laser spots and moves together.Finally, the real-time tracking to target object is achieved in the mode of man-machine cooperation.
Step 8, man-machine interactive controls mechanical hand.Manipulator control system, through above several step, achieves and locates (if make mechanical hand continue to keep tracking target state, then skip to step 3, carry out step cycle) the real-time tracking of random targets.Simultaneously, again by the collection of system to each joint position information of mechanical hand, and above-mentioned several step gained is controlled to the information feed back of output, operator carries out the control mechanical hand of man-machine interactive to realize corresponding action by control instruction, as controlled, mechanical hand realizes capturing target, crawl random targets target is unclamped, hand lifts, hand puts down, arrive teaching target location, stop mechanical hand etc.
(1). when operator sends crawl target instruction target word by control module, driver then drives the Manipulator Hand joint motor of tracking target object to operate, and makes mechanical hand carry out grasping movement to target object;
(2). when operator by control module send unclamp target instruction target word time, driver module just drives hand joint to open;
(3). when Manipulator Hand is in seized condition or open configuration, operator sends hand by control module and lifts instruction, hand decline instruction and teaching target instruction target word etc. and control Manipulator Hand and complete the action lifting, decline and arrive teaching target location;
(4) if. in control procedure, there is maloperation or control system itself produce mistake or mechanical hand everything must be stopped immediately in order to needs in operator, operator now sends the preferential halt instruction performed by control module, driver is ordered about all joint motors immediately and is shut down immediately, makes mechanical hand keep certain state.
(5) if. again receive instruction, then continue perform corresponding actions, when not receiving new instruction, mechanical hand performs current action always, until receive halt instruction.If operator wants to stop mechanical hand capturing the work event such as target, hand lifting, teaching target location, then can send halt instruction by control system, then its execution is made to return original state instruction, driver just drives each joint motor to operate immediately and regulates respective angle value, until each motor angle returns initial sizes values.
Step 9, control system quits work.By above eight steps, native system to binocular vision system module tracks, the random targets object of locating, identifying, carried out a series of control operation process, if think the work of finishing control system, then exited circulation, and then power-off.Now, before need ensureing system cut-off, mechanical hand is in init state, otherwise, bad hardware impact can be caused on arm-and-hand system pertinent modules.
The above embodiment only have expressed the specific embodiment of the present invention, and it describes comparatively in detail concrete, but can not be interpreted as the restriction to the scope of the claims of the present invention.It should be pointed out that for the person of ordinary skill of the art, without departing from the inventive concept of the premise, can also make some distortion and improvement, these all belong to protection scope of the present invention.Therefore, the protection domain of patent of the present invention should be as the criterion with appended claims.
Claims (2)
1. based on a man-machine interactive manipulator control system for binocular vision, it is characterized in that, it is made up of following four parts: real time image collection device, laser guide, programmable controller and driving device; Wherein:
(1) the real time image collection device described in is made up of two wireless camera machines or USB camera;
(2) laser guide described in wears frame by one and be located at the mini laser instrument worn on frame to form;
(3) programmable controller described in is made up of binocular stereo vision module, three-dimensional system of coordinate conversion module, reverse anti-solution manipulator joint Corner Block List Representation and control module; Wherein:
(4) binocular stereo vision module is used for beating laser instrument carrying out detection identification and location at the red laser point in wireless camera machine or the USB camera visual field;
(5) the three-dimensional system of coordinate conversion module relation be used between the video camera photocentre coordinate system that binocular stereo vision module recorded and laser point coordinates system is converted into the relation between the base coordinate system at mechanical hand top and target laser point world coordinate system;
(6) reverse anti-solution manipulator joint Corner Block List Representation is an artificial nerve network model trained, its Main Function is, after arm-and-hand system obtains the three dimensional space coordinate value of random work target in base coordinate system, owe the compensatory attitude description of definition as the input of network model using these three coordinate figures and certain, come fast with this, obtain concurrently certain of each joint of mechanical hand feasible solve value; The concrete training method of this module is:
The first step, according to the D coordinates value of random work target in working place, draw the structure coordinate graph of a relation of artificial limb on multiple degrees of freedom;
Second step, structure coordinate system relation according to artificial limb on multiple degrees of freedom, by its various parameter list, table comprises:
The torsion angle number of degrees α of upper each connecting rod of artificial limb
n
The common vertical line length a of upper each connecting rod of artificial limb
n,
The spacing d of upper artificial limb adjacent links
n;
Each joint variable and working range thereof, i.e. rotational angle range θ
n;
cos(α
n)，sin(α
n)；
3rd step: according to the α in above-mentioned list
n, α
n, d
n, θ
n, cos (α
n), sin (α
n) parameter, list A matrix and the T matrix of each rod member; Wherein each member coordinates of A matrix description is relative to the position of a coordinate system before it and attitude, T matrix description be the coordinate system of each rod member end of artificial limb in the position of working place and attitude, and the hand of artificial limb is in the position of working place and attitude;
4th step: according to the span in each joint in parameter list, A matrix and T matrix, scan the joint space of artificial limb, in scanning joint space process, the vector value of the joint space scanned i-th time is write as q
i, wherein i=1,2 ..., n;
5th step: the description of operation by human hand locus and attitude is simplified arrangement and is write as p
i(i=1,2 ..., n),
Simplifying method for sorting is:
That is: at p
i(i=1,2 ..., n), only remain hand Z
5the description of axle attitude and palm of the hand position;
6th step: after abundant n time scanning, the various interblock spaces in each joint of artificial limb or robot can be scanned by discretization, makes artificial limb be obtained by discretization in the position of working place and attitude; By aforesaid q
i(i=1,2 ..., n) { Q}, by aforesaid p for the set of data compilation composition
i(i=1,2 ..., the data compilation composition set { P} n); { Q} will be used as the target sample collection of training of human artificial neural networks in set; { P} will be used as the input amendment collection of training of human artificial neural networks in set;
7th step: { as the target sample collection of training of human artificial neural networks, { P}, as the input amendment collection of training of human artificial neural networks, to artificial neural network training, obtains the artificial neural network solving model trained to Q} with set with set; When the actual output of artificial neural network is
time, with error e} removes training of human artificial neural networks, wherein:
The result of training makes { e} → { 0};
8th step: by the input vector p of joint of artificial limb spatial parameter solving model on multiple degrees of freedom
nbe expressed as:
9th step: by p
ninput aforesaid artificial neural network solving model as input vector, obtain rapidly the movement angle in joint of artificial limb space on multiple degrees of freedom;
(7) control module be used for sending trace command, capture target instruction target word, unclamp target instruction target word, hand lifting instruction, hand decline instruction, teaching target instruction target word and halt instruction;
(8) driving device described in comprises the motor of driver and driving device hands; When driver receives the action command that control module sends, can at once drive corresponding motor rotation, each joint angle of mechanical hand be regulated and controled, finally completes the control to mechanical hand.
2., as claimed in claim 1 based on a control method for the man-machine interactive manipulator control system of binocular vision, it is characterized in that, step is as follows:
The first step: when operator need to operate certain target, opening control brings into operation, and image collecting device starts real-time image acquisition data;
Second step: control module sends trace command, operator open laser instrument, on the target object operated to needs by laser beam projects artificially, position, lay a good groundwork for Manipulator Hand accurately arrives target object location to target object; If target object there occurs motion, now, operator can swing laser guide artificially, and laser spots is beaten on target object always, servo-actuated with target, achieves man-machine interaction tracking target object;
3rd step: binocular stereo vision module carries out colour recognition and three-dimensional information location to the red feature laser spots that laser instrument is beaten on target object, namely three-dimensional coordinate measurement is carried out to the red feature laser spots identified, calculate the three-dimensional coordinate of the laser spots on target object relative to left video camera photocentre point, and three-dimensional coordinate information is fed back to three-dimensional system of coordinate conversion module;
4th step: the three-dimensional coordinate that feedback is come is converted to the three dimensional space coordinate value of random work target in base coordinate system by three-dimensional system of coordinate conversion module, and this three dimensional space coordinate value is fed back to control module and reverse anti-solution manipulator joint Corner Block List Representation;
5th step: the random work target that reverse anti-solution manipulator joint Corner Block List Representation obtains trying to achieve from three-dimensional system of coordinate conversion module is when the three-dimensional information of working place, these three coordinate figures and certain are owed the input of compensatory attitude description as network model of definition, come fast with this, obtain concurrently certain of each joint of mechanical hand feasible solve value, each joint motor of driver drives adjusts to corresponding angle, makes arm-and-hand system hand to the target location at laser spots place;
6th step: system operators sends action command by control module, by control module, instruction is sent to driving device, by driving device complete capture target, unclamp target, lifting of doing evil through another person, decline of doing evil through another person, teaching target location associative operation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310223530.8A CN103271784B (en) | 2013-06-06 | 2013-06-06 | Man-machine interactive manipulator control system and method based on binocular vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310223530.8A CN103271784B (en) | 2013-06-06 | 2013-06-06 | Man-machine interactive manipulator control system and method based on binocular vision |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103271784A CN103271784A (en) | 2013-09-04 |
| CN103271784B true CN103271784B (en) | 2015-06-10 |
Family
ID=49054012
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310223530.8A Expired - Fee Related CN103271784B (en) | 2013-06-06 | 2013-06-06 | Man-machine interactive manipulator control system and method based on binocular vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103271784B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20230013664A (en) * | 2021-07-15 | 2023-01-27 | 중앙대학교 산학협력단 | Prosthetic arm capable of automatic object recognition with camera and laser |
Families Citing this family (45)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103522291B (en) * | 2013-10-29 | 2016-08-17 | 中国人民解放军总装备部军械技术研究所 | The target grasping system of a kind of explosive-removal robot and method |
| CN104501736A (en) * | 2014-12-25 | 2015-04-08 | 安徽科鸣三维科技有限公司 | Machine vision automatic tracking control system based on laser source motion |
| CN104881026B (en) * | 2015-04-28 | 2017-11-24 | 国家电网公司 | High-tension line rushes to repair mechanical arm mobile route planning system and method |
| CN104942816B (en) * | 2015-04-28 | 2017-01-18 | 国家电网公司 | Automatic changing control system and automatic changing control method on basis of laser technologies for line dropping switches |
| CN104908030A (en) * | 2015-06-23 | 2015-09-16 | 江南大学 | SCARA robot accurate positioning method based on granularity model |
| CN106708028A (en) * | 2015-08-04 | 2017-05-24 | 范红兵 | Intelligent prediction and automatic planning system for action trajectory of industrial robot |
| CN106610666A (en) * | 2015-10-22 | 2017-05-03 | 沈阳新松机器人自动化股份有限公司 | Assistant robot based on binocular vision, and control method of assistant robot |
| WO2017130389A1 (en) * | 2016-01-29 | 2017-08-03 | 三菱電機株式会社 | Robot teaching device, and method for generating robot control program |
| JP6431495B2 (en) * | 2016-03-25 | 2018-11-28 | 本田技研工業株式会社 | Teacher data generation method |
| CN106003064A (en) * | 2016-06-17 | 2016-10-12 | 上海工程技术大学 | Complex-environment multi-sensor intelligent detection robot |
| CN106003076B (en) * | 2016-06-22 | 2019-01-01 | 徐琳 | A kind of plastic-spraying robot based on stereoscopic vision |
| CN105943207B (en) * | 2016-06-24 | 2017-12-05 | 吉林大学 | One kind is based on idiodynamic intelligent artificial limb kinematic system and its control method |
| CN106041993B (en) * | 2016-07-26 | 2018-08-31 | 中国人民解放军军械工程学院 | A kind of mechanical arm and its localization method being accurately positioned |
| CN106041937B (en) * | 2016-08-16 | 2018-09-14 | 河南埃尔森智能科技有限公司 | A kind of control method of the manipulator crawl control system based on binocular stereo vision |
| CN107917666A (en) * | 2016-10-09 | 2018-04-17 | 上海铼钠克数控科技股份有限公司 | Binocular vision device and coordinate scaling method |
| CN106695101A (en) * | 2017-01-23 | 2017-05-24 | 青岛同日机械电子有限公司 | High-frequency induction welding system based on binocular vision and multi-axis manipulator |
| CN106826822B (en) * | 2017-01-25 | 2019-04-16 | 南京阿凡达机器人科技有限公司 | A kind of vision positioning and mechanical arm crawl implementation method based on ROS system |
| EP3544773B1 (en) * | 2017-02-28 | 2022-08-31 | Siemens Industry Software Inc. | System and method for determining grasping positions for two-handed grasps of industrial objects |
| CN107045592B (en) * | 2017-03-29 | 2020-06-16 | 浙江大学 | Anthropological evaluation system for laparoscopic surgery operation |
| CN107020640A (en) * | 2017-04-28 | 2017-08-08 | 成都科力夫科技有限公司 | Robot interactive formula games system |
| JP6546618B2 (en) * | 2017-05-31 | 2019-07-17 | 株式会社Ｐｒｅｆｅｒｒｅｄ Ｎｅｔｗｏｒｋｓ | Learning apparatus, learning method, learning model, detection apparatus and gripping system |
| CN106975874A (en) * | 2017-06-02 | 2017-07-25 | 成都福莫斯智能系统集成服务有限公司 | The cutting robot control system of laser aiming |
| CN107272015A (en) * | 2017-07-05 | 2017-10-20 | 同济大学 | High-precision vision guides laser tracking |
| CN107953335B (en) * | 2017-12-24 | 2024-02-27 | 华南理工大学广州学院 | Behavior control system of robot |
| CN108247633B (en) * | 2017-12-27 | 2021-09-03 | 珠海格力节能环保制冷技术研究中心有限公司 | Robot control method and system |
| CN108502276A (en) * | 2018-03-30 | 2018-09-07 | 薛寓怀 | A kind of baling press and its working method with product testing function |
| CN108742957B (en) * | 2018-06-22 | 2021-02-09 | 上海交通大学 | Multi-sensor fusion artificial limb control method |
| CN108942929B (en) * | 2018-07-10 | 2020-11-27 | 广东电网有限责任公司广州供电局 | Mechanical arm positioning and grabbing method and device based on binocular stereo vision |
| CN109343363B (en) * | 2018-10-30 | 2020-09-22 | 清华大学 | Motion measurement and control system based on optical calculation |
| CN109352658B (en) * | 2018-12-04 | 2024-02-23 | 中冶赛迪工程技术股份有限公司 | Industrial robot positioning control method, system and computer readable storage medium |
| CN109623815B (en) * | 2018-12-19 | 2021-07-30 | 江苏科技大学 | Wave compensation double-robot system and method for unmanned salvage ship |
| CN109765090B (en) * | 2019-02-28 | 2020-06-02 | 中国石油大学(北京) | Device and method for accurately classifying and sampling streak layer structure of continental facies tight reservoir |
| CN109877802A (en) * | 2019-04-23 | 2019-06-14 | 宿州学院 | A kind of dedicated healing robot of handicapped patient |
| CN110340893B (en) * | 2019-07-12 | 2022-06-17 | 哈尔滨工业大学（威海） | Mechanical arm grabbing method based on semantic laser interaction |
| CN110347273B (en) * | 2019-07-12 | 2023-04-28 | 哈尔滨工业大学（威海） | Man-machine interaction method based on laser |
| CN110605714B (en) * | 2019-08-06 | 2021-08-03 | 华中科技大学 | Hand-eye coordination grabbing method based on human eye fixation point |
| CN112047113B (en) * | 2020-08-26 | 2022-02-22 | 苏州中科全象智能科技有限公司 | 3D visual stacking system and method based on artificial intelligence technology |
| CN112587285B (en) * | 2020-12-10 | 2023-03-24 | 东南大学 | Multi-mode information guide environment perception myoelectric artificial limb system and environment perception method |
| CN113180894B (en) * | 2021-04-27 | 2022-02-11 | 浙江大学 | Visual intelligence-based hand-eye coordination method and device for multiple-obstacle person |
| CN113977558B (en) * | 2021-11-29 | 2023-01-31 | 湖南交通职业技术学院 | Device and method for visually and dynamically displaying tail end track of parallel robot |
| CN115106312A (en) * | 2022-06-13 | 2022-09-27 | 中实洛阳重型机械有限公司 | Intelligent ore sorting device based on binocular camera laser guide |
| CN115410136B (en) * | 2022-11-01 | 2023-01-13 | 济钢防务技术有限公司 | Laser explosive disposal system emergency safety control method based on convolutional neural network |
| CN115624386B (en) * | 2022-12-20 | 2023-05-16 | 北京维卓致远医疗科技发展有限责任公司 | Holographic navigation system of binocular visible light camera |
| CN116175283B (en) * | 2023-04-26 | 2023-07-14 | 山东科技大学 | Online flatness detection platform and detection method based on multi-joint robot |
| CN117001715A (en) * | 2023-08-30 | 2023-11-07 | 哈尔滨工业大学 | Intelligent auxiliary system and method for visually impaired people |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101947150A (en) * | 2010-09-11 | 2011-01-19 | 山东科技大学 | Method for realizing space positioning of artificial limbs with multiple degrees of freedom to random targets |
| CN101953727A (en) * | 2010-09-11 | 2011-01-26 | 山东科技大学 | Solution method of joint space parameters of artificial limb in multiple degrees of freedom |
| CN102165880A (en) * | 2011-01-19 | 2011-08-31 | 南京农业大学 | Automatic-navigation crawler-type mobile fruit picking robot and fruit picking method |
| CN102452081A (en) * | 2010-10-21 | 2012-05-16 | 财团法人工业技术研究院 | Method and device for correcting system parameters of mechanical arm |
| CN102692214A (en) * | 2012-06-11 | 2012-09-26 | 北京航空航天大学 | Narrow space binocular vision measuring and positioning device and method |
| CN102871784A (en) * | 2012-09-21 | 2013-01-16 | 中国科学院深圳先进技术研究院 | Positioning controlling apparatus and method |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4262196B2 (en) * | 2004-12-14 | 2009-05-13 | 本田技研工業株式会社 | Autonomous mobile robot |
| US8082064B2 (en) * | 2007-08-24 | 2011-12-20 | Elite Engineering Corporation | Robotic arm and control system |
-
2013
- 2013-06-06 CN CN201310223530.8A patent/CN103271784B/en not_active Expired - Fee Related
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101947150A (en) * | 2010-09-11 | 2011-01-19 | 山东科技大学 | Method for realizing space positioning of artificial limbs with multiple degrees of freedom to random targets |
| CN101953727A (en) * | 2010-09-11 | 2011-01-26 | 山东科技大学 | Solution method of joint space parameters of artificial limb in multiple degrees of freedom |
| CN102452081A (en) * | 2010-10-21 | 2012-05-16 | 财团法人工业技术研究院 | Method and device for correcting system parameters of mechanical arm |
| CN102165880A (en) * | 2011-01-19 | 2011-08-31 | 南京农业大学 | Automatic-navigation crawler-type mobile fruit picking robot and fruit picking method |
| CN102692214A (en) * | 2012-06-11 | 2012-09-26 | 北京航空航天大学 | Narrow space binocular vision measuring and positioning device and method |
| CN102871784A (en) * | 2012-09-21 | 2013-01-16 | 中国科学院深圳先进技术研究院 | Positioning controlling apparatus and method |
Non-Patent Citations (2)
| Title |
|---|
| 基于Hamilton的机器人双目视觉伺服控制及观测器设计;杨波;《中国优秀博士学位论文全文数据库》;20130515(第5期);正文第1-91页 * |
| 基于神经元网络算法的六自由度手臂机器人的控制研究;许胜善;《中国优秀硕士学位论文全文数据库》;20080915(第9期);正文第1-60页 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20230013664A (en) * | 2021-07-15 | 2023-01-27 | 중앙대학교 산학협력단 | Prosthetic arm capable of automatic object recognition with camera and laser |
| KR102582861B1 (en) * | 2021-07-15 | 2023-09-27 | 중앙대학교 산학협력단 | Prosthetic arm capable of automatic object recognition with camera and laser |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103271784A (en) | 2013-09-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103271784B (en) | Man-machine interactive manipulator control system and method based on binocular vision | |
| CN111906784B (en) | Pharyngeal swab double-arm sampling robot based on machine vision guidance and sampling method | |
| CN106737673B (en) | A method of the control of mechanical arm end to end based on deep learning | |
| Katz et al. | Manipulating articulated objects with interactive perception | |
| CN108499054B (en) | A kind of vehicle-mounted mechanical arm based on SLAM picks up ball system and its ball picking method | |
| CN108202316A (en) | A kind of crusing robot and control method of automatic switch cabinet door | |
| CN109454638A (en) | A kind of robot grasping system of view-based access control model guidance | |
| CN107253192A (en) | It is a kind of based on Kinect without demarcation human-computer interactive control system and method | |
| CN109910011A (en) | A kind of mechanical arm barrier-avoiding method and mechanical arm based on multisensor | |
| JP2011110620A (en) | Method of controlling action of robot, and robot system | |
| CN109968310A (en) | A kind of mechanical arm interaction control method and system | |
| WO2024027647A1 (en) | Robot control method and system and computer program product | |
| Kumar et al. | Computer vision based object grasping 6DoF robotic arm using picamera | |
| Stückler et al. | Adaptive tool-use strategies for anthropomorphic service robots | |
| Yang et al. | Visual servoing control of baxter robot arms with obstacle avoidance using kinematic redundancy | |
| Vahrenkamp et al. | Visual servoing for dual arm motions on a humanoid robot | |
| Luck et al. | Real-time tracking of articulated human models using a 3d shape-from-silhouette method | |
| Galbraith et al. | A neural network-based exploratory learning and motor planning system for co-robots | |
| Gao et al. | Kinect-based motion recognition tracking robotic arm platform | |
| CN208020198U (en) | A kind of intelligent grabbing robot based on RGBD | |
| Wang et al. | A visual servoing system for interactive human-robot object transfer | |
| Tooyama et al. | Development of an assistive system for position control of a human hand with high speed and high accuracy | |
| Khokar et al. | Laser-assisted telerobotic control for enhancing manipulation capabilities of persons with disabilities | |
| Angel et al. | Vision-based control of the robotenis system | |
| Tsitos et al. | Handling vision noise through robot motion control in a real-time teleoperation system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20150610 Termination date: 20190606 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |