CN107581712A - Have the automation shoelace threading machine and its computer vision householder method of computer vision auxiliary - Google Patents

Have the automation shoelace threading machine and its computer vision householder method of computer vision auxiliary Download PDF

Info

Publication number
CN107581712A
CN107581712A CN201610751339.4A CN201610751339A CN107581712A CN 107581712 A CN107581712 A CN 107581712A CN 201610751339 A CN201610751339 A CN 201610751339A CN 107581712 A CN107581712 A CN 107581712A
Authority
CN
China
Prior art keywords
piece
hole
mechanical arm
shoe
lace hole
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201610751339.4A
Other languages
Chinese (zh)
Other versions
CN107581712B (en
Inventor
张文堡
徐伟信
柯昭全
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Pou Chen Corp
Original Assignee
Pou Chen Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Pou Chen Corp filed Critical Pou Chen Corp
Publication of CN107581712A publication Critical patent/CN107581712A/en
Application granted granted Critical
Publication of CN107581712B publication Critical patent/CN107581712B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Landscapes

  • Footwear And Its Accessory, Manufacturing Method And Apparatuses (AREA)

Abstract

A kind of automation shoelace threading machine for having computer vision auxiliary, is applicable a shoestring being arranged in a shoes.The shoestring has one section of flexible section, and two extremity pieces for being arranged at the flexible section both ends severally.The shoes have two panels lace hole piece, and are arranged at multiple lace holes per a piece of lace hole piece.The automation shoelace threading machine includes a frame, at least one mechanical arm, and a visual feedback unit.The frame puts footwear apparatus, and a clamping device for being used to press from both sides the lace hole piece of pulling including one.This is put footwear apparatus and puts shoe with what the confession shoes were nested with.The mechanical arm includes a clamping jaw module that can carry out pressing from both sides drawing or unclasping to a wherein extremity piece, and one can drive the horn that the clamping jaw module puts shoe movement relative to this.The visual feedback cell signal connects the mechanical arm and including an image acquisition module.The image acquisition module is used for the image for having the lace hole relevant position towards a lace hole piece acquisition at least width.

Description

Have the automation shoelace threading machine and its computer vision auxiliary of computer vision auxiliary Method
Technical field
The present invention relates to a kind of automatic processing machine and its householder method, more particularly to one kind tool computer vision auxiliary Automation shoelace threading machine and computer vision householder method.
Background technology
As global industry automatic technology increasingly improves, robot can little by little substitute people with computer processing platform Power is worked.But for shoe industry, because the shoes appearance difference to each other of different styles is very big, and per money shoes Also different size model numbers is suffered from so that footwear industry is also difficult to reach full-automatic operation, is still required for relying on a large amount of people The input of power.Therefore, under comparative advantage and the principle of the whole world division of labor, footwear industry can only constantly and manpower abundant toward labour The cheap area transfer of cost.
But, improve shoemaking efficiency and reduce human cost in face of labour's no longer cheap future at last, can assist in Automatic machines and equipment demand be extremely urgent.Especially this road of wear shoes band it is technical it is low, repeatability is high processed Journey, it is often necessary to configure most manpowers, not only waste of manpower resource, and influence production cost and processing efficiency one it is big because Element.So footwear industry there's no one who doesn't or isn't wishes to automatically to complete wear shoes band this procedure.Therefore, just like Chinese invention patent Shen A kind of automatic shoelace threading machine that please be disclosed in case No. 201210155160.4 and No. 201410117490.3, using multiple imitative One shoestring is arranged in multiple footwear holes of the footwear part by the mechanical arm of effect human hand in order, to reach wear shoes with the automatic of operation Change.
However, the automatic shoelace threading machine, which must all obtain, first ensures that the footwear hole is arranged on the punch position planned, The mechanical arm could be allowed successfully to carry out wear shoes band operation, so storing of the operating personnel from side to the footwear part still must be needed Ornaments are adjusted, and lack processing elasticity.In addition, the coordinative operation of the mechanical arm to each other can only also lean on the servo of motor Feed back to be controlled by, for this such trickle operation, laser intensity and control required for the mechanical arm as wear shoes band Precision processed certainly will all can it is padded its set cost.
The content of the invention
One of purpose of the present invention is to provide a kind of automation shoelace threading machine for having computer vision auxiliary so that Mechanical arm in equipment can optimize efficiency and flow of the wear shoes with operation, and can effectively carry by the feedback of computer vision The manipulation precision and efficiency of the high mechanical arm.
The automation shoelace threading machine of present invention tool computer vision auxiliary, is applicable a shoestring being arranged in a shoes. The shoestring has one section of flexible section, and two extremity pieces for being arranged at the flexible section both ends severally.The shoes have two panels shoestring Hole piece, and it is arranged at multiple lace holes per a piece of lace hole piece.The automation shoelace threading machine includes a frame, at least one Mechanical arm, and a visual feedback unit.
The frame puts footwear apparatus, and a clamping device for being used to press from both sides the lace hole piece of pulling including one.This puts footwear dress Put and put shoe with what the confession shoes were nested with.
The mechanical arm includes a clamping jaw module that can carry out pressing from both sides drawing or unclasping to a wherein extremity piece, and one can drive this Clamping jaw module puts the horn of shoe movement relative to this.
The visual feedback cell signal connects the mechanical arm and including an image acquisition module.The image acquisition module is used There is the image of the lace hole relevant position in capturing an at least width towards the lace hole piece.
It is preferred that the frame also includes a microscope carrier, and a slide rail for being arranged at the microscope carrier top surface.This puts footwear apparatus also With a slide that can be slippingly arranged at the slide rail.This is put shoe and is disposed on the slide one end.This puts shoe with this Slip of the slide on the slide rail, and can be relative to the microscope carrier between an automated job position and an artificial operating position It is mobile.When this put shoe together with the shoes in the automated job position when, clamping device folder is pulled the lace hole piece, together When, the image acquisition module captures the image towards the lace hole piece, and makes the visual feedback unit drive the mechanical arm to drive The extremity piece wears the lace hole in order.When this puts shoe in the manual operation position, operator can remove and wear The shoes of the shoestring, or another is not yet worn to the shoes of shoestring and is placed on this and puts shoe.
It is preferred that the frame also includes a frame plate.This is put footwear apparatus and is disposed on the frame plate.When operator will not yet wear If the shoes of shoestring are placed on this when putting footwear apparatus, the visual feedback unit drives the mechanical arm to drive the extremity piece to wear in order The lace hole, then the shoes for having worn the shoestring are removed by operator.
It is preferred that the shoes also have a flap between the lace hole piece, and a piece of it is arranged at the flap top Face, and can bend formed one pass around hole pass around piece.The frame also includes a hook tongue device.The hook tongue device has one Pedestal, one be arranged at the driver of the pedestal, a trace, an and hook ring module for including a hook ring part.The hook Ring is to be used to from this passing around hole side to be arranged in this and pass around hole.The both ends of the trace are linked in the driver and the hook respectively Ring moulds block so that the trace is driven by the driver, and shoe can be put relative to this lifting position and one Pivot is put between pressure drop position.When the trace is in the lifting position, the hook ring part drives the flap to put shoe away from this.When the company At the pressure drop position, the hook ring part drives the flap to press to this and puts shoe lever.
It is preferred that the hook ring module also including one be arranged at the connecting block of the trace one end, one be arranged at the company Lump the reset assembly of side, and one be connected with the reset assembly, and its one end is for the sliding piece of hook ring part setting.Should Sliding piece can wear position at one relative to the connecting block and non-wears is moved between position.When the sliding piece is worn at this During position, the hook ring part is jointly moved to that to pass around piece relative up and down with this, allows an operator to the hook ring part passing around hole from this Side is arranged in this and passed around in hole.When the sliding piece this it is non-wear position when, the hook ring part is jointly moved to this and passes around hole Side, and fully pass this and pass around hole.The reset assembly is to wear position for making the sliding piece perseverance be returned to this.
It is preferred that the hook ring part has the through hole of an insertion left and right sides, and a fluting for connecting the through hole.When this Trace enables the mechanical arm to drive the extremity piece to be arranged in the through hole at the lifting position, in addition, the flexible section can be by this The formed breach of fluting, and depart from the hook ring part.
It is preferred that the clamping device has two plier groups for being respectively used to the corresponding lace hole piece of folder tug.Each clamp Group includes one and is arranged at the interior clamp for putting shoe wherein side, and one can be relative to the interior clamp in a clip position The outer clamp moved between a non-clamping position.When the outer clamp is in the clip position, wherein a piece of lace hole piece is pressed from both sides Support is stood between the interior clamp and the outer clamp.When the outer clamp is in the non-clamping position, the outer clamp away from the interior clamp, Enable the shoes to put from this to remove or be nested with footwear apparatus.
It is preferred that the image acquisition module has at least one video camera.The photography function is arranged at the frame and the machine The one of which of tool arm.
It is preferred that the visual feedback unit also includes a light source module.The light source module has at least one light fixture.Should Light fixture can be arranged at the one of which that the frame puts shoe with this.Light quilt through the lace hole is diffused by the light fixture Capture on the image, make image feature intensity increase of the lace hole in the image, to lift the vision computing module To the analytic operation efficiency of the lace hole coordinate.
It is another object of the present invention to provide it is a kind of automate shoelace threading machine computer vision householder method, comprising The following steps:
(A) capturing a width has the image of multiple lace hole relevant location informations.
(B) by three-dimensional reconstruction algorithm or the two dimensional image signature analysis process analysis image, and the lace hole is found out Locus.
(C) servo feedback signal of a mechanical arm is relevant to according to one, and finds out posture and the action of the mechanical arm.
(D) it is relevant to a shoestring according to one and wears the controlling planning progress visual feedback of the lace hole, and exports One control signal for being relevant to the feedback result is to the mechanical arm.
The beneficial effects of the present invention are:The visual feedback carried out by the visual feedback unit to the mechanical arm, no It is only capable of as the mechanical arm in wear shoes with the operation planning in operation, the efficiency and flow for making operation are optimised, and are coordinated The servo feedback signal, can further strengthen the manipulation precision and efficiency of the mechanical arm, and effectively lift the property of the mechanical arm Can price ratio.
Brief description of the drawings
Other features and effect of the present invention, will clearly be presented in the embodiment with reference to schema, wherein:
Fig. 1 is a stereogram of one embodiment of the automation shoelace threading machine of present invention tool computer vision auxiliary, Illustrate that a shoestring is arranged in a shoes by the embodiment mechanical arm;
Fig. 2 is analogous to Fig. 1, but omits the mechanical arm and an imperfect stereogram of the shoestring, illustrates the embodiment In a frame, and two video cameras and two light fixtures of a visual feedback unit, also, a hook ring part is worn at one If position, and be arranged in the shoes one passes around hole;
Fig. 3 is analogous to Fig. 2, but eliminates an imperfect stereogram of the shoes, illustrates the light fixture another Embodiment aspect, and be arranged at one and put footwear apparatus, and can be by the shoes inside-out printing opacity;
Fig. 4 is analogous to Fig. 2, but eliminates the shoes and an imperfect stereogram of the visual feedback unit, explanation This puts two interior clamps of footwear apparatus, clamping device, and one hooks tongue device relative to a microscope carrier in an artificial behaviour Make position;
Fig. 5 is analogous to a Fig. 4 imperfect stereogram, illustrates that this puts footwear apparatus, the interior clamp, and hook tongue dress Put relative to the microscope carrier an automated job position, also, an operating parts of the frame is a stretching position;
Fig. 6 is an imperfect stereogram, illustrates that this puts footwear apparatus, and another embodiment aspect of the clamping device;
Fig. 7 is analogous to a Fig. 5 imperfect stereogram, illustrates the operating parts in a disengaged position;
Fig. 8 is a Fig. 1 part amplification stereogram, illustrates that the shoestring is relevant in the embodiment is arranged in the shoes Mechanism assembly;
Fig. 9 is a Fig. 8 part amplification stereogram, illustrate the hook ring part one it is non-wear position, and fully wear Go out this and pass around hole;
Figure 10 is a signal flow graph, illustrates the signal transitive relation between the mechanical arm and the visual feedback unit.
Embodiment
Before the present invention is described in detail, it shall be noted that the spare part in schema might not be drawn to scale.
Refering to Fig. 1 and Fig. 9, one embodiment of the automation shoelace threading machine of present invention tool computer vision auxiliary, is suitable For a shoestring 700 to be arranged in into a shoes 800.Wherein, the shoestring 700 has one section of flexible section 710, and two each Ground is arranged at the extremity piece 720 at the both ends of flexible section 710.The shoes 800 have two panels lace hole piece 810, one between the footwear Flap 820 between with holes 810, it is a piece of be arranged at the top surface of flap 820, and formation one can be bent and pass around passing around for hole 831 Piece 830, and it is arranged at multiple lace holes 840 per a piece of lace hole piece 810.
Cooperation refers to Figure 10, and the embodiment includes 100, mechanical arms 500 of a frame, and a visual feedback unit 600。
Cooperation refers to Fig. 2 and Fig. 4, and the frame 100 is arranged at the frame plate including a horizontally disposed frame plate 110, one The slide rail 130, one that the microscope carrier 120, one of 110 sides is arranged at the top surface of microscope carrier 120 puts 200, clamping dresses of footwear apparatus Put 300, and one and hook tongue device 400.
Wherein, coordinate and refer to Fig. 5, this, which puts footwear apparatus 200, has a slide that can be slippingly arranged at the slide rail 130 210, and one be arranged at the one end of slide 210 put shoe 220.In the present embodiment, this is put shoe 220 and surrounded including one Form the pedestal 221 of a seat chamber 222, and it is multiple be formed at the left and right sides of the pedestal 221, and connect the perforation of this chamber 222 223.Also, because the top surface of the microscope carrier 120 is to favour the frame plate 110 so that this puts shoe 220 as the slide 210 exists Slip on the slide rail 130, and can be relative to the microscope carrier 120 between an automated job position and an artificial operating position It is mobile.When the slide 210 slides down to the bottom of the slide rail 130 so that this is put shoe 220 and is located at directly over the frame plate 110 When, i.e., in the automated job position, as shown in Fig. 2 to carry out the band operation of the wear shoes of the mechanical arm 500;And work as the slide 210 Along the upward sliding of slide rail 130 so that when this puts shoe 220 away from the frame plate 110, i.e., in the manual operation position, such as Fig. 4 institutes Show, to receive the manual operation of operator.
Refering to Fig. 2, Fig. 4 and Fig. 5, the clamping device 300 has two plier groups, 310, wrench modules 320, and two Individual driver 330.Each plier groups 310 is arranged at the interior clamp 311 for putting the wherein side of shoe 220, and one including one It is individual to be driven by one of driver 330, and can be relative to the interior clamp 311 in a clip position and a non-clamping position Between the outer clamp 312 that moves.The wrench module 320 is arranged at including a seat 321, one of pulling for being arranged at the slide 210 This pulls the link assembly 322 of seat 321, and a wedge-shaped component 323 driven by the link assembly 322.The link assembly 322 With an operating parts 324 that seat 321 can be pulled relative to this and is moved between a stretching position and a disengaged position, and one Root interlocks with the phase of operating parts 324, and the push rod 325 that can be moved along a first axial L1.The wedge-shaped component 323 has one It is arranged at the trapezoid block 326 of the one end of push rod 325, and two wedges 327 for being each located on the both sides of trapezoid block 326.
It is noted that in the present embodiment, the interior clamp 311 is respectively to be arranged in the perforation by multiple 223 pilot assembly 328, and be connected with the wedge 327.Moreover, the wedge 327 is disposed in this chamber 222, And multiple elastic components 329 by being respectively set in the pilot assembly 328 provide it is butted in the pedestal 221 and the wedge 327 elastic force so that the wedge 327 can drive the interior clamp 311 to be moved towards the pedestal 221 by the elastic force.Its In, the pilot assembly 328 and the quantity of the elastic component 329 are not limited with embodiment, also can be respectively one, two or three More than;In addition, the elastic component 329 is also not limited to the spring presented such as the present embodiment, but bullet can be provided comprising all The elastomer of power potential energy.In addition, the trapezoid block 326 is engaged with the wedge 327 so that the trapezoid block 326 is pushed away by this Bar 325 drives, and along the first axial L1 move when, the wedge 327 just can be along the vertical first axial L1, and each other Move in the opposite direction.Therefore, when the operating parts 324 is in the stretching position, the drive of the push rod 325 is passed through so that the ladder Shape block 326 puts the direction movement of shoe 220 towards this, and pushes the wedge 327, and then allows the interior clamp 311 each to lean on It is butted on the inner side of the lace hole piece 810;When the operating parts 324 is in the disengaged position, by the drive of the push rod 325, make Obtain the trapezoid block 326 towards this pull seat 321 direction movement, meanwhile, the wedge 327 by the elastic component 329 bullet Power, and drive the interior clamp 311 to be each away from the inner side of the lace hole piece 810.So when the shoes 800 are placed on When this puts shoe 220, just it can be adjusted by the wrench module 320 between the interior clamp 311 and the inner side of the lace hole piece 810 Distance.
On the other hand, the outer clamp 312 is in the drive by the driver 330, and each towards corresponding inner clip When pincers 311 are moved to the clip position, the lace hole piece 810 stands on clamped support the interior clamp 311 of the plier groups 310 Between outer clamp 312;Also, the lace hole 840 on per a piece of lace hole piece 810 is each attached to one and passed around on face.Such as This, just can allow the mechanical arm 500 perform wear shoes band operation when, the lace hole 840 will not be because of the lace hole piece 810 It is flexible and produce beat, and the visual feedback unit 600 can be allowed not have to repeat and be continued for feedback control computing, with effective Ground lifts control efficiency of the visual feedback unit 600 to the mechanical arm 500.In addition, when the outer clamp 312 is by described The drive of driver 330, and during respective remote corresponding interior clamp 311, as in the non-clamping position.
Refering to Fig. 7, Fig. 8 and Fig. 9, there is the hook tongue device 400 pedestal for being arranged at the slide 210 410, one to set It is placed in 420, traces 430 of driver of the pedestal 410, and a hook ring module 440.The both ends of the trace 430 point The driver 420 and the hook ring module 440 are not connected to so that the trace 430 is by the drive of the driver 420, and energy The pivot between a lifting position and a pressure drop position of shoe 220 is put relative to this to put.The hook ring module 440 includes a setting The reset assembly 442, one that connecting block 441, one in the one end of trace 430 is arranged at the side of connecting block 441 is with being somebody's turn to do The sliding piece 443 that reset assembly 442 is connected, and a hook ring part 444 for being arranged at the one end of sliding piece 443.
It is noted that in the present embodiment, the reset assembly 442 is to be connected to the company by a catch 445, two Pilot assembly 446 between caking 441 and the catch 445, and two elastic components 447 for being each sheathed on the pilot assembly 446 are formed; Moreover, the sliding piece 443 is by the pilot assembly 446, and position and one can be worn at one relative to the connecting block 441 Non- wear is moved between position;And the elastic component 447 is all butted in the catch 445 and the sliding piece 443 so that this is stirred Piece 443 can be returned to this by the elastic force perseverance of the elastic component 447 and wear position.Wherein, the pilot assembly 446 and the elasticity The quantity of part 447 is not limited, and also can be respectively one or more;In addition, the elastic component 447 be also not limited to as The spring that the present embodiment is presented, but include all elastomers that elastic force potential energy can be provided;Also, the reset assembly 442 It is not limited only to the aspect presented such as the present embodiment, but the perseverance of sliding piece 443 can be allowed to be returned to this wear position comprising all Mechanism or device, such as pulsometer, motor, or even can also be interlocked with the wrench module 320, make the operating parts 324 band simultaneously Move the interior clamp 311 and the sliding piece 443.In addition, the hook ring part 444 has the through hole 448 of an insertion left and right sides, and One connects the through hole 448, and the fluting 449 of breach upward, and causes the section of the hook ring part 444 to be in generally C fonts, but not with This is limited, also can be U-shaped, or under narrow wide other aspects such as trapezoidal.In this way, when the sliding piece 443 wears position at this When putting, the hook ring part 444 also can jointly be moved to this pass around piece about 830 it is relative, allow an operator to the hook ring part 444 The side of hole 831 is passed around from this be arranged in this pass around in hole 831.Now, the hook ring part 444 just can pass around piece 830 to this and support , the size of hole 831 is passed around so as to fix this, hole 831 is passed around to facilitate the mechanical arm 500 to drive the extremity piece 720 to be arranged in this;And And the relation for just thering is connection to drive between the hook ring part 444 and the flap 820.Therefore, when the trace 430 is in the lifting position When putting, the hook ring part 444 drives the flap 820 to put shoe 220 away from this;When the trace 430 is in the pressure drop position, the hook Ring 444 drives the flap 820 to press to this and puts shoe 220.When the sliding piece 443 this it is non-wear position when, the hook ring part 444 Also it can jointly be moved to this and pass around the side of hole 831, and fully pass this and pass around hole 831.Now, the hook ring part 444 is with being somebody's turn to do Just the relation of connectionless drive between flap 820.
Refering to Fig. 1 and Figure 10, the mechanical arm 500 includes 510, horns 520 of a clamping jaw module, and a controller 530.The controller 530 is used to receive a control signal, and parses the control signal, with respectively to the clamping jaw module 510 and The horn 520 exports a pen clip discharge signal and a drive signal.The clamping jaw module 510 is folded up signal control by this, and is used for A wherein extremity piece 720 is carried out pressing from both sides drawing or unclasped.The horn 520 is controlled by the drive signal, and can drive the clamping jaw module 510 Shoe 220 is put relative to this to move.Also, the controller 530 is according to the clamping jaw module 510 and the sensing component of the horn 520, Encoder or limit switch such as on servo motor, find out the space for folding up state and the horn 520 of the clamping jaw module 510 Posture, and export one and be relevant to the servo feedback signal for finding out result.It is noted that in the present embodiment, the machine The quantity of tool arm 500 is one, but also can should not be more than one as limit;Moreover, the clamping jaw module 510 is by a pair Clip claw mechanism 511 is formed, and is driven simultaneously by the horn 520.Wherein, the clip claw mechanism 511 can pass through press pump or electricity The modes such as power driving are clamped or unclasped;However, this mechanism assembly for known those skilled in the art institute can be known, and it is non-for the present invention Technical characteristic;So the composition and action of the clip claw mechanism 511 are not illustrated herein.Focus on, this implementation The mechanical arm 500 of example is to be used to imitate human hand to capture the extremity piece 720, and in order can be arranged in the extremity piece 720 described Lace hole 840.Therefore, the framework without departing substantially from the mechanical arm 500 and action purpose in the case of, to the mechanical arm 500 The outward appearance change of progress and setting adjustment, should all be protected according to the claim of the present invention.
Refering to Fig. 1, Fig. 5 and Figure 10, the visual feedback unit 600 includes 610, vision fortune of an image acquisition module Calculate 620, feedback control modules 630 of module, and a light source module 640.It is noted that in the present embodiment, the shadow The frame plate 110 is arranged at as acquisition module 610 has two, and respectively can put shoe towards positioned at the automated job position 220 capture the video camera 611 of an at least width flat image;And the light source module 640 has two and is arranged at the frame plate 110, And can be respectively towards the light fixture 641 for putting the irradiation of shoe 220 positioned at the automated job position.
It is noted that in the present embodiment, the quantity of the video camera 611 and the light fixture 641 is all two, but not Also can should be one or more as limit.Moreover, the light fixture 641 is to be diffused using it through the lace hole 840 Light be subtracted on the flat image, make the lace hole 840 in the flat image image feature intensity increase, with Lift analytic operation efficiency of the vision computing module 620 to the coordinate of lace hole 840.Therefore, the visual feedback unit 600 It it is not absolutely required to the light source module 640;It is preferred that be by the efficiency of operating environment light source, the image acquisition module 610, and Accounted in terms of the accuracy of the vision computing module 620 analysis, and decide whether to set the light source module 640.In addition, As shown in figure 3, another light source module 640 ' equally has two light fixtures 641 ', but the light fixture 641 ' is then disposed on this and put Shoe 220, and can be to the printing opacity of lace hole 840, this embodiment aspect is equally to allow the video camera 611 to be easily discernible in footwear Go out the lace hole 840.In addition, the setting of the video camera 611 is to be relevant to the calculation of 620 bases of vision computing module Method-three-dimensional reconstruction (3D Reconstruction);Also, it is using " hand (Eye-to-Hand) soon " in the present embodiment Framework, the i.e. mechanical arm 500 act independently of one another with the video camera 611, but are not limited, also can be using " eye is in hand (Eye-in-Hand) framework ", will the video camera 611 be arranged on the mechanical arm 500, and can be with the mechanical arm 500 It is mobile.On the other hand, existing considerable document and research are delivered, so the computing flow not to the algorithm is further explained herein State.
But, it is worth mentioning at this point that, the present invention is by three-dimensional reconstruction algorithm, and it is alive to find out the lace hole 840 Three-dimensional coordinate in boundary's coordinate system, and coordinate the servo feedback signal to plan the operation of the mechanical arm 500, this is relevant to output The control signal of program results enables the mechanical arm 500 to drive clamped extremity piece 720 to wear in order to the mechanical arm 500 If the lace hole 840, that is to say, that a loop circuit (Close is formed between the mechanical arm 500 and the visual feedback unit 600 Loop) system.But by three-dimensional reconstruction algorithm, it can similarly obtain three-dimensional seat of the extremity piece 720 in world coordinate system Mark.Therefore, if the coordinate information is combined with the control framework of the mechanical arm 500, just can not have to according to the servo feedback Signal carries out feedback control, but directly exports the control signal to the mechanical arm 500 so that the mechanical arm 500 regards with this Loop (Open Loop) system is opened in formation one between feeling feedback unit 600.However, the closed loop system opens loop compared to this System can not only correct error so that control accuracy is higher, and the stability of system and anti-interference are also preferable.These characteristics It is indispensable and important, so the present embodiment uses the loop circuit for the wear shoes band operation that the mechanical arm 500 is carried out The framework of system.
Whereby, this is put shoe 220 and is first moved to the manual operation position, and allows the pivot of operating parts 324 to put to this and release position Put, allow operator the shoes 800 can be placed on into this and put shoe 220.Then, the hook ring part 444 is arranged in this by operator again Pass around in hole 831, and the pivot of operating parts 324 is put to the stretching position, the interior clamp 311 is each abutted against in the shoestring The inner side of hole piece 810.On the other hand, operator must be according to the design of the lace hole 840, to decide whether the shoestring first 700 are arranged in some lace holes 840 manually.For example, the lace hole 840 of most leading edge is usually located at because being adjacent Junction between vamp and the flap 820 so that even if positioned at lace hole 840 herein by the folder of the clamping device 300 Tug, it is also difficult to and other lace holes 840 are passed around on face with this.Therefore, can be by operator first by the shoestring 800 in practice The lace hole 840 of most leading edge is arranged in, and suitably adjusts the flexible section 710 so that the flexibility passed by the lace hole 840 Section 710 to the extremity piece 720 equal length.
Come again, this is put shoe 220 by the slide rail 130 and is pushed into the automated job position together with the shoes 800 by operator Put;Meanwhile the driver 330 of the clamping device 300 also can by the operation of operator or the signal output of other control units, And the outer clamp 312 is moved to the clip position, allow the clamped support of the lace hole piece 810 to stand on corresponding pass around Face.Then, the clamping jaw module 510 clamps a wherein extremity piece 720, proceeds by wear shoes band operation.
In the process, the vision computing module 620 carries out analytic operation to the flat image, and with output, one is relevant to The coordinate signal of the position of lace hole 840.The feedback control module 630 coordinates the servo feedback according to the coordinate signal Signal, and one be relevant to the shoestring 700 and wear order and the controlling planning of action, the operation to the mechanical arm 500 is carried out Visual Feedback Control, the control signal for being relevant to the control result with output enable the mechanical arm 500 to the mechanical arm 500 Drive that the extremity piece 720 wears the lace hole 840 in order and this passes around hole 831.Wherein, how to be worn for the extremity piece 720 It will be further illustrated in the lace hole 840 and the process for passing around hole 831.
First, the feedback control module 630 is according to the coordinate signal and the servo feedback signal, and finds out the extremity piece Relative position between 720 and the lace hole 840.Meanwhile by the controlling planning, to determine the extremity piece 720 being arranged in Which lace hole 840, to set the target location of the clamping jaw module 510, and the horn 520 is allowed to drive the clamping jaw module 510 The target location is moved to, the clip claw mechanism 511 is passed around face both sides positioned at where the lace hole 840 respectively.The opposing party Face, the signal of driver 420 of the hook tongue device 400 of the present embodiment connect the feedback control module of the visual feedback unit 600 630, for the feedback control module 630 according to the controlling planning and feedback result output lift drop signal, it is anti-that the driver 420 receives this The lift drop signal of control module 630 is presented, and the pivot of trace 430 is put to the pressure drop position, allows the hook ring part 444 to drive the footwear Tongue 820 presses to this and puts shoe 220, to expand the working space of the clamping jaw module 510.Then, by one of clip claw mechanism 511 After driving clamped extremity piece 720 to insert the lace hole 840, and continued clamping by other in which clip claw mechanism 511.It is so anti- Multiple operation, as the extremity piece 720 is arranged in the lace hole 840 by the artificial mode for carrying out wear shoes band and order in order.
When little by little wearing to the lace hole 840 of trailing edge, the feedback control module 630 is found out by the controlling planning should The extremity piece 720 is arranged in this and passes around hole 831, just sets the target location of the clamping jaw module 510, and allows the horn 520 to drive The clamping jaw module 510 is moved to the target location, the clip claw mechanism 511 is passed around the both sides in hole 831 positioned at this respectively.It is another Aspect, the driver 420 also can be by the feedback control module 630 or the signal output of other control units, and makes the gearing The pivot of bar 430 is put to the lifting position, is allowed the hook ring part 444 to drive the flap 820 to put shoe 220 away from this, is passed around with highlighting this Hole 831, and the 510 suitable working space of clamping jaw module is provided.Then, it is arranged in the dynamic of the lace hole 840 as foregoing Make, the extremity piece 720 is arranged in this and passes around hole 831.
It is noted that for various styles, the shoes of size, as long as shoes of the wear shoes with operation must be carried out 800, no matter whetheing there is this passes around hole 831, or the shoestring 700 is arranged in order of the lace hole 840 etc., can be transferred through changing The controlling planning, flow of the wear shoes with operation is accordingly adjusted, to meet manufacturer, path business, and consumer demand.
After the extremity piece 720 is arranged in all lace holes 840 and this passes around hole 831, that is, complete wear shoes band operation.This When, the driver 330 of the clamping device 300 also can be by the operation of operator or the signal output of other control units, and makes The outer clamp 312 is moved to the non-clamping position.Then, this is put shoe 220 together with wearing by operator by the slide rail 130 The shoes 800 of good shoestring 700 are pushed into the manual operation position.Then, the pivot of operating parts 324 is put to this release position again by operator Put, the interior clamp 311 is each away from the inner side of the lace hole piece 810;Meanwhile also by pulling the sliding piece 443, It is moved to the hook ring part 444 fully to pass this and pass around hole 831, allows what the flexible section 710 can be formed by the fluting 449 Breach, and depart from the hook ring part 444.Consequently, it is possible to which operator just can remove the shoes 800, and another is not yet worn The shoes 800 of shoestring 700 are placed on this and put shoe 220, to carry out the band operation of the wear shoes of a new round.
In addition, it is necessary to which one is mentioned that, above-mentioned puts footwear apparatus 200, and clamping device 300 is intended merely to illustrate how this Shoes 800 are arranged in the frame 100, and are held fixedly the one of which embodiment aspect that lace hole piece 810 is stated in residence.But It is, should not be as limit, also can be as shown in fig. 6, being configured as another kind with more succinct mechanism puts footwear apparatus 200 ', and folder Hold the embodiment aspect of device 300 '.Wherein, this put footwear apparatus 200 ' with one be arranged on the frame plate 110 put shoe 220 ', And the clamping device 300 ' has two plier groups 310 ', and two drivers 330.Each plier groups 310 ' is set including one The interior clamp 311 ' for putting shoe 220 ' is placed in, and one is driven by one of driver 330, and equally can be interior relative to this The outer clamp 312 that clamp 311 ' moves between the clip position and the non-clamping position.In addition, the pedestal of the hook tongue device 400 410 are also changed to be directly arranged at the frame plate 110, so remaining component is moved also with the pedestal 410, and make the hook ring module 440 between the interior clamp 311 ', moreover, the drive of the driver 420 and the trace 430 is similarly subjected to, and can be at this Pivot is put between lifting position and the pressure drop position.For the interior clamp 311 ', be for combine it is foregoing put footwear apparatus 200 with it is interior The function of clamp 311, the shoes 800 can not only be allowed to be easier to be nested with thereon or removed from it, and not need the wrench mould Block 320 so that integrated model is simplified.In addition, in the embodiment aspect, referring also to Fig. 3 embodiment aspect, in the interior clamp The light fixture 641 ' is respectively set on 311 ', the coordinate of lace hole 840 divided so as to lifting the vision computing module 620 Analyse operation efficiency.
It should be noted that in above-mentioned work flow, the description of any process or mode should be considered as the mould of the work flow Block, fragment, part, or step, it includes one or more specific functions for implementing to present in the work flow, if energy The mode understood by known those skilled in the art enters row order exchange, to reach identical function or effect, ought to be included in the present invention In the range of.
In summary, the computer vision householder method of present invention automation shoelace threading machine can be summarized, comprising following Step:
(A) capturing a width has the image of multiple relevant location informations of lace hole 840.
(B) by three-dimensional reconstruction algorithm or the two dimensional image signature analysis process analysis image, and the lace hole is found out 840 locus.
(C) servo feedback signal of a mechanical arm 500 is relevant to according to one, and finds out the posture of the mechanical arm 500 And action.
(D) according to one be relevant to a shoestring 700 wear the lace hole 840 controlling planning carry out visual feedback, And a control signal for being relevant to the feedback result is exported to the mechanical arm 500.
In this way, the present invention tool computer vision auxiliary automation shoelace threading machine and computer vision householder method have it is as follows The effect of described and advantage, so the purpose of the present invention can be reached really:
First, the position of the lace hole 840 is found out by the visual feedback unit 600, and according to the servo received Feedback signal, and the set controlling planning, and to the mechanical arm 500 carry out visual feedback so that the mechanical arm 500 with The closed loop system is formed between the visual feedback unit 600.In this way, can not only be lifted the stability of total system with it is anti-interference Property so that the shoes 800 expressly need not be ajusted alignment by operator again, with increase processing elasticity and shorten operation time-histories.Together When, also flow of the wear shoes with operation can be adjusted by the controlling planning so that the present invention is suitable to any different styles, size Shoes 800 carry out wear shoes band operation, and allow the efficiency optimization of the operation, and possess the adjustment elasticity of flow.
2nd, the mechanical arm compared to existing shoelace threading machine is only capable of controlling to carry out wear shoes band against the servo feedback of itself Operation.The present invention aids in the action control of the mechanical arm 500 by the visual feedback unit 600, can not only further strengthen The manipulation precision and efficiency of the mechanical arm 500 so that the mechanical arm 500 is not required to fabulous laser intensity and SERVO CONTROL precision, Also it can similarly reach wear shoes band operation, effectively lift the ratio of performance to price of the mechanical arm 500.
3rd, coordinate the clamping device 300 (300') of the frame 100 and hook tongue 400 operation in the work flow of device Auxiliary so that the visual feedback unit 600 does not have to repeat and is continued for feedback control computing, effectively to lift the vision Control efficiency of the feedback unit 600 to the mechanical arm 500.
As described above, only embodiments of the invention are when the scope that the present invention can not be limited with this implement, i.e., all The simple equivalent changes and modifications made according to claims of the present invention and description, all still belongs to the scope of the present invention.

Claims (10)

1. a kind of automation shoelace threading machine for having computer vision auxiliary, is applicable a shoestring being arranged in a shoes, the footwear Band has one section of flexible section, and two extremity pieces for being arranged at the flexible section both ends severally, the shoes have two panels lace hole piece, And it is arranged at multiple lace holes per a piece of lace hole piece;The automation shoelace threading machine includes a frame, at least one machinery Arm, and a visual feedback unit;It is characterized in that:
The frame puts footwear apparatus, and a clamping device for being used to press from both sides the lace hole piece of pulling including one, and this puts footwear apparatus tool There is what the confession shoes were nested with to put shoe;
The mechanical arm includes a clamping jaw module that can carry out pressing from both sides drawing or unclasping to one of extremity piece, and one can drive the clamping jaw Module puts the horn of shoe movement relative to this;
The visual feedback cell signal connects the mechanical arm and is used for court including an image acquisition module, the image acquisition module The lace hole piece, which captures an at least width, has the image of the lace hole relevant position.
2. the automation shoelace threading machine of tool computer vision auxiliary according to claim 1, it is characterised in that:The frame is also Including a microscope carrier, and a slide rail for being arranged at the microscope carrier top surface, this is put footwear apparatus and can be also slippingly arranged at one The slide of the slide rail, this is put shoe and is disposed on the slide one end, and this puts slip of the shoe with the slide on the slide rail, and It can be moved relative to the microscope carrier between an automated job position and an artificial operating position, when this puts shoe together with the footwear Son at the automated job position, pull the lace hole piece by clamping device folder, meanwhile, the image acquisition module is towards the footwear With holes captures the image, and makes the visual feedback unit drive the mechanical arm to drive the extremity piece to wear the shoestring in order Hole, when this puts shoe in the manual operation position, operator can remove the shoes for having worn the shoestring, or by another The shoes for not yet wearing shoestring are placed on this and put shoe.
3. the automation shoelace threading machine of tool computer vision auxiliary according to claim 1, it is characterised in that:The frame is also Including a frame plate, this is put shoe and is disposed on the frame plate, and footwear are put when the shoes for not yet wearing shoestring are placed on this by operator During seat, the visual feedback unit drives the mechanical arm to drive the extremity piece to wear the lace hole in order, then is removed by operator The shoes of the shoestring are worn.
4. the automation shoelace threading machine of the tool computer vision auxiliary according to Claims 2 or 3, it is characterised in that:The footwear Son also there is a flap between the lace hole piece, and it is a piece of be arranged at the flap top surface, and can bend formed one Pass around hole passes around piece, and the frame also includes a hook tongue device, and the hook tongue device is arranged at the base with a pedestal, one Driver, a trace for seat, and a hook ring module for including a hook ring part, the hook ring part are to be used to pass around hole from this Side is arranged in this and passes around hole, and the both ends of the trace are connected to the driver and the hook ring module so that the trace Driven by the driver, and shoe pivot between a lifting position and a pressure drop position can be put relative to this and is put, when this For trace at the lifting position, the hook ring part drives the flap to put shoe away from this, when the trace is in the pressure drop position, The hook ring part drives the flap to press to this and puts shoe.
5. the automation shoelace threading machine of tool computer vision auxiliary according to claim 4, it is characterised in that:The hook ring mould Block is also arranged at the connecting block of the trace one end, a reset assembly for being arranged at the connecting block side, and one including one It is individual to be connected with the reset assembly, and its one end supplies the sliding piece that the hook ring part is set, the sliding piece can be relative to the connecting block Position is worn at one and non-wears is moved between position, when the sliding piece wears position at this, the hook ring part is jointly It is moved to that to pass around piece relative up and down with this, allows an operator to the hook ring part passing around hole side from this and be arranged in this and pass around in hole, When the sliding piece this it is non-wear position when, the hook ring part is jointly moved to this and passes around hole side, and fully passes this and wear Around hole, the reset assembly is to wear position for making the sliding piece perseverance be returned to this.
6. the automation shoelace threading machine of tool computer vision auxiliary according to claim 4, it is characterised in that:The hook ring part Through hole with an insertion left and right sides, and a fluting for connecting the through hole, when the trace is in the lifting position, make The mechanical arm can drive the extremity piece to be arranged in the through hole, in addition, the breach that the flexible section can be formed by the fluting, and depart from In the hook ring part.
7. the automation shoelace threading machine of tool computer vision auxiliary according to claim 1, it is characterised in that:The clamping fills Put and be arranged at this with two plier groups for being respectively used to the corresponding lace hole piece of folder tug, each plier groups including one and put shoe The wherein interior clamp of side, and one can move relative to the interior clamp between a clip position and a non-clamping position Outer clamp, when the outer clamp is in the clip position, one of lace hole piece is clamped support and stands on the interior clamp and the outer clip Between pincers, when the outer clamp is in the non-clamping position, the outer clamp enables the shoes to be put from this and taken on shoe away from the interior clamp Down or it is nested with.
8. the automation shoelace threading machine of tool computer vision auxiliary according to claim 1, it is characterised in that:The image is picked Modulus block has at least one video camera, and the video camera is arranged at the one of which of the frame and the mechanical arm.
9. the automation shoelace threading machine of tool computer vision auxiliary according to claim 1, it is characterised in that:The vision is anti- Feedback unit also includes a light source module, and the light source module has at least one light fixture, and the light fixture is arranged at the frame and put with this The one of which of shoe, the light diffused by the light fixture through the lace hole are subtracted on the image, make the footwear Image feature intensity increase with holes in the image, transported with lifting analysis of the visual feedback unit to the lace hole coordinate Calculate efficiency.
A kind of 10. computer vision householder method for automating shoelace threading machine, it is characterised in that:The computer vision householder method Comprise the steps of:
(A) capturing a width has the image of multiple lace hole relevant location informations;
(B) by three-dimensional reconstruction algorithm or the two dimensional image signature analysis process analysis image, and the sky of the lace hole is found out Between position;
(C) servo feedback signal of a mechanical arm is relevant to according to one, and finds out posture and the action of the mechanical arm;And
(D) it is relevant to a shoestring according to one and wears the controlling planning progress visual feedback of the lace hole, and exports one The control signal of the feedback result is relevant to the mechanical arm.
CN201610751339.4A 2016-07-07 2016-08-29 Automatic shoelace threading machine with computer vision assistance and computer vision assistance method thereof Active CN107581712B (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610530886 2016-07-07
CN201610530886X 2016-07-07

Publications (2)

Publication Number Publication Date
CN107581712A true CN107581712A (en) 2018-01-16
CN107581712B CN107581712B (en) 2020-05-01

Family

ID=61046097

Family Applications (2)

Application Number Title Priority Date Filing Date
CN201610751359.1A Active CN107581713B (en) 2016-07-07 2016-08-29 Automatic shoelace wearing system
CN201610751339.4A Active CN107581712B (en) 2016-07-07 2016-08-29 Automatic shoelace threading machine with computer vision assistance and computer vision assistance method thereof

Family Applications Before (1)

Application Number Title Priority Date Filing Date
CN201610751359.1A Active CN107581713B (en) 2016-07-07 2016-08-29 Automatic shoelace wearing system

Country Status (1)

Country Link
CN (2) CN107581713B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113277142A (en) * 2021-07-22 2021-08-20 苏州德科斯自动化设备有限公司 Cross type quick tying method and equipment
CN113474131A (en) * 2019-02-25 2021-10-01 库卡德国有限公司 Method for automatically pulling shoe upper onto shoe last
CN115035201A (en) * 2022-06-06 2022-09-09 易麦斯智能科技(无锡)有限公司 Automatic shoelace threading method and system based on 3D vision

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108323872A (en) * 2018-04-08 2018-07-27 福建工程学院 A kind of flip-flop automatic setup system and method
CN110731577A (en) * 2019-12-03 2020-01-31 中山市志捷鞋业技术服务有限公司 Auxiliary shoelace binding clamp

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103653545A (en) * 2012-09-26 2014-03-26 陈金柱 Detaching, replacing and assembling method for shoe tightening buckle and device thereof
CN103876393A (en) * 2014-03-27 2014-06-25 宁波舒普机电科技有限公司 Automatic lacing machine
CN203818149U (en) * 2014-04-23 2014-09-10 磊登自动控制有限公司 Clamping device
US20140277726A1 (en) * 2013-03-18 2014-09-18 Kabushiki Kaisha Yaskawa Denki Robot system and method of controlling the robot system
CN204997670U (en) * 2015-08-17 2016-01-27 谢燕娟 Multi -platform gripper

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9447532B2 (en) * 2014-01-23 2016-09-20 Nike, Inc. Automated assembly and stitching of shoe parts
CN202588529U (en) * 2012-04-10 2012-12-12 洪志强 Pneumatic type shoelace-crossing fixing device
CN204969763U (en) * 2015-05-20 2016-01-20 东莞市鑫拓智能机械科技有限公司 Industrial robot wears shoelace machine automatically
CN107343680B (en) * 2016-05-05 2020-02-07 所罗门股份有限公司 Method and apparatus for automatically threading shoelaces

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103653545A (en) * 2012-09-26 2014-03-26 陈金柱 Detaching, replacing and assembling method for shoe tightening buckle and device thereof
US20140277726A1 (en) * 2013-03-18 2014-09-18 Kabushiki Kaisha Yaskawa Denki Robot system and method of controlling the robot system
CN103876393A (en) * 2014-03-27 2014-06-25 宁波舒普机电科技有限公司 Automatic lacing machine
CN203818149U (en) * 2014-04-23 2014-09-10 磊登自动控制有限公司 Clamping device
CN204997670U (en) * 2015-08-17 2016-01-27 谢燕娟 Multi -platform gripper

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113474131A (en) * 2019-02-25 2021-10-01 库卡德国有限公司 Method for automatically pulling shoe upper onto shoe last
CN113474131B (en) * 2019-02-25 2024-03-19 库卡德国有限公司 Method for automatically pulling shoe upper onto shoe last
CN113277142A (en) * 2021-07-22 2021-08-20 苏州德科斯自动化设备有限公司 Cross type quick tying method and equipment
CN115035201A (en) * 2022-06-06 2022-09-09 易麦斯智能科技(无锡)有限公司 Automatic shoelace threading method and system based on 3D vision
CN115035201B (en) * 2022-06-06 2023-09-29 易麦斯智能科技(无锡)有限公司 Automatic shoelace threading method and system based on 3D vision

Also Published As

Publication number Publication date
CN107581712B (en) 2020-05-01
CN107581713B (en) 2020-04-14
CN107581713A (en) 2018-01-16

Similar Documents

Publication Publication Date Title
CN107581712A (en) Have the automation shoelace threading machine and its computer vision householder method of computer vision auxiliary
TWI703942B (en) Method and system for positioning shoe parts in an automated manner during a shoe-manufacturing process
EP3939466B1 (en) Automated assembly and stitching of shoe parts
CN105610030B (en) A kind of automatic double-head, which press against, double wears thermo-contracting-tube machine
TW201733478A (en) Method and system for manipulating shoe parts in automated manner during manufacturing process and method of positioning and joining shoe parts
TW201334724A (en) Automated manufacturing of shoe parts
CN102825812A (en) Apparatus for manufacturing tire ply material
EP3068253B1 (en) Adjustable surface for use in manufacturing shoe parts
CN107920631B (en) Vamp mounting machine who contains vamp draw group's device
DE102018203621A1 (en) LACE-UP CLIP AUTOMAT AND METHOD FOR PROCESSING A SHOE TOP
CN103015059A (en) Sewing machine cloth collecting device
CN108480546A (en) A kind of flexible riveting autonomous learning systems of view-based access control model guiding
CN105552687A (en) Automatic rubber shell penetrating machine
JP5133721B2 (en) Production system with work sharing function
EP3806687B1 (en) Method and apparatus for mounting a leather article such as a shoe
CN208395414U (en) A kind of industrial robot wrapping cloth automation equipment
TWI629015B (en) Computer vision assisted automated lace lacing machine and computer vision aid
CN209303984U (en) Auto parts and components automatic feeding device
TW201825014A (en) Pickup tool, method of manufacturing article and system for manufacturing article
Mai et al. Three-dimensional trajectory recognition technology for the junction trace of a material assembly used in footwear manufacturing automation
CN210737069U (en) Cut-parts closing device, receive arm and bury clamp equipment
CN211420508U (en) Automatic edge folding mechanism of full-automatic double-color machine
CN104841815B (en) Semi-automatic braiding fractures equipment
CN110552118A (en) cloth suction manipulator based on industrial robot vision and fabric grabbing method
CN109551195A (en) A kind of auto parts precision assembly vision measurement system and method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant