EP1915239A2 - Method for generating an environmental image - Google Patents
Method for generating an environmental imageInfo
- Publication number
- EP1915239A2 EP1915239A2 EP07703187A EP07703187A EP1915239A2 EP 1915239 A2 EP1915239 A2 EP 1915239A2 EP 07703187 A EP07703187 A EP 07703187A EP 07703187 A EP07703187 A EP 07703187A EP 1915239 A2 EP1915239 A2 EP 1915239A2
- Authority
- EP
- European Patent Office
- Prior art keywords
- environment
- sensors
- sensor
- manipulator
- individual images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 230000007613 environmental effect Effects 0.000 title claims abstract description 10
- 230000033001 locomotion Effects 0.000 claims description 8
- 238000004088 simulation Methods 0.000 description 8
- 230000004807 localization Effects 0.000 description 3
- GNFTZDOKVXKIBK-UHFFFAOYSA-N 3-(2-methoxyethoxy)benzohydrazide Chemical compound COCCOC1=CC=CC(C(=O)NN)=C1 GNFTZDOKVXKIBK-UHFFFAOYSA-N 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- FGUUSXIOTUKUDN-IBGZPJMESA-N C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 Chemical compound C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 FGUUSXIOTUKUDN-IBGZPJMESA-N 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000003416 augmentation Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000001454 recorded image Methods 0.000 description 1
- 238000002604 ultrasonography Methods 0.000 description 1
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
- B25J9/1671—Programme controls characterised by programming, planning systems for manipulators characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B2219/00—Program-control systems
- G05B2219/30—Nc systems
- G05B2219/40—Robotics, robotics mapping to robotics vision
- G05B2219/40323—Modeling robot environment for sensor based robot system
Definitions
- the invention relates to a method for generating a spatial environment image of a real environment.
- a real environment is in particular the working space of a manipulator or the manipulator cell of an industrial robot.
- the invention preferably relates to the use in an automatically controlled manipulator according to EN ISO 8373 and in particular to a robot or most preferably to an articulated robot according to this standard.
- simulation models of the real environment are required.
- objects of the real environment are measured or their dimensions are taken over from existing CAD geometry data and the simulation model is created on the basis of the survey data or CAD data.
- Stepocameras special laser scanner
- position data also referred to as point clouds
- Manual coarse overlay of several images or point clouds is used to construct a realistic model of the environment using geometry primitives.
- the spatial positions of the objects contained in the simulation model can be corrected by means of matching methods such that the objects of the simulation model have the greatest possible coverage with the realistic model or the point clouds.
- the object of the invention is to provide, while avoiding the aforementioned drawbacks, a method by means of which a spatial environment image of the real environment of a manipulator can be created, on the basis of which a model without manual intermediate steps, i. built up automatically or an existing model can be corrected.
- a simple method is to be provided which is fast and therefore can also be used for real-time requirements.
- the object is achieved in that individual recordings are created by at least one sensor from different sensor poses, wherein each sensor pose is determined on the basis of the axis position values of a manipulator having the sensor for a coarse overlay of the individual recordings.
- At least partially overlapping individual images of the environment containing distance information can be created by at least one sensor from different sensor poses and brought to overlay.
- the term environment picture refers to a consistent point cloud whose points are related to a real object, for example as points of one Object surface or edge. Each point is clearly described by its coordinates in space.
- a model of the environment of use can be created or an adaptation of such can take place.
- a model of this is a representation of the surrounding image in which geometric primitives, such as lines, surfaces, cuboids, ellipsoids, etc., are placed in the point cloud are that the distance error of a point to the associated geometry primitive is minimal.
- the geometry primitives then correspond individually or in combination to the real objects in the environment or cell.
- an exact detection of the environment of the automatically controlled manipulator and thus the creation of an environment image is carried out, on the basis of which a model of the environment can be constructed in the aforementioned sense or an existing such model can be adapted to the real conditions.
- the sensors used are non-contact, preferably optical sensors.
- the sensors can additionally be designed as distance-giving sensors.
- 1D distance sensors such as infrared sensors, or by means of laser triangulation, wherein a fixed relation to the mechanical interface of the manipulator (hand flange) is given.
- the manipulator must then behave like a deflection unit of a 2D laser scanner with a mechanical interface and position the measuring beam with high precision.
- Such a simple configuration is particularly useful if the manipulator is to calibrate only a specific path, if necessary again, and / or to monitor compliance with such a path by means of a 1D distance sensor. or even if some measuring points are sufficient to match the model with reality.
- scanning sensors are used which, in particular, provide distance information not only over one line, but at least in a specific (area) field of view, ie sensors, such as 2D laser scanners, but most preferably 3D sensors.
- sensors such as 2D laser scanners, but most preferably 3D sensors.
- Cameras such as stereo cameras, depth-field cameras or even time-of-flight cameras.
- the sensors can basically be arranged in a fixed assignment to the mechanical interface of the manipulator (hand flange).
- independently of the manipulator actively moving sensors such as laser scanners are used with motorized tilting units when the sensor does not, like a camera, of itself from the outset absorbs a two-dimensional field of view.
- sensors such as conventional video cameras and infrared distance meters
- Sensor combinations may be designed in such a way that the sensors complement each other in their measuring range, whereby, for example, ultrasound sensors for a remote range and infrared or laser triangulation for a precise determination of the distance in the near range can be combined.
- two or more similar sensors can be used in such a way that their measuring field complements each other, such as two laser scanners arranged at 90 ° to each other, by means of which two measuring planes, which are pivoted by 90 ° relative to one another, are scanned.
- the environment through two opposing sensors is detected.
- the sensors, such as laser scanners, are thus "back to back", so that almost a 360 ° field of view is covered in one plane.
- sensors are distributed over the entire robot structure, so that all areas surrounding the robot can be detected in the same dimensions.
- the invention is also preferably used to detect the normal environment of a manipulator, such as a robot line, then, especially if the environment is not very distinctive, additional artificial landmarks may be inserted as reference points, such as domes of known diameter.
- the superimposition of the individual images is brought to coincide by means of a simultaneous localization and mapping process (SLAM process) known for producing a precise 3D image, wherein either the individual images be brought to coincidence by means of a correlation method or the frames are brought to coincide by iterative displacement, in which case the procedure is such that the frames are moved until errors in the measuring space are minimal or falls below a predetermined error limit.
- SLAM process simultaneous localization and mapping process
- An essential advantage of the invention is that it can generate an accurate image or model of the cell with relatively few means, namely an automatically controlled manipulator and a distance-giving sensor.
- Another advantage is that the model can be built from the point of view of the manipulator and thus contains all relevant environmental features that are needed for operation.
- the method is thus also suitable for measuring workpieces.
- the presented method is also generally capable of improving the positioning accuracy of manipulators in their work area. This can be advantageous, for example, if an absolutely precisely measured industrial robot loses its calibration in the system. With the presented method is a simple survey of the robot in his workspace in the plant possible.
- a further advantage of the invention results in conjunction with the method of DE 102004026185.7, the disclosure of which is made to the disclosure content of the present application.
- the exact knowledge of the attachment point of a camera mounted on the robot is critical for the spatial exact augmentation of TCP lanes etc.
- the corresponding information can also be derived with the invention described here.
- Fig. 1 is a simplified pictorial representation of the use of the method according to the invention
- FIG. 2 shows a flow diagram of a preferred embodiment of the method according to the invention.
- FIG. 1 shows first how a plurality of individual images of an environment are superimposed in three objects which have been recorded from different poses (including location (position) and orientation) of a sensor, such as a conventional video camera from the overlay procedure with high accuracy, the relation (coordinate transformation) of the exact poses in the production of the individual images and also - with known posi- tion of objects in space - the absolute poses can be determined.
- a sensor such as a conventional video camera from the overlay procedure with high accuracy
- FIG. 1 shows, under 1.1, first schematically three objects 1, 2, 3.
- step 1.2 again the three objects 1, 2, 3 are shown.
- the sensors which detect from a two-dimensional image detail, are represented by different positions from recorded image sections A, B, C, D, E, F, G in superposition with the objects 1 to 3.
- step 1.3 the pictures taken by the sensors A 1 to G 'of the environment with the captured from the respective image A 1 to G 1 sub-images IA to IC, 2C to 2E, 3E to 3G of the objects 1 to 3 are shown as the Sensor picks these (and the environment) at different times during its movement from different poses.
- a first coarse estimation step which is based, for example, on detected poses of the image sensor receiving environmental sensors by means of robot-internal sensors, such as angle sensors, a first coarse estimation or superimposition 1.1, 2.1, 3.1 of the images of objects 1 to 3 is undertaken.
- step 1.5 a precise method of providing the data is then carried out by means of a method known per se, such as a correlation method, an iterative method or another simultaneous localization and card construction method (called SLAM method), so that an accurate 3D image the environment of use of the robot, as in a point cloud of the environment arises.
- An iteration for example, be carried out until the error in the measuring space is minimal or falls below a predetermined error limit. This is the result of frames A to G
- Environmental image compiled with different positions of the sensor with the images I 1 , 2 ⁇ 3 'of the objects 1, 2, 3 is then shown in step 1.6 of FIG. 1.
- the relation positions of the sensor in the respective recordings and using the positions of the objects 1 to 3 in space can then be determined from the transformation data obtained during the overlay.
- the exact pose in the room and thus also that of the tool center point (TCP) can be determined during subsequent operation of the robot by means of image recognition.
- Fig. 2 shows the essential process sequence of the method according to the invention.
- a motion sensor or even a plurality of motion sensors which are preferably arranged on a robot, in particular in a defined position and orientation relative to the hand flange of such, is moved (step b).
- the movement path can basically be generated in a simulation system, wherein the objects to be measured in a robot cell are initially predetermined by a user. On the basis of identifiable characteristic geometric features of these objects, eg edges or surfaces, and with the aid of known algorithms for collision-free path planning, the paths are automatically generated.
- a robot at the beginning of the exploration, a robot first determines obstacle-free regions in its current pose.
- the robot For this purpose, for example, it is equipped with sensors by means of which it can determine in which direction it is allowed to move.
- the robot can first start with small movements to create an image of the immediate environment. When his immediate surroundings are explored, more, until then out of sight de targeted yet unknown regions, ie approached and explored by means of collision-free path planning determined tracks.
- One advantage of this refinement is that no a priori model knowledge must be available for the automatic generation of the exploration movement.
- the sensor system may be manually controlled by a user, such as traversing keys, a joystick, a 6D mouse, a robot guide bracket, or the like. At the same time, a collision-free and targeted procedure and alignment of the robot, which can be monitored by the user, is possible.
- step d the pose of the environmental sensor is determined according to step c, and the image of the real cell is recorded (step d).
- the recording is digitized and assigned to the associated pose (step f).
- steps e and f can be performed even after all the recordings have been taken (after step g).
- step h the images are coarsely merged on the basis of the poses of the environmental sensor or the environmental sensors determined in step c, as explained in detail with reference to FIG. 1.4.
- step i and j recordings according to a suitable method until the errors in the measuring space are minimal or a predetermined error limit is undershot (steps i and j); if this is not the case, a return takes place before step i and a renewed execution thereof.
- the exact recording position of the ambient sensors can be determined - with higher accuracy than is possible by the robot-internal sensors, such as angle encoders.
- the positioning accuracy of industrial robots in their work cell can be improved.
Landscapes
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Manipulator (AREA)
- Image Processing (AREA)
- Measuring And Recording Apparatus For Diagnosis (AREA)
- Photoreceptors In Electrophotography (AREA)
Abstract
Description
Claims
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
DE102006005958A DE102006005958A1 (en) | 2006-02-08 | 2006-02-08 | Method for generating an environment image |
PCT/EP2007/000854 WO2007090557A2 (en) | 2006-02-08 | 2007-02-01 | Method for generating an environmental image |
Publications (2)
Publication Number | Publication Date |
---|---|
EP1915239A2 true EP1915239A2 (en) | 2008-04-30 |
EP1915239B1 EP1915239B1 (en) | 2010-07-21 |
Family
ID=38178599
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP07703187A Active EP1915239B1 (en) | 2006-02-08 | 2007-02-01 | Method for generating an environmental image |
Country Status (4)
Country | Link |
---|---|
EP (1) | EP1915239B1 (en) |
AT (1) | ATE474694T1 (en) |
DE (2) | DE102006005958A1 (en) |
WO (1) | WO2007090557A2 (en) |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
DE102007060653A1 (en) * | 2007-12-15 | 2009-06-18 | Abb Ag | Position determination of an object |
DE102015214858A1 (en) * | 2015-08-04 | 2017-02-09 | Bayerische Motoren Werke Aktiengesellschaft | Method and system for creating a three-dimensional model of a production environment and resources |
FR3067957B1 (en) * | 2017-06-26 | 2020-10-23 | Capsix | DEVICE FOR MANAGING THE MOVEMENTS OF A ROBOT AND ASSOCIATED TREATMENT ROBOT |
DE102017010718A1 (en) * | 2017-11-17 | 2019-05-23 | Kuka Deutschland Gmbh | Method and means for operating a robot arrangement |
Family Cites Families (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH07239219A (en) * | 1990-04-30 | 1995-09-12 | Korea Mach Res Inst | Method and device for measuring profile shape of tire edge without contact |
DE4115846A1 (en) * | 1991-05-15 | 1992-11-19 | Ameling Walter | Contactless spatial position measurement in robot processing chamber - acquiring images of robotic actuator with defined marking enabling calibration of imaging units in coordinate system |
US5745387A (en) * | 1995-09-28 | 1998-04-28 | General Electric Company | Augmented reality maintenance system employing manipulator arm with archive and comparison device |
JP3309743B2 (en) * | 1996-11-27 | 2002-07-29 | 富士ゼロックス株式会社 | Shape measuring method and device |
US5946645A (en) * | 1997-04-09 | 1999-08-31 | National Research Council Of Canada | Three dimensional imaging method and device |
JP3421608B2 (en) * | 1999-04-08 | 2003-06-30 | ファナック株式会社 | Teaching model generator |
JP3403668B2 (en) * | 1999-05-27 | 2003-05-06 | 理化学研究所 | Method of synthesizing partial measurement data |
KR100468857B1 (en) * | 2002-11-21 | 2005-01-29 | 삼성전자주식회사 | Method for calibrating hand/eye using projective invariant shape descriptor for 2-dimensional shape |
JP3834297B2 (en) * | 2003-05-12 | 2006-10-18 | ファナック株式会社 | Image processing device |
WO2006084385A1 (en) * | 2005-02-11 | 2006-08-17 | Macdonald Dettwiler & Associates Inc. | 3d imaging system |
-
2006
- 2006-02-08 DE DE102006005958A patent/DE102006005958A1/en not_active Withdrawn
-
2007
- 2007-02-01 WO PCT/EP2007/000854 patent/WO2007090557A2/en active Application Filing
- 2007-02-01 AT AT07703187T patent/ATE474694T1/en active
- 2007-02-01 DE DE502007004463T patent/DE502007004463D1/en active Active
- 2007-02-01 EP EP07703187A patent/EP1915239B1/en active Active
Non-Patent Citations (1)
Title |
---|
See references of WO2007090557A3 * |
Also Published As
Publication number | Publication date |
---|---|
ATE474694T1 (en) | 2010-08-15 |
WO2007090557A3 (en) | 2008-03-06 |
DE102006005958A1 (en) | 2007-08-16 |
DE502007004463D1 (en) | 2010-09-02 |
WO2007090557A2 (en) | 2007-08-16 |
EP1915239B1 (en) | 2010-07-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
DE102015101710B4 (en) | A method of calibrating a moveable gripping member using a remote digital camera | |
DE102018109774B4 (en) | Image processing system, image processing device and image processing program | |
DE60127644T2 (en) | Teaching device for a robot | |
EP1763845B1 (en) | Method and device for determining optical overlaps with ar objects | |
EP3330922B1 (en) | Method and device for representing an object | |
DE102018129143B4 (en) | Variable measurement object-dependent camera structure and calibration thereof | |
DE112019004517T5 (en) | ROBOT CALIBRATION FOR AR (AUGMENTED REALITY) AND DIGITAL TWIN | |
DE102015000587B4 (en) | A robot programming device for creating a robot program for taking an image of a workpiece | |
WO2017207436A1 (en) | Industrial robot | |
EP1521211A2 (en) | Method and apparatus for determining the position and orientation of an image receiving device | |
EP2402124B1 (en) | Method and device for determining structural parameters of a robot | |
DE102009012590A1 (en) | Device for determining the position of a robot arm with camera for taking pictures | |
EP1966661A1 (en) | Method and apparatus for establishing a trajectory of a robot device | |
EP1910999B1 (en) | Method and device for determining the relative position of a first object with respect to a second object, corresponding computer program and a computer-readable storage medium | |
DE102021103726B4 (en) | Measurement parameter optimization method and device and computer control program | |
DE102018113336A1 (en) | A method of using a machine to set an augmented reality display environment | |
DE102012112025A1 (en) | Method for determining position of kinematics, such as multi-axis kinematics as robot arm, involves positioning kinematics for interaction with object, and positioning another kinematics in line of sight to former kinematics | |
DE102019215903A1 (en) | Method and device for generating training data for a recognition model for recognizing objects in sensor data of a sensor, in particular of a vehicle, method for training and method for actuation | |
DE102019207448A1 (en) | Simultaneous localization and map creation in 2D using a 3D scanner | |
WO2022101178A1 (en) | Apparatus and method for measuring, inspecting or processing objects | |
DE102021209178A1 (en) | Method and device for determining relative poses and for calibration in a coordinate measuring machine or robot | |
EP3640583A1 (en) | Method for 3d capturing of an object to be measured | |
EP1915239B1 (en) | Method for generating an environmental image | |
EP3571462A1 (en) | Method and device for producing a 3-d thermogram | |
DE102022202563B3 (en) | Planning a trajectory of a robot |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20071201 |
|
AK | Designated contracting states |
Kind code of ref document: A2 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC NL PL PT RO SE SI SK TR |
|
AX | Request for extension of the european patent |
Extension state: AL BA HR MK RS |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
DAX | Request for extension of the european patent (deleted) | ||
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC NL PL PT RO SE SI SK TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D Free format text: NOT ENGLISH |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REF | Corresponds to: |
Ref document number: 502007004463 Country of ref document: DE Date of ref document: 20100902 Kind code of ref document: P |
|
REG | Reference to a national code |
Ref country code: SE Ref legal event code: TRGR |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: VDEP Effective date: 20100721 |
|
LTIE | Lt: invalidation of european patent or patent extension |
Effective date: 20100721 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20100721 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20100721 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20100721 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FD4D |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20100721 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20101021 Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20100721 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20101121 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20100721 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20101122 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20101022 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20100721 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20100721 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20100721 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20100721 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20100721 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20100721 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20100721 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R081 Ref document number: 502007004463 Country of ref document: DE Owner name: KUKA LABORATORIES GMBH, DE Free format text: FORMER OWNER: KUKA ROBOTER GMBH, 86165 AUGSBURG, DE Effective date: 20110418 Ref country code: DE Ref legal event code: R081 Ref document number: 502007004463 Country of ref document: DE Owner name: KUKA ROBOTER GMBH, DE Free format text: FORMER OWNER: KUKA ROBOTER GMBH, 86165 AUGSBURG, DE Effective date: 20110418 |
|
26N | No opposition filed |
Effective date: 20110426 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20101101 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 502007004463 Country of ref document: DE Effective date: 20110426 |
|
BERE | Be: lapsed |
Owner name: KUKA ROBOTER G.M.B.H. Effective date: 20110228 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20110228 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
GBPC | Gb: european patent ceased through non-payment of renewal fee |
Effective date: 20110201 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20110228 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20110228 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: ST Effective date: 20111102 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20110228 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20110228 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20110201 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MM01 Ref document number: 474694 Country of ref document: AT Kind code of ref document: T Effective date: 20120201 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20110201 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AT Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20120201 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20100721 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20100721 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R081 Ref document number: 502007004463 Country of ref document: DE Owner name: KUKA ROBOTER GMBH, DE Free format text: FORMER OWNER: KUKA LABORATORIES GMBH, 86165 AUGSBURG, DE |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R081 Ref document number: 502007004463 Country of ref document: DE Owner name: KUKA DEUTSCHLAND GMBH, DE Free format text: FORMER OWNER: KUKA ROBOTER GMBH, 86165 AUGSBURG, DE |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: IT Payment date: 20230110 Year of fee payment: 17 |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230528 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: SE Payment date: 20231213 Year of fee payment: 18 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20231205 Year of fee payment: 18 |