WO2013099373A1 - 作業管理装置および作業管理システム - Google Patents

作業管理装置および作業管理システム Download PDF

Info

Publication number
WO2013099373A1
WO2013099373A1 PCT/JP2012/073904 JP2012073904W WO2013099373A1 WO 2013099373 A1 WO2013099373 A1 WO 2013099373A1 JP 2012073904 W JP2012073904 W JP 2012073904W WO 2013099373 A1 WO2013099373 A1 WO 2013099373A1
Authority
WO
WIPO (PCT)
Prior art keywords
work
marker
unit
tool
storage unit
Prior art date
Application number
PCT/JP2012/073904
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
浩延 松下
誠 海田
隆紀 清水
篤 片山
Original Assignee
三菱電機エンジニアリング株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 三菱電機エンジニアリング株式会社 filed Critical 三菱電機エンジニアリング株式会社
Priority to CN201280064775.4A priority Critical patent/CN104023906B/zh
Publication of WO2013099373A1 publication Critical patent/WO2013099373A1/ja

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/002—Measuring arrangements characterised by the use of optical techniques for measuring two or more coordinates
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25B—TOOLS OR BENCH DEVICES NOT OTHERWISE PROVIDED FOR, FOR FASTENING, CONNECTING, DISENGAGING OR HOLDING
    • B25B23/00—Details of, or accessories for, spanners, wrenches, screwdrivers
    • B25B23/14—Arrangement of torque limiters or torque indicators in wrenches or screwdrivers
    • B25B23/142—Arrangement of torque limiters or torque indicators in wrenches or screwdrivers specially adapted for hand operated wrenches or screwdrivers
    • B25B23/1422—Arrangement of torque limiters or torque indicators in wrenches or screwdrivers specially adapted for hand operated wrenches or screwdrivers torque indicators or adjustable torque limiters
    • B25B23/1425—Arrangement of torque limiters or torque indicators in wrenches or screwdrivers specially adapted for hand operated wrenches or screwdrivers torque indicators or adjustable torque limiters by electrical means
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B19/00—Programme-control systems
    • G05B19/02—Programme-control systems electric
    • G05B19/418—Total factory control, i.e. centrally controlling a plurality of machines, e.g. direct or distributed numerical control [DNC], flexible manufacturing systems [FMS], integrated manufacturing systems [IMS], computer integrated manufacturing [CIM]
    • G05B19/41805—Total factory control, i.e. centrally controlling a plurality of machines, e.g. direct or distributed numerical control [DNC], flexible manufacturing systems [FMS], integrated manufacturing systems [IMS], computer integrated manufacturing [CIM] characterised by assembly
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B23—MACHINE TOOLS; METAL-WORKING NOT OTHERWISE PROVIDED FOR
    • B23P—METAL-WORKING NOT OTHERWISE PROVIDED FOR; COMBINED OPERATIONS; UNIVERSAL MACHINE TOOLS
    • B23P19/00—Machines for simply fitting together or separating metal parts or objects, or metal and non-metal parts, whether or not involving some deformation; Tools or devices therefor so far as not provided for in other classes
    • B23P19/04—Machines for simply fitting together or separating metal parts or objects, or metal and non-metal parts, whether or not involving some deformation; Tools or devices therefor so far as not provided for in other classes for assembling or disassembling parts
    • B23P19/06—Screw or nut setting or loosening machines
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/31—From computer integrated manufacturing till monitoring
    • G05B2219/31027—Computer assisted manual assembly CAA, display operation, tool, result
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/37—Measurements
    • G05B2219/37618—Observe, monitor position, posture of tool
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/45—Nc applications
    • G05B2219/45127—Portable, hand drill
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Definitions

  • the present invention relates to a work management apparatus and a work management system that detect a work position using a captured image of an AR marker and determine whether the work is correct or incorrect.
  • Patent Document 1 discloses a technique for specifying a position where screw tightening is performed when an operator performs screw tightening with a hand-held tightening tool.
  • the tightening position detection device includes a deformable tape member that connects a reference position and a tightening tool, an optical fiber sensor that detects torsion and bending of the tape member, and an optical fiber provided in the tape member.
  • a calculation unit that calculates a three-dimensional position of the tightening tool with respect to a reference position from the entire shape of the tape member based on a signal from the sensor.
  • the tape member that connects the reference position and the tightening tool may come into contact with the member to be tightened and the operator, and there is a problem that workability is poor. It was. Moreover, since the optical fiber sensor provided in the tape member is always bent during work, there is a problem that the possibility of failure is high. Furthermore, since a dedicated tightening tool and a reference position are required for each tightening position detection device, there is a problem that the cost becomes high.
  • the present invention has been made to solve the above-described problems, and it is an object of the present invention to provide a work management apparatus and a work management system that can be constructed at low cost without interfering with the work of an operator. And
  • the work management device acquires image data obtained by capturing a work area where a work is performed on a workpiece using a tool provided with a first marker, and the first marker included in the image is acquired.
  • the work procedure storage unit in which information on the three-dimensional position of the tool to be performed is set, and the three-dimensional position of the first marker calculated by the marker position calculation unit are collated with the three-dimensional position of the tool in the work procedure storage unit.
  • an operation correctness determination unit for determining correctness of the position where the operation is performed.
  • the work management device performs image processing on a workpiece using a tool provided with a first marker, and image data obtained by imaging a work area provided with a second marker at a predetermined position.
  • a marker recognition unit that recognizes the first marker and the second marker included in the image, and a three-dimensional position of the second marker recognized by the marker recognition unit based on the image data
  • a marker position calculation unit for calculating the three-dimensional position of the first marker recognized by the marker recognition unit with respect to the origin, a work procedure storage unit in which information on the three-dimensional position of the tool when performing the work is set, and a marker
  • the apparatus includes a work correctness determination unit that compares the three-dimensional position of the first marker calculated by the position calculation unit with the three-dimensional position of the tool included in the work procedure storage unit to determine whether the position where the work is performed is correct.
  • the work management system includes at least a marker attached to a tool, one or more imaging devices that image a work area in which work is performed on a workpiece using the tool, and an imaging device.
  • the image processing apparatus includes the above-described work management apparatus that acquires image data and determines correctness of work.
  • the three-dimensional position of this marker is calculated using image data obtained by imaging the marker attached to the tool, and collated with a preset three-dimensional position of the tool at the time of work, and correctness of work is confirmed.
  • FIG. 6 is a diagram illustrating an example of a captured image used by the work management apparatus according to Embodiment 1.
  • FIG. 3 is a flowchart illustrating an operation of the work management apparatus according to the first embodiment. 6 is a diagram illustrating an example of a work history stored in a work history storage unit of the work management apparatus according to Embodiment 1.
  • FIG. It is a block diagram which shows the structure of the work management system which concerns on Embodiment 2 of this invention. It is a perspective view explaining the management object of the work management system concerning Embodiment 3 of this invention. It is a perspective view explaining the management object of the work management system concerning Embodiment 4 of this invention.
  • FIG. 1 The work management system shown in FIG. 1 takes an image of a worker attaching a fastening component such as a screw to a workpiece (work) using a hand-held tool, and monitors and manages the screw tightening work based on the image data. It is a system to do.
  • FIG. 2 shows an example of the management target.
  • an operator uses a torque wrench 2 to fasten a screw (not shown) in the screw holes 1a to 1d provided at the upper four corners of the work 1
  • a case of monitoring and managing the order of screw tightening will be described.
  • screw holes 1a to 1d for fastening screws are provided at four corners on the upper surface of the workpiece 1.
  • an origin marker 20 is attached to a predetermined position (for example, the center) of the upper surface of the work 1.
  • the shape of the workpiece 1 and the positions and number of the screw holes 1a to 1d are not limited to the illustrated example, and may be arbitrary.
  • An inspection marker 21 is attached to the torque wrench 2 which is a tool.
  • a communication device 2a is attached to the torque wrench 2.
  • torque increase a signal indicating that the specified torque has been reached
  • the torque-up signal and the torque value are received by the wirelessly connected receiver 2 b and output to the tool state detection unit 15 of the work management device 10.
  • the torque wrench 2 may not be configured to transmit both the torque up signal and the torque value at that time, but may be configured to transmit only the torque up signal or only the torque value.
  • the torque wrench 2 was illustrated as a tool, it is not limited to this, For example, an electric driver etc. may be sufficient. In this case as well, information such as a torque-up signal, torque value, and rotation speed can be output to the work management apparatus 10.
  • the origin marker 20 and the inspection marker 21 are so-called AR (Augmented Reality) markers.
  • FIG. 3 shows another example of the AR marker.
  • the AR marker for example, a square two-dimensional image in which a black pattern is arranged on a white background is used. Different black patterns are drawn for the origin marker 20 and the inspection marker 21 on the white background portion surrounded by the black frame of the AR marker. In addition, this pattern has a shape that is not line-symmetric to distinguish the top, bottom, left, and right of the AR marker.
  • the AR marker is imaged by the camera 3 and image processing is performed by the work management apparatus 10, it is preferable to devise so that the surface of the AR marker does not shine.
  • the AR marker may have a shape other than those shown in FIGS. 2 and 3, and colors other than black and white may be used.
  • the camera 3 is fixed at a predetermined position with respect to the work 1 placed on the pedestal.
  • the work area including the screw holes 1a to 1d of the work 1 is imaged at a predetermined sampling time interval to obtain image data.
  • a general-purpose imaging device such as a WEB camera that can output image data to the work management device 10 in real time may be used.
  • the upper surface of the work 1 is the XY plane
  • the camera 3 is installed at a predetermined height position in the Z direction perpendicular to the XY plane.
  • FIG. 4 shows an example of a captured image. This image is an image taken at the timing when the screw tightening of the screw hole 1a is completed and the next screw hole 1b is being tightened.
  • the origin marker 20 attached to the work 1 and the torque wrench 2 The inspection marker 21 attached to is shown.
  • a polarizing filter may be attached to the camera 3 to reduce reflection of light, or a neutral density filter may be attached to lower the illuminance. Thereby, the origin marker 20 and the inspection marker 21 can be reliably imaged.
  • the work management apparatus 10 monitors and manages the screw tightening work using the torque wrench 2 by using the image data of the camera 3, the torque-up signal of the receiver 2b and the torque value at that time as input.
  • a personal computer is used as the work management apparatus 10, and a CPU (Central Processing Unit) executes a program stored in a program memory (not shown) to perform a marker recognition unit 12, a marker position calculation unit 13, a tool state detection.
  • the processing contents of the unit 15, the work correctness determination unit 17, the notification unit 18, and the tool drive control unit 19 are executed.
  • the work management apparatus 10 repeats the processes of steps ST1 to ST9 shown in FIG. 5 every predetermined sampling time.
  • definition information for each AR marker is stored in advance.
  • As the definition information of the origin marker 20, information on a pattern drawn on a white background portion of the origin marker 20, information indicating the origin position, and the like are set.
  • As the definition information of the inspection marker 21, information on a pattern drawn on a white background portion of the inspection marker 21, information indicating the position of the torque wrench 2, information on the capability range of the torque wrench 2, and the like are set. Further, if there is a tool other than the torque wrench 2, an AR marker is attached, and definition information such as a tool type is set for each AR marker. When it is necessary to distinguish between torque wrenches with different capability ranges, different AR markers are attached to the respective torque wrenches, and definition information is set for each.
  • the marker recognizing unit 12 acquires image data from the camera 3 (step ST1), recognizes the image of the AR marker on the acquired image, calculates information such as a two-dimensional position, size, and shape, and Based on the definition information stored in the definition storage unit 11, the recognized AR marker is identified as the origin marker 20 or the inspection marker 21 (step ST2). Then, the recognition result is output from the marker recognition unit 12 to the marker position calculation unit 13.
  • the marker position calculation unit 13 calculates the three-dimensional position and orientation (direction, inclination, etc.) of the origin marker 20 and the inspection marker 21 based on the recognition result of the marker recognition unit 12 (step ST3).
  • a known technique for example, Japanese Patent Laid-Open No. 7-98214
  • the marker position calculation unit 13 uses the three-dimensional position of the origin marker 20 as the origin, and calculates the three-dimensional position of the inspection marker 21 with respect to this origin. Thereafter, the calculated three-dimensional position of the inspection marker 21 is treated as the position of the torque wrench 2. Then, the position coordinates of the torque wrench 2 are output from the marker position calculation unit 13 to the work history storage unit 14.
  • the tool state detection unit 15 is an interface for inputting / outputting information to / from the receiver 2b. If the receiver 2b has received the torque-up signal and the torque value at that time, the tool state detection unit 15 acquires the information (step ST4) and outputs it to the work history storage unit 14. Note that the timing of performing the process of step ST4 is not limited to this, and may be performed at an arbitrary timing before the process of step ST6 is performed. Further, in the operation example of FIG. 5, the processing of steps ST1 to ST3 is repeated every sampling time, and the process proceeds to processing after step ST6 with the acquisition of the torque-up signal as a trigger.
  • the work history storage unit 14 is input at any timing from the position coordinates (X, Y, Z coordinates with respect to the origin) of the torque wrench 2 for each sampling time input from the marker position calculation unit 13 and the tool state detection unit 15. Torque up signal and the torque value at that time are stored (step ST5).
  • FIG. 6 shows an example of the work history.
  • the positions of the torque wrench 2 calculated by the marker position calculation unit 13 the values of the X coordinate, the Y coordinate, and the Z coordinate are registered. Further, “*” is registered at the time when the tool state detection unit 15 acquires the torque-up signal, and the torque value at that time is also registered. The screw tightening work position and the correctness / incorrectness determination result will be described later.
  • the torque wrench 2 is a type that outputs only one of the torque-up signal and the torque value, only one of the information may be registered in the work history storage unit 14.
  • the image data captured by the camera 3 may be output from the marker recognition unit 12 to the work history storage unit 14, and the image data may be accumulated in the work history storage unit 14.
  • the work procedure can be confirmed after the screw tightening operation. Further, it can be left as evidence that the screws are fastened to all the screw holes 1a to 1d.
  • only the image data when the screw tightening operation for the screw holes 1a to 1d is completed may be stored. In this case, for example, when the screw tightening work is completed, a signal is automatically or manually output to the work management apparatus 10, and on the work management apparatus 10 side, image data when a work completion signal is input from the outside. Is stored in the work history storage unit 14. With this configuration, it is possible to leave as evidence that the screws are fastened to all the screw holes 1a to 1d while suppressing the data capacity.
  • a search key associated with AR marker information may be assigned to the image data and stored.
  • the marker recognition unit 12 recognizes the inspection marker 21
  • a search key representing the inspection marker 21 is assigned to the image data at that time, and the work history storage unit 14 records the search key and the image data.
  • the work procedure storage unit 16 stores in advance position information and order information of screw tightening work.
  • the positional relationship between the camera 3 and the workpiece 1 is fixed, and the origin marker 20 is attached at a predetermined position of the workpiece 1, so that the positions of the screw holes 1a to 1d with respect to the origin on the captured image Can be obtained in advance.
  • the coordinates of the screw tightening work position for performing the screw tightening work are determined and registered in the work procedure storage unit 16.
  • a screw tightening work area may be provided by giving a width to the screw tightening work position.
  • the screw tightening work area for the screw hole 1a is A
  • the screw tightening work area for the screw hole 1b is B
  • the screw tightening work area for the screw hole 1c is C
  • the screw tightening work area for the screw hole 1d is D. Further, it is assumed that the order of the screw tightening work areas A, B, C, and D is registered as the order information.
  • the work correctness determination unit 17 refers to the work history storage unit 14 and the work procedure storage unit 16 to determine whether the position and order of the screw tightening work performed by the worker is correct (step ST6). Then, when it is determined that the position and order of the screw tightening work are correct (step ST7 “YES”), the work correctness determination unit 17 outputs the determination result to the work history storage unit 14 (step ST8). On the other hand, when it is determined that the position and order of the screw tightening work are wrong (step ST7 “NO”), information to that effect is output to the notification unit 18 (step ST9), and the result of the determination is stored in the work history storage unit 14. (Step ST8).
  • the work correctness determination unit 17 collates the position of the torque wrench 2 stored in the work history storage unit 14 with the screw tightening work area stored in the work procedure storage unit 16 to determine the position of the torque wrench 2. Is in the screw tightening work area (or if it matches the screw tightening work position), it is determined that the screw tightening work is being performed at this position, and this screw tightening work area is stored in the work history storage unit. 14 to output. Thereby, in the work history shown in FIG. 6, position information such as A and B is registered in the item of the screw tightening work area at the time when it is determined that the screw tightening work is being performed.
  • FIG. 7 is a diagram for explaining correctness determination processing of the work correctness determination unit 17.
  • the image in FIG. 7 corresponds to the captured image in FIG. 4, and the black circle is the origin calculated by the marker position calculation unit 13 based on the origin marker 20.
  • the gray circle is the position of the torque wrench 2 at each time calculated by the marker position calculation unit 13 based on the inspection marker 21, and corresponds to the X, Y, and Z coordinates of the work history in FIG.
  • a hatched area is a screw tightening work area with respect to the origin, which is registered in advance in the work procedure storage unit 16. For reference, positions corresponding to the screw holes 1a to 1d are indicated by crosses.
  • the work correctness determination unit 17 determines that the screw tightening work is being performed.
  • the screw tightening work area is set in the Z direction in addition to the X and Y directions, and when the torque wrench 2 is at a predetermined height or higher from the upper surface of the workpiece 1. It is determined that the screw tightening operation is not performed.
  • the work correctness / incorrectness determination unit 17 performs the order of the screw tightening work area (the screw tightening work position A at time 8:01:00 in the example of FIG. 6) at the torque up signal acquisition time stored in the work history storage unit 14. Are compared with the order information stored in the work procedure storage unit 16, and if they match, it is determined that the screw tightening work has been performed in the correct order. As a result, it is possible to detect a case where the operator makes a mistake in the order of the screw tightening work or a case where the operator forgets the tightening. In this example, the acquisition of the torque up signal is regarded as the completion of the screw tightening operation.
  • the position of the torque wrench 2 is the screw tightening operation. It may be determined whether or not the screw tightening operation has been performed according to the length of time matching the region. Further, if the tool generates a sound during the screw tightening operation such as an electric screwdriver, it may be determined whether or not the screw tightening operation is completed depending on the presence or absence of the sound. Or you may use the rotation speed of an electric driver, the signal of completion
  • the work correctness determination unit 17 determines that the order of the screw tightening work is correct, the work correctness determination unit 17 registers “ ⁇ ” indicating correctness in the item of the correctness determination result in the work history storage unit 14. On the other hand, if it is determined that it is incorrect, “x” indicating an error is registered in the item of the correctness / incorrectness determination result in the work history storage unit 14, and screw tightening operations are performed in an incorrect order to the notification unit 18. The information which shows that is output. Further, information indicating the position of correct screw tightening work may be output.
  • the work correctness determination unit 17 may further determine whether the item is other than the position and the order. For example, if the torque wrench 2 is a type that outputs a torque value, the work correctness determination unit 17 compares the torque value stored in the work history storage unit 14 with the torque value set in the work procedure storage unit 16 in advance. Then, it may be determined whether or not the screw tightening operation has been performed with a correct torque value, and the appropriateness determination result of the torque value may be registered in the operation history storage unit 14.
  • the work correctness determination unit 17 determines which tool is used (torque wrench, electric screwdriver, etc.) based on the definition information of the inspection marker 21, and registers the tool type in the work history storage unit 14. You may make it do. Further, even for the same type of tool, the capability range (a torque wrench for 50 Nm, a torque wrench for 100 Nm, etc.) is determined based on the definition information of the inspection marker 21, and the suitability of the tool is registered in the work history storage unit 14. You may do it. For example, in FIG. 6, the tool ID “2” of the torque wrench 2 is registered in the tool ID item.
  • correct / incorrect determination can be made when the screw tightening operation is performed in two stages of temporary tightening and final tightening.
  • the order information in the work procedure storage unit 16 the order of the temporary tightening screw tightening work areas A, B, C, and D, the torque value at that time, and the final tightening screw tightening work areas A, B, C, The order of D and the torque value at that time are stored in advance. Then, the work correctness determination unit 17 determines whether each screw tightening work of temporary tightening and final tightening has been performed according to the information of the work procedure.
  • the notification unit 18 is an interface that outputs an appropriate signal to a notification device such as a display device or a speaker, and presents information to the worker (step ST9).
  • a notification device such as a display device or a speaker
  • the notification unit 18 displays information indicating that the screw tightening work is performed in the wrong order or the wrong torque value, or outputs the information.
  • reporting part 18 may display the information which shows the position of the correct screwing operation
  • the correctness / incorrectness of the screw tightening work position is determined after obtaining the torque-up signal, but the determination timing is not limited to this.
  • the work correctness determination unit 17 may perform the correctness determination of this position and notify the operator of the determination result from the notification unit 18. Thereby, the operator can know whether the position of the screw tightening work is correct or wrong before the start of the screw tightening work.
  • the torque wrench 2 is driven only when the communication device 2a receives a drive permission signal from the outside (that is, the work management device 10), and when the torque is increased, either the torque up signal or the torque value is set. Alternatively, both are transmitted from the communication device 2a to the receiver 2b.
  • the tool drive control unit 19 of the work management apparatus 10 refers to the tool suitability determination result registered in the work history storage unit 14, so that the capability range of the torque wrench 2 is preset in the work procedure storage unit 16. If it is within the capability range, a drive permission signal is output to the communication device 2a. As a result, the screw tightening operation can be started only when the operator selects the torque wrench 2 in an appropriate capability range, and the screw tightening operation using the torque wrench 2 in the incorrect capability range can be prevented.
  • the tool drive control unit 19 determines whether or not the position of the torque wrench 2 for each sampling time registered in the work history storage unit 14 is in an appropriate screw tightening work position, and at the appropriate screw tightening work position. If there is, a drive permission signal is output to the communication device 2a.
  • a drive permission signal is output to the communication device 2a.
  • the tool drive control unit 19 compares the position coordinate with the position information in the work procedure storage unit 16. Then, it is determined whether or not the position of the torque wrench 2 is correct. If it is correct, a drive permission signal is output to the torque wrench 2.
  • the tool drive control unit 19 stops the output of the drive permission signal based on the torque up signal at time 8: 00: 1: 00, and further, based on the position coordinates of the screw tightening work position B at time 8:01:30.
  • a drive permission signal is output.
  • the work management device 10 is a work area where the work 1 is screwed to the work 1 using the torque wrench 2 provided with the inspection marker 21, and the origin marker is at a predetermined position.
  • a marker recognizing unit 12 that acquires image data obtained by imaging a work area with 20 attached from the camera 3 and recognizes the origin marker 20 and the inspection marker 21 included in the image based on the definition information in the marker definition storage unit 11; Based on the image data, a marker position calculation unit that calculates the three-dimensional position of the inspection marker 21 recognized by the marker recognition unit 12 with respect to the origin using the three-dimensional position of the origin marker 20 recognized by the marker recognition unit 12 as an origin.
  • the screw tightening work areas A to D as the information of the three-dimensional position of the torque wrench 2 when performing the screw tightening work, and the order of performing the screw tightening work;
  • the work procedure storage unit 16 in which information indicating the movement order of the screw tightening work areas A to D is set, the three-dimensional position of the inspection marker 21 calculated by the marker position calculation unit 13 and the movement order thereof are stored in the work procedure storage unit 16.
  • the screw tightening work areas A to D and the moving order thereof are collated with each other, and a work correct / incorrect determination unit 17 for determining the correctness of the position and order of the screw tightening work is provided.
  • the work management apparatus 10 notifies the worker of the determination result of the work correctness determination unit 17 and moves the screw tightening work areas A to D included in the work procedure storage unit 16 in the moving order. Based on this, a notification unit 18 for instructing the position of the next screw tightening operation is provided. For this reason, an operator can be made to perform a screw fastening operation
  • the work management apparatus 10 acquires a torque value signal indicating that the torque value at the time of completion of the tightening or a specified torque output from the torque wrench 2 has been reached. And the three-dimensional position of the inspection marker 21 calculated by the marker position calculation unit 13 and the torque value at the time of completion of fastening acquired by the tool state detection unit 15 or a torque up signal indicating that the specified torque has been reached as date information Since it is configured to include the work history storage unit 14 that stores it together, traceability can be managed.
  • torque value information is set for each screw tightening work area A to D of the torque wrench 2 when the screw tightening work is performed in the work procedure storage unit 16, and the work correctness determination unit 17 determines whether the screw tightening work is correct or incorrect. Since the torque value at the completion of fastening acquired by the tool state detection unit 15 and the torque value of the work procedure storage unit 16 are collated when determining the position of the screw tightening work, The correctness / incorrectness of the torque value can also be determined.
  • the work history storage unit 14 is configured to store image data obtained by imaging the work area, the work procedure can be confirmed after the work. Furthermore, since the work history storage unit 14 is configured to store the image data together with the search key associated with the inspection marker 21 included in the image, the target image is checked when the work procedure is confirmed. You can search easily.
  • the camera 3 that captures the work area is configured to be attached with the filter that reduces the reflection of light in the work area, so that the origin marker 20 and the inspection marker 21 can be reliably imaged. Can do.
  • the origin marker 20 is attached to the work 1.
  • the present invention is not limited to this, and is not limited to this, but a place where the positional relationship with the work 1 is fixed, such as a pedestal on which the work 1 is placed. It may be attached.
  • the three-dimensional position of the inspection marker 21 is calculated using the three-dimensional position of the origin marker 20 as the origin, but the origin marker 20 for defining the origin is not essential.
  • the origin marker 20 can be omitted and the three-dimensional position of the virtual origin can be set in advance. is there.
  • the marker position calculation unit 13 may calculate the three-dimensional position of the inspection marker 21 with respect to the virtual origin.
  • FIG. FIG. 8 is a block diagram showing the configuration of the work management system according to Embodiment 2 of the present invention. The same or corresponding parts as those in FIG. In the second embodiment, a plurality of cameras 3-1 and 3-2 are used to image the workpiece 1 from a plurality of directions.
  • the work management apparatus 10 performs the same processing as that in the first embodiment on each of the image data captured by the plurality of cameras 3-1 and 3-2 to check whether the position and order of the screw tightening work are correct. Determine.
  • the origin marker 20 for the camera 3-1 and the origin marker 20 for the camera 3-2 may be attached to one work 1, or the common imaging of the camera 3-1 and the camera 3-2.
  • One origin marker 20 may be attached within the range.
  • the origin marker 20 may be omitted and a virtual origin may be set.
  • the number of cameras 3-1 and 3-2 may be three or more.
  • the marker recognizing unit 12 of the work management apparatus 10 acquires a plurality of image data obtained by imaging a work area from a plurality of directions, and at least one image among the plurality of images.
  • the marker position calculation unit 13 is configured to calculate the three-dimensional position of the inspection marker 21 with respect to the origin based on the image data in which the inspection marker 21 is recognized.
  • FIG. FIG. 9 shows an example of a management target of the work management system according to Embodiment 3 of the present invention.
  • the workpiece 1 is freely rotatable in the clockwise direction, and the worker rotates the workpiece 1 to change the posture and perform the screw tightening operation.
  • two origin markers 20-1 and 20-2 are attached at predetermined positions of the workpiece 1 as shown in FIG.
  • the work management apparatus has the same configuration as that of the work management apparatus 10 shown in FIG. 1 and will be described below with reference to FIG.
  • definition information of the origin markers 20-1 and 20-2 and the inspection marker 21 is stored in advance in the marker definition storage unit 11.
  • the marker recognizing unit 12 recognizes the AR marker on the image acquired from the camera 3 and determines whether the AR marker is the origin marker 20-1 or the origin marker 20-2 based on the definition information in the marker definition storage unit 11.
  • the inspection marker 21 is identified.
  • the marker position calculation unit 13 calculates the three-dimensional positions and orientations of the origin markers 20-1 and 20-2 and the inspection marker 21 based on the recognition result of the marker recognition unit 12. At this time, the marker position calculation unit 13 uses the three-dimensional position of either one of the origin markers 20-1 and 20-2 as the origin, and determines the three-dimensional position relative to the other origin of the origin markers 20-1 and 20-2. calculate. Further, the marker position calculation unit 13 calculates the three-dimensional position of the inspection marker 21 relative to the origin based on the triangulation principle based on the three-dimensional positions of the origin markers 20-1 and 20-2. The calculated three-dimensional position of the inspection marker 21 is stored in the work history storage unit 14 as the position of the torque wrench 2. The subsequent processing is the same as that in the first embodiment, and a description thereof will be omitted.
  • two origin markers 20-1 and 20-2 are used to calculate the three-dimensional position of the inspection marker 21 using the principle of triangulation, but three or more origin markers may be used.
  • the method of the third embodiment is applied when the workpiece 1 moves with respect to the fixed camera 3, it can also be applied when the camera 3 moves with respect to the fixed workpiece 1. is there.
  • the marker recognizing unit 12 of the work management apparatus 10 acquires image data obtained by capturing the work area to which the origin markers 20-1 and 20-2 are attached, and includes them in the image.
  • the marker position calculation unit 13 recognizes the origin markers 20-1 and 20-2 and the inspection marker 21 to be detected, and uses the three-dimensional positions of the origin markers 20-1 and 20-2 recognized by the marker recognition unit 12 as origins.
  • the three-dimensional position of the inspection marker 21 is calculated using the principle of triangulation. For this reason, also when changing the attitude
  • the position of the torque wrench 2 can be detected when the positions of the work 1 and the camera 3 change relatively, but the work 1 as in the first embodiment. This is also applicable when the positional relationship between the camera 3 and the camera 3 is fixed. In this case, the detection accuracy of the position of the torque wrench 2 is improved.
  • FIG. 10 shows an example of a management target of the work management system according to Embodiment 4 of the present invention.
  • the fourth embodiment it is assumed that screws are fastened to the screw holes 1e and 1f in the previous process, and screws are fastened to the screw holes 1a to 1d in this process. If there is a deficiency such as forgetting to tighten the screw in the previous process, a caution marker 22 is attached to the workpiece 1 as shown in FIG. In this process, the operator first performs screw tightening of the screw holes 1e and 1f on the work 1 provided with the caution marker 22, and then screw tightens of the screw holes 1a to 1d as in the first embodiment. We will do the work.
  • the work management apparatus Since the work management apparatus according to the fourth embodiment has the same configuration as that of the work management apparatus 10 shown in FIG. 1, the following description will be given with reference to FIG.
  • definition information of the origin marker 20, the inspection marker 21, and the attention marker 22 is stored in the marker definition storage unit 11 in advance.
  • the definition information of the caution marker 22 information of a pattern drawn on a white background part for identifying the caution marker 22, information indicating that the work 1 is defective in the previous process, and the like are set.
  • the screw tightening work is performed for the screw holes 1e and 1f of the previous process.
  • Information on the work procedure indicating the position and the order is stored in advance.
  • information indicating that the work procedure is performed in the order of the work procedure of the previous process and the work procedure of this process, and the work 1 without the attention marker 22
  • Information indicating that only the work procedure of this process is performed is stored in advance.
  • the marker recognizing unit 12 recognizes the AR marker on the image acquired from the camera 3 and determines whether the AR marker is the origin marker 20, the inspection marker 21, or the attention marker 22 based on the definition information in the marker definition storage unit 11. Identify. At this time, when the attention marker 22 is identified, attention information indicating that the attention marker 22 is attached to the work 1 is output from the marker recognition unit 12 to the work history storage unit 14.
  • the work history storage unit 14 is input from the marker recognition unit 12 in addition to the torque wrench 2 position input from the marker position calculation unit 13 and the torque up signal input from the tool state detection unit 15 and the torque value at that time.
  • the work correctness determination unit 17 switches the information of the work procedure according to whether or not the caution information is registered in the work history storage unit 14, and performs the correctness determination of the screw tightening work. Specifically, the work correctness determination unit 17 first checks whether or not the caution information is registered in the work history storage unit 14, and if the caution information is not registered, the same as in the first embodiment. Then, it is determined whether or not the screw tightening work is performed in the order of the screw holes 1a to 1d, and the correctness / incorrectness determination result is output to the work history storage unit 14.
  • the information of the work procedure to be referred is switched, it is determined whether or not the screw tightening work is performed in the order of the screw holes 1e, 1f, 1a to 1d, and the correct / incorrect determination result Is output to the work history storage unit 14.
  • the notification unit 18 presents information indicating that to the worker. Thereby, for example, when the worker does not notice the defect of the work 1 and starts the screw tightening operation of the screw hole 1a according to the normal work procedure, the correct screw tightening work position (screw hole 1e) is notified to the worker. be able to. Since other processes are the same as those in the first embodiment, description thereof is omitted.
  • the work procedure storage unit 16 of the work management apparatus 10 has a plurality of pieces of information for determining the correctness of the screw tightening work, and the marker recognizing unit 12 adds new attention to the image.
  • the marker 22 is included, the attention marker 22 is recognized, and the work correctness determination unit 17 performs the screw tightening operation depending on whether the marker recognition unit 12 recognizes the attention marker 22 or not. It is configured to perform the determination by switching information for determining correctness. For this reason, the correctness of the work can be determined not only in the normal work procedure but also in an irregular work procedure such as when the work 1 is defective.
  • the work management device and the work management system acquire image data obtained by imaging a work area, recognize a first marker included in the image, and a first recognition recognized by the marker recognition unit.
  • Marker position calculation unit for calculating the three-dimensional position of the marker work procedure storage unit in which information on the three-dimensional position of the tool when performing the work is set, and the three-dimensional of the first marker calculated by the marker position calculation unit It is not necessary to connect the tool to the work management device by wire, because the position is collated with the three-dimensional position of the tool in the work procedure storage unit and the work correctness judgment unit for judging the correctness of the work position is provided. Since the person can move without limitation to the action range, the present invention can be applied to a work management apparatus and a work management system that determine whether work is correct or incorrect.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Manufacturing & Machinery (AREA)
  • Quality & Reliability (AREA)
  • Automation & Control Theory (AREA)
  • Mechanical Engineering (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Automatic Assembly (AREA)
  • General Factory Administration (AREA)
  • Details Of Spanners, Wrenches, And Screw Drivers And Accessories (AREA)
PCT/JP2012/073904 2011-12-27 2012-09-19 作業管理装置および作業管理システム WO2013099373A1 (ja)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201280064775.4A CN104023906B (zh) 2011-12-27 2012-09-19 操作管理装置及操作管理系统

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2011-285845 2011-12-27
JP2011285845A JP5930708B2 (ja) 2011-12-27 2011-12-27 作業管理装置および作業管理システム

Publications (1)

Publication Number Publication Date
WO2013099373A1 true WO2013099373A1 (ja) 2013-07-04

Family

ID=48696879

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2012/073904 WO2013099373A1 (ja) 2011-12-27 2012-09-19 作業管理装置および作業管理システム

Country Status (3)

Country Link
JP (1) JP5930708B2 (zh)
CN (1) CN104023906B (zh)
WO (1) WO2013099373A1 (zh)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2508090A (en) * 2012-11-09 2014-05-21 Hitachi Ltd Operation management system for displaying and recording completion of operations
WO2018037107A1 (en) 2016-08-26 2018-03-01 Shell Internationale Research Maatschappij B.V. Guidance device and method for installing flanges
CN112116193A (zh) * 2019-06-20 2020-12-22 株式会社日立制作所 作业辅助装置、作业辅助方法以及计算机可读记录介质
EP3865252A4 (en) * 2018-10-10 2022-05-18 Hitachi, Ltd. METHOD FOR MANAGING A MECHANICAL FASTENING UNIT USING AUGMENTED REALITY
EP4173755A4 (en) * 2020-06-30 2023-11-08 Panasonic Intellectual Property Management Co., Ltd. TOOL SYSTEM, TOOL, WORK TARGET IDENTIFICATION SYSTEM, WORK TARGET IDENTIFICATION METHOD AND PROGRAM

Families Citing this family (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101452302B1 (ko) 2013-07-29 2014-10-22 주식회사 하이딥 터치 센서 패널
KR101592206B1 (ko) * 2014-07-24 2016-02-05 지이티 주식회사 스크류 체결시스템 및 그 체결방법
JP6527343B2 (ja) 2014-08-01 2019-06-05 株式会社 ハイディープＨｉＤｅｅｐ Ｉｎｃ． タッチ入力装置
JP2017100201A (ja) * 2015-11-30 2017-06-08 中国電力株式会社 工事管理システム及び携帯端末
JP6625894B2 (ja) * 2016-02-12 2019-12-25 株式会社東日製作所 管理装置および管理システム
CN105750892A (zh) * 2016-04-07 2016-07-13 内蒙古工业大学 一种基于实时图像处理的自动螺旋装配到位检测系统
JP2018108633A (ja) 2016-12-28 2018-07-12 パナソニックＩｐマネジメント株式会社 工具システム
WO2018123433A1 (ja) 2016-12-28 2018-07-05 パナソニックＩｐマネジメント株式会社 工具システム
JP6855801B2 (ja) * 2017-01-16 2021-04-07 沖電気工業株式会社 異常検知システム、異常検知装置、異常検知方法およびプログラム
JP2018147440A (ja) * 2017-03-09 2018-09-20 株式会社日立プラントコンストラクション ケーブル施工管理システム
CN109507913B (zh) * 2017-11-30 2022-04-01 蔚来(安徽)控股有限公司 换电加解锁控制系统及控制方法
WO2019111846A1 (ja) * 2017-12-08 2019-06-13 コネクテックジャパン株式会社 工具、作業管理装置、作業管理方法及び作業管理システム
JP6596530B2 (ja) * 2018-03-23 2019-10-23 アトラスコプコ株式会社 方法及び装置
JP6844582B2 (ja) * 2018-04-27 2021-03-17 京セラドキュメントソリューションズ株式会社 ビス締め装置
CN110634122A (zh) * 2018-06-05 2019-12-31 成都精工华耀科技有限公司 一种扣件弹条移位检测方法
KR102550920B1 (ko) * 2018-12-14 2023-07-04 가부시키가이샤 후지킨 작업 관리장치, 작업 관리방법 및 작업 관리 시스템
CN109465630A (zh) * 2018-12-21 2019-03-15 南京林城亿轮轨道交通技术有限公司 一种智能化扭矩操作监管系统及控制方法
JP7266451B2 (ja) * 2019-04-19 2023-04-28 株式会社ディスコ ドライバー
JP7417994B2 (ja) * 2020-01-17 2024-01-19 京都機械工具株式会社 作業管理装置又は作業管理装置用プログラム
KR102181016B1 (ko) * 2020-03-18 2020-11-19 주식회사 디지쿼터스 Iot 기반의 구조물 모니터링 장치 및 이를 포함하는 안전 관리 시스템
WO2022079810A1 (ja) * 2020-10-13 2022-04-21 三洋機工株式会社 ナットランナの制御システム
CN113369796A (zh) * 2021-05-26 2021-09-10 上海御建装饰设计工程有限公司 一种文物修复智能自动化跟踪监控系统

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH06226565A (ja) * 1993-01-28 1994-08-16 Toyota Motor Corp ねじ締め制御装置
JP2003150230A (ja) * 2001-11-14 2003-05-23 Sharp Corp 作業画像管理システムおよび作業画像管理方法
JP2005177919A (ja) * 2003-12-19 2005-07-07 Yamaha Motor Co Ltd 作業用工具及びこれを用いた作業システム
JP2007243846A (ja) * 2006-03-10 2007-09-20 Matsushita Electric Ind Co Ltd 画像抽出分析装置、画像抽出分析システム及び画像抽出分析方法
JP2008181344A (ja) * 2007-01-24 2008-08-07 Fujitsu Ltd 作業位置を特定するためのシステム、作業セル、方法、製品の製造方法、およびマーカ
JP2009001132A (ja) * 2007-06-21 2009-01-08 Jamco Corp 航空機用ホイルの自動締付け装置
JP2009017540A (ja) * 2007-05-31 2009-01-22 Panasonic Corp 画像撮影装置、付加情報提供サーバ及び付加情報フィルタリングシステム
JP2011053781A (ja) * 2009-08-31 2011-03-17 Seiko Epson Corp 画像データベース作成装置、画像検索装置、画像データベース作成方法および画像検索方法

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7703669B2 (en) * 2007-10-31 2010-04-27 The Boeing Company Intelligent fastener installation system

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH06226565A (ja) * 1993-01-28 1994-08-16 Toyota Motor Corp ねじ締め制御装置
JP2003150230A (ja) * 2001-11-14 2003-05-23 Sharp Corp 作業画像管理システムおよび作業画像管理方法
JP2005177919A (ja) * 2003-12-19 2005-07-07 Yamaha Motor Co Ltd 作業用工具及びこれを用いた作業システム
JP2007243846A (ja) * 2006-03-10 2007-09-20 Matsushita Electric Ind Co Ltd 画像抽出分析装置、画像抽出分析システム及び画像抽出分析方法
JP2008181344A (ja) * 2007-01-24 2008-08-07 Fujitsu Ltd 作業位置を特定するためのシステム、作業セル、方法、製品の製造方法、およびマーカ
JP2009017540A (ja) * 2007-05-31 2009-01-22 Panasonic Corp 画像撮影装置、付加情報提供サーバ及び付加情報フィルタリングシステム
JP2009001132A (ja) * 2007-06-21 2009-01-08 Jamco Corp 航空機用ホイルの自動締付け装置
JP2011053781A (ja) * 2009-08-31 2011-03-17 Seiko Epson Corp 画像データベース作成装置、画像検索装置、画像データベース作成方法および画像検索方法

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2508090A (en) * 2012-11-09 2014-05-21 Hitachi Ltd Operation management system for displaying and recording completion of operations
GB2508090B (en) * 2012-11-09 2015-01-28 Hitachi Ltd Operation management system for displaying and recording completion of operations
WO2018037107A1 (en) 2016-08-26 2018-03-01 Shell Internationale Research Maatschappij B.V. Guidance device and method for installing flanges
US10589406B2 (en) 2016-08-26 2020-03-17 Cumulus Digital Systems, Inc. Guidance device and method for installing flanges
US11338415B2 (en) 2016-08-26 2022-05-24 Cumulus Digital Systems, Inc. Guidance device and method for installing flanges
EP3865252A4 (en) * 2018-10-10 2022-05-18 Hitachi, Ltd. METHOD FOR MANAGING A MECHANICAL FASTENING UNIT USING AUGMENTED REALITY
CN112116193A (zh) * 2019-06-20 2020-12-22 株式会社日立制作所 作业辅助装置、作业辅助方法以及计算机可读记录介质
EP4173755A4 (en) * 2020-06-30 2023-11-08 Panasonic Intellectual Property Management Co., Ltd. TOOL SYSTEM, TOOL, WORK TARGET IDENTIFICATION SYSTEM, WORK TARGET IDENTIFICATION METHOD AND PROGRAM

Also Published As

Publication number Publication date
JP2013132736A (ja) 2013-07-08
CN104023906B (zh) 2016-08-24
CN104023906A (zh) 2014-09-03
JP5930708B2 (ja) 2016-06-08

Similar Documents

Publication Publication Date Title
JP5930708B2 (ja) 作業管理装置および作業管理システム
US8761921B2 (en) System, operation cell, method, product manufacturing method, and marker for locating operation position
JP4681856B2 (ja) カメラの校正方法及びカメラの校正装置
US8295588B2 (en) Three-dimensional vision sensor
US20070071310A1 (en) Robot simulation device
JP2015196231A (ja) ねじ締付け装置
JP2012030320A (ja) 作業システム、作業ロボット制御装置および作業プログラム
CN113714789B (zh) 基于视觉定位的螺纹拧紧装置及控制方法
JP2010239409A (ja) 車載カメラの校正装置
JP5081313B2 (ja) 車載カメラの校正装置
JP2020169680A (ja) ボルトの締結状態検知装置及び方法、並びにプログラム
JP2011112401A (ja) ３次元計測のためのキャリブレーション方法および３次元視覚センサ
US10782671B2 (en) Method for operating a pass-through machine and a pass-through machine for edge machining and trimming of workpieces
JP6016760B2 (ja) 作業確認システム
JP2012230041A (ja) 位置検出方法、位置検出装置、ロボットシステム
JP2009289046A (ja) ３次元データを用いた作業支援装置及び方法
KR101535801B1 (ko) 깊이맵 센서를 이용한 조립공정에서의 조립공정 검사장치, 방법 및 시스템
JPH0460817A (ja) カメラの位置ずれ検出方法
JP2013003773A (ja) 画像判別装置、画像特定装置、この画像特定装置を備えた作業工程管理システム、および、プログラム
JP2008254157A (ja) 連結部材装着評価装置、方法、プログラムおよび記憶媒体
WO2019182084A1 (ja) 方法及び装置
JP2020197806A (ja) 検出システム
JPH02276901A (ja) 視覚センサの位置ズレ修正方法
JPH06259530A (ja) 円形穴の認識方法
WO2023157380A1 (ja) ロボット監視システム、監視装置、監視装置の制御方法およびプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12862629

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12862629

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 12862629

Country of ref document: EP

Kind code of ref document: A1