EP3682303A1 - Method for production control of production processes in the metalworking industry by means of image-capturing devices - Google Patents

Abstract

The invention relates to a method for production control of process sequences in the industrial machining of workpieces (23) according to machining plans (37) in a production hall for producing an end product, wherein multiple image-capturing devices (13), e.g. cameras, are arranged in the production hall and capture image sequences (41), e.g. video recordings, of overlapping observation regions (17) of the production hall, wherein an object to be tracked, e.g. a mobile unit (15), a transport means (21), a workpiece (23) or a person (31), is identified in a captured image using at least one object feature, e.g. a geometric shape, colour or a light signal of an LED, and the position of said object is determined by means of an interior location system, and it is established whether the object to be tracked is in a predefined spatial zone (27) and/or has passed through a spatial barrier (29), andwherein the production control system emits signals to support location of the object (15, 21, 23, 31) to be tracked, displays information about the position and/or production status of the object to be tracked, sets operating parameters of a machine tool (7), updates a production log, or assigns the object to be tracked to a production task.

Description

 MANUFACTURING PROCESSES IN THE METAL-PROCESSING INDUSTRY

 BY PICTURE RECORDING DEVICES

The present invention relates to a method for monitoring and control of manufacturing processes, in particular of process sequences in the industrial production of workpieces in the metal and / or sheet metal processing. Furthermore, the invention relates to a system for workpiece position monitoring in production halls of the metalworking industry, especially in steel and / or sheet metal processing. Furthermore, the invention relates to the assignment of workpieces to process flows in the context of manufacturing control.

By way of example for the metalworking industry, in industrial steel and / or sheet metal processing many parts of different sizes are often supplied to a very wide variety of processing steps. For example, workpieces on machine tools, z. As laser cut or punched sheet metal parts, sorted out and further processing steps supplied. Cut or punched workpieces are often made available after the machining process in a composite the respective downstream production step. The various process flows are usually performed manually based on a visual comparison with paper-based documents. However, if many different part shapes are cut, a variety of processing steps carried out and approached different areas in a production hall for metal and / or sheet metal processing, such monitoring and control operations are complex and error-prone. Thus, a high variety of parts errors in the parts allocation and subsequent processing steps, eg. B. during order-specific storage in provided workpiece collecting units or when transferring to the subsequent processing step, cause. Are z. B. parts stored incorrectly, a subsequent processing step can be adversely affected, eg. B. be carried out incorrectly.

Thus, a physical material flow is usually synchronized discretely at booking stations manually with processing steps to be performed, so that often no detailed or only a delayed evaluation can take place.

For example, from (still unpublished) German patent applications DE 10 2016 120 132.4 ("workpiece collection unit and method for supporting the machining of workpieces") and DE 10 2016 120 131.6 ("Absortierunterstützungsverfahren and flatbed machine") with filing date 21 October 2016 are methods for sub - supporting the sorting operation of workpieces produced by a flatbed machine tool, in general, methods for assisting the machining of workpieces. Furthermore, from the German patent application DE 10 2017 107 357.4

 "Absortierunterstützungsverfahren und Flachbettwerkzeugmaschine" with filing date April 5, 2017 A support method for sorting, for example, cuttings of a flatbed machine tool is known, the cited German patent applications are incorporated herein in their entirety.

Furthermore, the following German patent application DE 10 2017 120 358.8 relates to "Interior location-based control of production processes in the metalworking industry", DE 10 2017 120 381.8 "Assisted allocation of a workpiece to a mobile unit of an indoor location system", DE 10 2017 120 382.6 "Image-based assignment of an editing plan to a mobile unit data record of a mobile unit of an indoor location system" and DE 10 2017 120 383.4 "Interior passenger location-based production control in the metalworking industry", each with filing date 5 September 2017, the use from Z. B. ultra-wideband technology for locating mobile units in manufacturing halls revealed. These mentioned German patent applications are incorporated herein in full. It is an object of this disclosure to propose methods and systems that can intelligently support manufacturing processes, particularly in the field of metal processing, for example steel and / or sheet metal processing.

At least one of these objects is achieved by a method for manufacturing control of process sequences in the industrial machining of workpieces according to claim 1, by a production control system for controlling manufacturing processes in a production hall according to claim 16 and a method for industrial manufacturing of an end product according to claim 19. Further developments specified in the dependent claims. In one aspect, a method for manufacturing control of process sequences in the industrial machining of workpieces according to machining plans, in particular in the metal and / or sheet metal processing, in a production hall for manufacturing a final product comprises the following steps: Providing a plurality of image capture devices arranged in the production hall and configured to record image sequences of assigned observation areas in the production hall,

 Identifying an object to be tracked in a first image of a first one of the image sequences using at least one object feature that is present as a pixel structure in the pixels of the first image,

 Determining the machining plan that relates to the object to be tracked,

Determining the positions of the object feature in pictures of the first of the picture sequences which were subsequently recorded in the first picture,

 Determining a current position of the object to be tracked by locating the object feature in a temporally current image of one of the image sequences and assigning the position of the object feature in the temporally current image to a current position in the observation region of the image sequence comprising the temporally current image, and

 Integration of the determined current position of the object to be tracked into the production control of the industrial production plant for the production of the final product.

In another aspect, a method for industrially manufacturing an end product using a manufacturing control system includes the steps of:

 Receiving a manufacturing order to manufacture the final product from a workpiece with an MES (Machine Control System) of the manufacturing control system implemented in a data processing device,

 Selecting individual processing steps with the MES,

 Determining a sequence of processing steps with the MES, the processing steps comprising one or more of the following operations: cutting, especially laser cutting, punching, bending, drilling, tapping, grinding, joining, welding, riveting, screwing, pressing, treating the edges and surfaces;

 Data-technical assignment of the processing steps to a machine or a workstation unit,

 Data-technical assignment of the production order to an object to be tracked in the MES,

Manufacturing a workpiece for the final product, wherein it is processed in particular to a part of the final product after a first of the processing steps at the machine or workstation unit assigned to the processing step, (optional step of spatially allocating a mobile unit associated with the production order to the finished workpiece,)

 Saving a status change of the production order in the MES,

 Transport of the finished workpiece (optionally together with the mobile unit) according to the production order to the next machine or workstation unit in the predetermined order,

 Performing the processing step on this machine or workstation unit, saving a status change of the production order in the MES and

 Performing the processing steps of the production order with the MES, wherein the position of the object to be tracked by the MES according to the method outlined above is determinable and the MES has at any time data on the current status and the current position of the object to be tracked.

 In a further aspect, a production control system for controlling production processes in a production hall, in particular a metal and / or sheet-metal processing industrial production facility, comprises:

 a plurality of image pickup devices fixedly arranged in the production hall, wherein the observation areas of at least two image pickup devices overlap one another and / or wherein the observation areas of at least two image pickup devices are spatially separated from one another, and

 an analysis unit, which is designed to determine, exchange and provide data for the position of an object to be tracked in the production hall and to carry out the method outlined above.

The manufacturing control system may further include a display unit configured to display the position of the at least one mobile unit in a floor plan of the production hall. At least one of the image recording devices may be fixedly installed and / or movable, in particular freely movable about a pivot axis and / or along a displacement axis, installed and / or installed on a drone.

The shop floor control system may further include an indoor location system having a plurality of transceiver units permanently installed in the shop floor, at least one mobile unit, and a (location system) analysis unit. The transceiver units and the mobile unit may be configured to transmit and receive electromagnetic signals. The analysis unit is designed to run from the terms of electromagnetic signals between the transceiver units and the mobile unit to determine the positions of the mobile unit and thus the position of a mobile unit to be tracked object to be tracked in the production hall. In some embodiments, the following aspects may be implemented. The image pickup devices can be designed differently. By way of example, the image recording devices can be designed as (video) cameras. An image pickup device can be easily installed on the machine tool or at a workstation unit or in its vicinity, and the image information can be directly evaluated. Thus, machine tools or workstation units can also be retrofitted with such an image acquisition device and with such a location system. The image pickup device may be fixed or movable. You can also for other tasks z. B. when unloading and loading or used for security imaging devices can work in visible to the human region of the light. But you can alternatively or additionally also in the non-visible area, eg. B IR or UV range work. They can also be equipped with their own light sources for additional lighting. Furthermore, inventory cameras, generally inventory imaging devices, e.g. B. on, inside and outside of machines already in the manufacturing plant for other purposes are provided in the described system of image-based tracking are included.

The image recording device can also be formed by an ultrasonic sensor, in particular with an additional sound source.

The image sequences of the image recording devices can have a clear time reference for position determination and with one another.

In some embodiments, the object to be tracked may be tracked within the provided images solely on the basis of typical pixel changes associated with an object feature. The identification of the object to be tracked can take place in the first image with a software system which provides information about the position of the object in the first image, in particular by means of an object information area superimposed on the observation area. Locating the object feature may include determining a An object feature vector that describes the movement of the object feature between images of the first of the image sequences, and projecting the object feature vector into an image of a second of the image sequences. An object feature may be a specific geometric shape, a color, a light signal of an LED, or an extent of an area in the image. An object feature vector may be a directional or velocity vector of an object feature.

In some embodiments, when leaving the object feature, one of the image sequence associated observation area in the production hall can be checked whether an object feature corresponding to the object feature and / or object feature vector in an appropriate location / time window another image sequence - in particular with sufficiently high probability - is identifiable. This can be done, in particular, with the aid of suitable image evaluation means and image evaluation methods / programs. In general, the object to be tracked may be a workpiece, a means of transport, a tool, a mobile unit or a person.

In some embodiments, the observation areas of the first image sequence and a second image sequence may have an overlap area, and a transfer of the object features from the first image sequence to the second image sequence may be based on pixel values in the overlap area of substantially simultaneous images.

In some embodiments, associating the location of the object feature in the current time image with a current position in the observation area of the image sequence may include providing reference points in the observation areas, and calculating an accurate location / space time of the object to be tracked at least one reference point.

In some embodiments, the method may further comprise recording and / or providing the determined position via a suitable interface and / or incorporating the determined position into a motion analysis of the object to be tracked

In some embodiments, the method may further include defining zones and / or spatial barriers in the shop floor, particularly in a site plan of the manufacturing hall, and matching the particular location of the object to be tracked. visually of the zones and / or the physical barriers and control of manufacturing based on the alignment. The matching of the specific position can, for. B. revealed that the object to be tracked is located in a zone or has left this or that the object to be traced has passed through a spatial barrier. The zones and / or the spatial barriers can be defined in two or three dimensions.

In some embodiments, including the particular position of the object to be tracked may include one or more of the following steps: outputting signals to assist in locating the object to be tracked; displaying information about the position and / or the manufacturing state of the object to be tracked, in particular on a mobile output device such as a tablet, smartphone or a surveillance monitor; setting operating parameters of a machine tool; updating a log that records the manufacturing, in particular processing steps; assigning the object to be tracked to an order for manufacturing one or more end products in a plurality of different work steps;

If the object to be tracked is a workpiece, the incorporation of the specific position of the object to be tracked can include: controlling and / or monitoring a machining of the workpiece in a plurality of different work steps carried out at different locations, but in particular within the production hall; Taxes and / or

Monitoring a transport of the workpiece between different operations performed at different locations, but especially within the shop floor; Controlling and / or monitoring a machining of the workpiece at workplaces that are networked or integrated with the production control; and / or control and / or monitoring einet work on workplaces that are not networked with the production control or not integrated into this.

In a production control assisted by indoor locating, the method may include the steps of providing an indoor location system having a plurality of transceivers fixedly installed in the production hall, at least one mobile unit, and a location analysis unit, the transceiver units and the mobile unit for broadcasting and receiving electromagnetic signals is formed and the location analysis unit is formed, from running times of the electromagnetic signals between the transceiver units and the mobile unit, the positions of the mobile unit in to determine the production hall; an assignment of the mobile unit to at least one, in particular traceable object; determining the location of the at least one associated object by locating the associated mobile unit with the indoor location system; incorporation of the particular position in the mapping process in which an object to be tracked determines a pixel structure in the image, and / or in the manufacturing control of the industrial manufacturing facility for manufacturing the final product, in particular image capture device based tracking (also referred to herein as image based Tracking). For example, the incorporation of the imager-based tracking into a manufacturing system may be accomplished with the following entities:

 an image capture device unit that provides image sequences (video streams) at a time sufficient frame rate and pixel resolution.

 a position recognition system having an object recognition unit which, with the aid of suitable object recognition methods, selects the object to be tracked for a specific object

Time and location in a (first) image sequence (video stream) of one of the image pickup devices located.

 - An image sequence analysis unit, using an image sequence analysis software, the object within the image sequences provided only on the basis of the typical pixel changes (movement) tracks without uniquely identifying the object to be tracked as such.

 an image sequence changing unit which associates the object to be tracked with a region in an image sequence of a second image pickup device. The image sequence change unit can also be designed to derive at least one object feature vector from a first image sequence and, based on the object feature vector, to associate the object to be tracked with a pixel region in an image of a (second) image sequence.

Several workpieces, in particular those which have the same shape in the final state and have passed through the same process steps, and especially those which also belong to a common order, are referred to as workpiece collection units or workpiece assembly. These are usually deposited on a workpiece collection unit. It is advantageous to assign a workpiece collection unit and possibly a mobile unit to each workpiece collection unit. In the manufacturing control system can see a list of all orders (including machining plans). Each of the jobs can be assigned to a workpiece collection unit. When imager-based tracking of objects is implemented, each job with job-related workpieces, workpiece collection units, or mobile units in the shop floor can be located at any time. This can also be combined with feedback information from workstations and / or machines in the production control system.

The image pickup-based tracking of objects is distinguished by the fact that the position of the objects can be determined solely by observation with the various image pickup devices provided in the production hall, ie without manual interaction. Previous systems for locating workpieces or jobs in manufacturing facilities have the disadvantage that lost workpieces or jobs must be searched manually. It has been recognized that these manual searches, especially in manufacturing facilities with a high number of small and constantly changing orders, ie z. As in toll manufacturing, account for a tremendous proportion of non-productive time. With the tracking and the described system according to the invention, the positions of the workpieces and thus the orders z. B. on a screen, filtered or localized. The need for time-consuming manual searches of workpieces, but also of tools or persons, can thus be drastically reduced, especially in the (metal and / or sheet-metal processing) industrial production.

In some embodiments, the machining of the workpiece or workpieces is controlled or monitored at workplaces that are networked or integrated with the manufacturing controller. Such machine workplaces include machines that receive and execute manufacturing instructions by data connection, in particular digitally. This can be done by the operator no or only a minimal intervention. Such machines are commonly referred to as automated or fully automated machines. Such machines can also report the status of production to a production control.

In some embodiments, the machining of the work piece (s) at workplaces that are networked or not integrated with the manufacturing control to a very low degree or not at all is controlled and / or controlled. These can be jobs in which the work steps are carried out manually by human hands or sol- che, which probably have machines, but only to a very low degree or not at all crosslinked or can be networked very complex, z. B. so-called manual workstations, as described in DE 10 2016 220 015.1, "manual workstation, work data processing device, manual workstation system, manual workstation operating method and manual workstations" method with filing date October 13, 2016. This said German patent application is incorporated herein in its entirety.

Jobs that are networked only to a very limited extent can be, for example, workplaces with simple machines, such as workplaces for drilling, sawing, milling and bending. Their only networking can consist in a monitoring system, as described in DE 10 2016 220 015.1. Another networking option is the monitoring of the power consumption of such machines and the networking of the information from the power consumption. So if a machine has not consumed any power, z. B. concluded that the machine may not have completed the job yet.

In particular, the combination of manufacturing processes with workplaces that are networked with or integrated into the production control and those that are not, or only to a very small degree, still represents an important hindrance to effective and efficient production control, because Jobs are still printed on paper when they come from an automated workstation to a non-automated workstation. This slows down the production. This also makes flexibility more difficult if, for example, a job that has to be processed particularly quickly, with several process steps at several workstations, should be processed within a short time. A manufacturing company that can ensure this smoothly has advantages over its competitors who can not. With the image picker based tracking of workpieces and the linkage of the positional information obtained with the tracking to the manufacturing control, the concepts disclosed herein can enable flexible and fast manufacturing of end products.

Based on the concepts disclosed herein, intelligent assistance systems in manufacturing halls can provide 2D or 3D position determination of workpieces (generally material) and optionally persons, (eg, operators), transport media, machines, tools, and much more to support manufacturing processes. This makes it possible to use 2D or 3D positions as information, which may be in addition to other sensor information and determined according to the concepts disclosed herein, in the context of overall manufacturing control and digitization of factories.

It has been recognized that the use of such image capture device-based tracking systems is possible precisely in view of the expected process operations in a production hall in the specific environment of a metal and / or sheet metal processing industrial production. Accordingly, such imager-based tracking systems can be integrated into a shop floor control system (also referred to herein as MES (Manufacturing Execution System)).

With reference to the above-mentioned accounting of physical material flow and processing steps, the use of the image acquisition device-based tracking system can result in complexity in the low-cost, dynamic assignment of acquired location information to physical components. The concepts disclosed herein address this complexity and allow, for example, position information to be assigned to production orders with an assigned identifier without complex interaction.

Imaging device-based tracking systems can allow detailed mapping of material flows in manufacturing within a manufacturing shop into digital process processing. The tracking systems can thereby simplify the localization of the objects / persons participating in the production in the production environment. If tools, operating resources or charge carriers are initially recorded once in the image pickup device-based tracking system, they can be integrated in the digital control system. This also applies to objects that are temporarily involved in production, such as production orders or service personnel. Temporarily required dynamic follow-ups can always arise again and are required only a few hours, days or weeks in the production hall.

The embodiments of the integration of such tracking into the processes of metal processing disclosed herein may include the following method steps, uses and advantages: - Illustration of the changing assignment of orders.

 - Illustration of an assistant to a person, eg. As agents, using the tracking system and other sensors, especially in the localization of workpieces and tools.

 - Ensuring a process-reliable and low-effort production by automated processes with low degrees of freedom for the operator.

 - Intuitive production process without time-consuming information gathering for the processor.

When tracking workpieces, jobs, people (eg, operators) and / or tools, many requirements must be considered. It has been recognized that industrial manufacturing is becoming more and more focused on producing small batches with many individual operations (manufacturing processes such as cutting, bending, grinding, surface treatment) at different workplaces such as machine workstations and manual workplaces. So often several hundred different jobs are done in one day, which all require different steps.

As soon as only a fault occurs, the production control can very quickly become very confusing. Time-consuming, semi-finished or not yet processed orders are searched in the production hall by individual persons and their status is determined. This is then transmitted to the production control. This can lead to a considerable loss of time in the actual production.

Due to the ever faster processing steps in productive processing and the increase in the number of different orders with ever smaller number of identical parts, such failures can occur more frequently. The resulting downtime reduces the productive time. If jobs, workpieces, persons, z. As agents, and tools are quickly found, the location of at least some of these units disclosed herein is helpful in reducing absences. In particular, it meets the very high requirements for industrial production.

In industrial production, a position inquiry in real time is desired. It should be locally so accurate that workpieces can be safely found and / or assigned to the processing steps. The tracking should also be flexible, several orders should be An order should be groupable into multiple orders, etc. Tracking should be easy to use and it should be failsafe.

In general, the concepts disclosed herein may allow for increased process reliability, optimization of cycle times, and, correspondingly, cost optimization of production. In particular, the concepts disclosed herein may be e.g. T. cause significant time savings in the manufacturing process, with the manufacturing process z. From the production of a required number of parts to the correct transfer to a subsequent process (eg, a subsequent metal working step). Several orders can also be implemented at the same time process reliable. The concepts disclosed herein further allow easy assignment of workpieces within the scope of the imager-based tracking system. This allows open jobs to be optimized despite the complexity of multiple jobs to be processed simultaneously.

Furthermore, flexible processing of various process sequences can be carried out with the concomitant saving of time when machines such as laser cutting machines and / or punching machines are integrated into the partially automated production process. Furthermore, the avoidance of errors and the automatic, correct booking of workpieces, processing steps, etc. can lay the foundation for a data-based real-time control of metalworking (eg steel and sheet metal fabrication). Accordingly, machine tools used in the production of small batches of workpieces can also be integrated into a production controlled by an MES in the context of Industry 4.0. Compared to object identification and localization via radio technologies (eg Bluetooth LE) or optical methods (BarCodes, DotMatrixCodes) at defined locations such as gates in the production hall, the concept of tracking and the like proposed herein may be used. a. have the following advantages:

 - Objects to be tracked do not need to be modified (no labeling or attachment of electronics necessary).

- If there is no modification of an object, it is not necessary to fall back on its identification via a load carrier (carriage, pallet, etc.) on which the object is stored. - The available position information is not only available at certain times and at defined locations.

 - There is no energy supply necessary as with active and mobile radio technologies.

As explained below, a z. B. in the aforementioned DE 10 2017 120 358.8 combined indoor-location-based control with the Bildaufnahmevomchtung-based tracking system. The combinability encompasses the integration of the interior location in the identification of an object in an image (determination of a pixel structure) and the common use for a comprehensive and complementary provision of position information of objects involved in the production and people. In this case, a mobile unit of an indoor location system itself may be an object to be tracked by means of the image sequences. Furthermore, two methods / systems can be used for localization, which can align and learn from each other.

Herein, concepts are disclosed that allow to at least partially improve aspects of the prior art. In particular, further features and their expediencies emerge from the following description of embodiments with reference to the figures. From the figures show:

1 is an exemplary schematic illustration of a production control system with an image picker based tracking system;

2 shows a representation of an exemplary transport carriage for workpieces,

3 shows an exemplary digital site plan of a production hall,

4 shows another exemplary digital site plan,

 5 shows an exemplary representation of a machine tool that is integrated in an image acquisition device-based tracking system,

 FIG. 6 is a flowchart illustrating a manufacture supported by an image pickup device based tracking system; and FIG

 7 shows a flow chart to illustrate process steps for the industrial production of a final product.

Aspects described herein are based, in part, on the recognition that with the accuracy and reliability of imager-based tracking systems, For example, with an accuracy in the position determination of less than 30 cm, in particular less than 10 cm, the use of image pickup-based tracking system in the context of industrial production makes sense. The image-based tracking concept disclosed herein allows for consistent object tracking during a multi-level manufacturing process within a manufacturing shop, and thus is easily deployed in intralogistics applications. In particular, it is proposed to use image sequence-based object tracking, which enables object tracking without optical geometry recognition with the aid of smart data applications (machine learning, artificial intelligence, etc.).

An exemplary implementation of the concept for object tracking comprises z. For example, the following steps:

 The production hall, at least the area relevant for object tracking, is monitored overlapping with several image acquisition devices. The image capture devices provide the production control system with video streams (image sequences) at sufficiently high frame rate and pixel resolution.

 - An object is selected as part of the production control. The position of the selected object is determined in the production hall and remains subsequently available in the production control system despite position changes in the production hall, which arise as a result of the manufacturing process.

 With the aid of a suitable object recognition method, the object to be tracked is localized at one point in time at one location. This is z. B. an unloading station of a machine tool, such as a sorting table. The localization is clearly via a software (SW) system provided by superordinate systems.

 An image sequence analysis SW tracks the object within the provided image sequence exclusively on the basis of typical pixel changes (movement), without the actual object as such being uniquely identifiable. This means that no object recognition is necessary, which runs over the image sequences. In the field of image analysis software and image sequence analysis software, reference is made to known methods for real-time capable image analysis.

The exemplary implementation may further comprise the aspect of tracking with multiple image capture devices, ie, the object to be traversed leaves the observer. tion range of a first image pickup device, a transfer of tracking to a second image pickup device takes place.

When the images of the first image pickup device and the second image pickup device overlap (slightly), the object to be tracked can be uniquely identified when changing to the detected area of the second image pickup device and subsequently tracked in the image sequence of the second image pickup device.

If there is no spatial overlapping of the observation areas of the first image recording device and the second image recording device (non-overlapping image sequences), speed and / or direction vectors of the object to be tracked can be determined based on the first image sequence with analytical methods of the locations known in the first image sequence , These allow the object to be tracked to be assigned to an object appearing in the second picture sequence. As a rule, the identification can be carried out with sufficiently high probability, since the suitable image sequence features (object feature vectors such as directional and / or velocity vectors, object features such as colors, object sizes, etc.) make an appropriate assignment possible.

The exemplary implementation may further include the aspect that the probability of assigning objects to a subsequent image sequence is further improved with the aid of machine learning algorithms.

Fully or partially automated processing plan assignments connect, for example, a production order with a specific mobile unit of a z. B. UWB location system. You can, for example, by a combined use of an assistance system in the environment of the operator, the image pickup-based tracking system and a z. B. UWB location system can be performed.

In addition to the assignment to be made z. In addition, an assignment of the object (eg the assigned workpiece) to an associated production order (eg the production order of the workpiece) can be carried out. Such fully or partially automated machining plan assignments, for example, connect a production job to a specific tracked object.

A system that allows the above exemplary implementation of the concept includes, for. B .:

 - A monitoring of the production hall with several image recording devices.

- The provision of image sequences with a clear time reference within an analysis environment in real time.

 An initial recognition (image sequence-frame-object assignment) of an object to be tracked with the aid of suitable image recognition in a frame of an image sequence for the provision of object position, object identity, at least one object feature, which is an object feature vector can be assigned in a picture sequence (object feature vector assignment).

 - Object tracking with appropriate streaming tools and algorithms within a sequence of images.

 When checking the object of an area assigned to the image sequence in the production hall (using suitable image evaluation means and image evaluation methods / programs, also known as analytics tools and routines), check whether an object corresponding to the object feature vector is located in a suitable location. / Time window of another image sequence with sufficiently high probability is identifiable.

 - Improvement of the object recognition with the help of machine learning procedures, so that the object geometry no longer has to be recognized by optical methods.

- Calculation of a precise location / space time of the object based on reference points and recording and / or providing the same via a suitable interface. An example of an assistance system for the first detection is z. As an optical assistance system, detected in the optical sensors used by the operator workpieces or tools and in the context of the available production data from the production orders (preferred) are clearly identified for the processing plan assignment. An exemplary assistance system is disclosed in the initially cited DE 10 2016 120 131.6. Such assistance systems can also be used for the processing plan assignment, if, for example, image data relate to the workpiece and the processing plan. The machining plan assignment can then support the further tracking of the object during the subsequent manufacturing process. The close integration of the process sequence ensures process reliability in the manual environment. With the digital processing plan assignment, the objects can be linked to production orders. The production orders relate to machining processes at various production stations, for example on a laser cutting machine or on a punching machine and z. B. at a picking station. The digital assignment can be done, for example, by placing an object in a geometrically defined zone. In this case, information about this order can be loaded with regard to the follow-up or, depending on requirements, always be loaded up-to-date.

The digitally assigned objects can, for. B. workpiece collection points such as carts or pallets, generally load carriers, on which the workpieces are possibly stored during production image-picking device assisted. Similarly, tools can be digitally assigned as an object.

Alternatively or in addition to the image pickup device-based assistance system, the dynamic processing plan assignment can be done by z. B. order papers and / or a code (eg barcode, QR code, etc.) are scanned. Alternatively or in addition to the processing of the information on order papers, the desired geometry of the workpiece can be used. After a comparison of the geometry of the workpiece, for example with the imaging device-based assistance system, with the desired geometry, information from the central production data system can be reloaded and used during follow-up.

If the first allocation image processing does not allow unambiguous identification, the operator may be presented with a subset of the active production orders that are eligible for the acquired geometry. The operator then makes the final selection and makes the digital assignment. This improves process reliability. Particularly similar workpieces / tools can be tracked as an object, without them being confused by the operator, for example, correspondingly assigned incorrectly and processed incorrectly. In the following, image-based tracking concepts will be exemplified in connection with the figures.

1 schematically shows a production control system 1 comprising an MES (Manufacturing Execution System) 3 and an image capture device-based tracking system 5 (also referred to here as a short tracking system).

The MES 3 is connected to one or more machine tools 7 located in a production hall via wireless or wired communication links 9. In general, the MES 3 is used to control process sequences / manufacturing steps in the industrial production of workpieces with the machine tools 7. It thus serves in particular to control the machine tools 7. For this purpose, the MES 3 receives information about the process sequences / production steps and status information of the machine tools MES 3 represents a data processing system or generally a data processing method that may be implemented in a data processing device. This can be a single electronic data processing device (server) or a combination of several data processing devices (server network / cloud). The data processing device or the composite may be provided locally in the manufacturing facility or may be set up remotely.

A platform on which the data processing devices can be available - eg. B. on which the MES 3 can be implemented, may be a so-called cloud. The cloud includes z. For example, an external server with computing and storage capacity that multiple product manufacturers can use simultaneously. Access authorizations, passwords, etc. can ensure that no manufacturer can access the data of another manufacturer or the operator of the production facility. It can be ensured that no outside third party can access the stored data. Protection against unintentional access can be ensured by the fact that the data stored in the cloud is also processed there, and the manufacturer or operator of the production plant who wishes to use the data, in turn, only processes the data in the cloud. Such cloud usages can lead to significant simplification of system configurations and concomitant cost savings. The data processing device may include a user interface (GUI) with various application programs (APPs). With the provision of various APPs that can run a particular application program, the manufacturing software that a business needs can be segmented so that it can only be accessed as needed, as with a specific APP, if it is to be used , This allows usage to the provider providing the manufacturing software to be remunerated as needed.

The tracking system 5 may comprise an analysis unit 11 and a plurality of image recording devices 13, which are provided for observing objects 15. The image recording devices 13 may in particular be cameras. The tracking system 5 may also interact with the MES 3. For example, the tracking system 5 may be formed as a part of the MES 3. The object 15 may be a workpiece, a means of transport for workpieces, a tool or a person. The object 15 can be located during production in one or more observation areas 17 of the image recording devices 13 and is thus imaged in recorded images 39 of the image recording devices 13 stored in the MES 3 (or the analysis unit 11), the images 39 (also referred to as frames) ) of an image pickup device 13 form an image sequence 41 (also referred to as video stream).

The position of the object 15 can be determined in the observation area 17 of an image recording device 13, for example via reference points 19, which are also shown and are indicated schematically in FIG. 1.

In general, the data communication of an image recording device 13 with the production control system 1, in particular with the MES 3, can be bidirectional. For example, an image pickup device 13 may be appropriately aligned by the MES 3 or set in the operating parameters such as frame rate, exposure, etc.

In some embodiments, WLAN senders may be integrated into an image capture device 13 for data access into the shop floor control system 1, such that via this image capture device 13, digital data may be mobilized in the shop floor, e.g. B. accessible via smartphones or tablets. The integration of the WLAN transmitter stations in the Image pickup device 13 can simplify the installation and operation of a data communication system in the shop floor.

For example, the analysis unit 11 may serve as a central master positioning unit (also referred to herein as a "server"). It defines the follow-up process and data exchange with the MES 3.

A group of image capture devices 13 may be assigned to a master position determination unit and the image sequences may be transmitted to them. For capturing positions in large production halls or over several buildings or rooms, several groups of image capturing devices 13 can be provided, which are each assigned to a separate master position determination unit. These master position determination units can in turn communicate with each other. Depending on the position of the object 15 to be tracked, the tracking of different master position determination units (servers) can take place.

FIG. 1 further shows, by way of example, machining plans 37. A machining plan 37 may comprise a geometry data record 37A and / or an encoding data record 37B identifying the workpiece. Further, the machining plan 37 may include one or more machining and workpiece parameters 37C of the corresponding workpiece 23.

2 shows a transporting carriage 21 with a workpiece 23 and a mobile unit 20. The transporting carriage 21 comprises a storage area 24 for this or a plurality of workpieces 23, which were generated by a machine tool 7 as part of a processing plan 37.

The mobile unit 20 is stored by way of example on the transport vehicle 21, but can also be structurally integrated into it. For operator interaction with the mobile unit 20, it may include an electronically-drivable display 20A, such as an e-ink display (also referred to as electronic paper display), and / or one or more signal dispensers 20B (LEDs) for outputting information. The display 20A indicates, for example, information specific to these workpieces 23. This can be, for example: type of the current and the next processing step, for example "deburring", or "bending"; Name and / or identification of the contract; Date, for example, the date of Completion or commencement of the contract; Number of parts to be manufactured; Name or identification of the customer or client.

On the display 20A z. B. information on the order, readable for human and / or machine, coded and / or presented in writing and / or as a figure. The display 20A can also be used as a signal output device for a feedback to the user. Further examples of a signal output device are a device for outputting sound, in particular in the audible range, in particular for outputting speech information and a lamp such as an LED, which can be recorded in the images (39) of an image recording device 13 as significant object features. The LED can communicate information coded by different colors, flashing frequencies and / or flashing patterns visually via the image pickup device 13 to the MES 3.

A particular flashing LED can also be used for position identification, as it is easier to recognize even at long distances than z. B is a specific text on the display 20A. Therefore, a signaling device such as an LED has special advantages when z. For example, a mobile unit 20 is to be tracked in an image sequence, or the mobile unit 20 is searched during the manufacturing process. The LED can be remotely addressed by an operator and then make a signal. Such a remote-controlled addressing can be achieved, for example, by a further mobile unit or by another, in particular, portable device, eg. As smartphone, tablet, or via the analysis unit 11. But you can also directly, for. B. via near field transmitter (eg., Bluetooth, NFC, IR) done. In general, the mobile unit 20 may comprise a modulatable signal source for generating modulated light, sound or vibration signals as a signal output device. It can then - similar to the communication device described in the German Utility Model DE 20 2016 106 352 Ul - used as data transmitting communication device for the wireless transmission of data. In this case, the communication device can have at least one sensor for receiving light, sound or vibration signals and the signal processing unit can be programmed such that it recovers the data contained therein from received, modulated signals. Furthermore, at least one signal input device (for example a key) for inputting parameters can be integrated in the mobile unit 20. The mobile unit 20 may further comprise as a signal input device a simple sensor for receiving light, sound or vibration signals, in particular a brightness sensor. It can then, as described, for example, in the aforementioned DE 20 2016 106 352 U1, be used as a data-receiving communication device for the wireless transmission of data, in particular access data, from a machine tool.

The mobile unit 20 may further include its own image pickup device configured to capture / scan images of workpieces and / or codes (eg, bar codes or QR codes) on the workpieces or on other documents, tools, or products , In this way, an assignment of objects (eg recorded workpieces) to orders can be made. In addition, the mobile unit may have functionality to detect, process and transmit the imager data.

As part of an industrial production of workpieces in the metal and / or sheet metal processing, the mobile units 20 are usually assigned to workpieces. Optionally, the mobile units 20 may be carried by persons in the factory or attached to aids such as trolleys, machines, and tools, thereby again assigning the mobile unit 20 to a person, e.g. As a worker, or an aid can be made to assist the tracking.

FIG. 3 shows a schematic digital layout 25 of an exemplary production hall equipped with several machine tools 7 of different types. Examples of machine tools 7 in steel, aluminum, copper processing, generally in metalworking are cutting machines, in particular laser cutting machines, punching, grinding, bending machines, etc. Also shown in site plan 25 is a workstation 26 which is networked to a very low degree. such as a manual workstation with simple machines, eg. As for drilling, sawing, milling, bending, no networking or only a network via a monitoring system, such as. B. in DE 10 2016 220 015.1 described, have. Further, one recognizes in the site plan zones 27, 27 'and barriers 29. The zones 27, 27' and barriers 29 have been defined by an operator regarding the use of the machine tools 7 and associated operations. The barriers 29 extend spatially (for example wise linear) in the production hall and define boundaries whose violation by a mobile unit can trigger specific actions. The zones 27, 27 'and barriers 29 can generally be associated with workpiece-specific or object / operator-specific properties.

A view as shown in Fig. 3, z. On a screen (monitor) of a data processing device (eg PC). When activating individual zones, barriers or tracked objects on the monitor (eg by means of a cursor or touchpad by means of a touch) status information can be displayed. For certain objects can be filtered, z. B. all mobile units / workpieces with assignment to orders of a particular customer.

Thus, actions can be triggered using the spatial position determination in the production control system when an object to be tracked is within a specific zone or exceeds a specific barrier, these actions being dependent on the associated object, eg. B. workpiece, and its manufacturing relevance, processing / processing state, can vary. The zones 27, 27 'and the barriers 29 can also be marked in the production hall on site color for better identification in the image sequences.

Furthermore, it can be seen in the site plan 25 schematically workpiece collection points, such as trolley 21 or portions of this, for example, are located near a machine tool 7 or in one of the zones 27. Furthermore, one knows schematically operator 31, who operate the machine tools 7.

In the digital layout 25 not only stationary elements (machine tools), but also movable elements (workpieces, trolleys, operators) are displayed. The integration of movable elements into the floor plan is made possible by the image acquisition device-based position determination, for example, by the transport tracking 21, workpieces 23 and operator 31 as objects to be tracked by the tracking system in their movement.

Furthermore, exemplary positions of a plurality of image recording devices 13 can be seen in the digital map 25. The image recording devices 13 provide machine-readable image information. mation. The positions of the image pickup devices 13 are selected such that a 2D position determination or a 3D position determination can be performed in the associated observation areas 17 in the production hall. By way of example, image sequence recordings for two movable objects 15 (exemplary transport carriages 21) are illustrated by means of double arrows 33 in FIG. 3.

The primary application of the imager-based tracking system 5 is locating workpieces 23, generally material, as well as mobile units used in manufacturing such as trolleys 21, stackers, tools, and other mobile devices, as well as people. The fact that these objects can be located more easily during production, using the position information obtained, reduces or avoids search times. The gained spatial information about the objects additionally allows an analysis of process flows and of an (exploitation) of z. B. tools.

4 shows a plan view of another digital layout plan 25 'of another exemplary production hall. It can be seen several positions of image pickup devices 13 with schematically indicated observation areas 17. Furthermore, one recognizes again several zones 27 and barriers 29th

With the tracking system, the positions of objects 15 in the site map 25 'can be displayed and their location relative to zones 27 and barriers 29 used for control purposes in shop floor control, particularly in the machining of workpieces. For this it is again necessary that an object 15, z. B. a workpiece (or a group of workpieces), an operator, a means of transport, a tool, etc. in the observation areas in its position, in particular its change in position, is determined.

Image pickup devices 13 are installed, for example, stationary in the production hall or they can move along Bewegungsstrajektorien to z. For example, it is possible to detect a zone with a pivoting movement (see observation area 17 ") or to perform a movement movement along a zone Furthermore, image recording devices can be provided on drones in order to be used flexibly in the production hall. In a control area 30, the position of a production control device of the production control system 1 is indicated. Here, the analysis unit 11 may be located. Here also a data processing device 30A (eg a PC) with a screen (monitor) can be located on the z. B. the illustrated in Fig. 3 or Fig. 4 digital map 25 or 25 'is displayed.

In particular, the assignment of an object to a pixel area in an image of a recorded image sequence can be effected by various interactions with the production control system 1 (hereinafter also briefly the production control). For example, in an operator interface, the production control, which may be on a

Smartphone or tablet is provided, the respective workpiece / the respective object selected and assigned to a pixel (pixel area) of a displayed image of an image pickup device by entering, for example, an associated reference number.

Instead of a manual input, for example, alternatively automated or semi-automated activation of the display 20A or LED 20B of the mobile unit 20 support the assignment. Furthermore, for example, by means of image processing of an image of the object, which is provided, for example, with a marking such as a bar code (see display 20A in FIG. 2), the assignment can be made. Furthermore, a graphical assignment can be made via a dialog displayed on the user interface. Furthermore, the assignment can take place with reference to location information of a UWB location system.

In a usage scenario, a person, for. B. a processor, according to an order workpieces 23 bend. For this purpose, he accesses data from the production control system (MES) and opens, for example, the digital site plan 25, 25 'of the production hall. Documents the workpieces of the tracking, the operator of the location of the workpieces to be bent 23 in the site plan 25, 25 'is displayed. For example, a mobile unit 20 and workpieces 23 were placed on a trolley 21 as shown in FIG. 2 and tracked via the image capture devices 15 according to the procedure described above in the various image sequences. Accordingly, in the site map, the symbol trolley z. B. displayed together with a schematic shape of the workpiece. To clarify the first assignment, FIG. 5 shows a sorting process in which an operator 31 sorts / deposits the cut material of a laser cutting machine 7 onto the transport carriage 21. A mobile unit 20 has been activated in accordance with a specific order (machining plan assignment) and assigned spatially to the workpieces 23. For example, after the sorting operation, the operator 31 has activated a key of the mobile unit 20 so that the shop floor control system is informed of completion of the sorting operation. In an image of the image recording device 13, an object feature for an object to be traced is now determined, for. For example, a specific shape of the workpiece 23 or a specific color of an LED on the mobile unit 20.

Thus, the analysis unit 11 knows at which position in the picture the object (transport carriage / workpiece) can be found. If the object moves across a plurality of observation zones, a transfer can be carried out, for example, according to the concepts disclosed herein.

If the sorting process is completed and the workpieces are pushed to the next processing process, this can be detected by passing a barrier 29 'and forwarded to the production control system.

The first assignment may be assisted by an assistance system 38 provided on an equipment tractor 7 or generally at a work station.

For this purpose, FIG. 5 shows a machine tool 7 with an optical assistance system 38, which is based on image data acquisition with an image recording device 13 and supports the assignment of a workpiece as an object to be tracked.

In the assisted spatial assignment, the image acquisition device 13 recognizes a rejected workpiece 23 and for this purpose generates a measurement assistance workpiece data set 38A (see also FIG. 1). The measurement assist workpiece data set 38A is compared with geometry data sets 37A of the machining plans 37 in the production control system 1 to identify the machining plan 37 that belongs to the detected workpiece. The incorporation of the fabrication supported by an imager-based tracking system described herein will be discussed in conjunction with FIG. 6. Reference is additionally made to FIGS. 1 to 5. 6 shows by way of example method steps of a method for production control of process sequences in the industrial machining of workpieces 23, the method being assisted by an image acquisition device-based tracking system. For the method, image pickup devices are provided for an imager-based tracking system (step 51). Each image acquisition device is assigned to a viewing area and preferably continuously delivers image sequences to a common analysis unit. By way of example, a first image recording device with a first image sequence (step 51A) and a second image recording device with a second image sequence (step 51 B) in FIG. 6 are indicated. To clarify a modular structure, further image sequences can be generated, which are assigned to the same or another analysis unit (step 5 with steps 51A, 51B ').

Based on an image of one of the image sequences, an association process is carried out in which a pixel structure in the image is determined for an object to be tracked (step 53). For this purpose, the object to be tracked z. B. identified in the first image based on at least one object feature that exists as a pixel structure in the pixels of the first image. By way of example, the object to be tracked is identified in a first image of the first image sequence whose observation region is assigned to a sorting process.

The choice of the object feature to be used as the pixel structure of tracking the movement of the object in the shop floor may be e.g. B. the color and / or geometric shapes, z. B. are stored in a machining plan of a workpiece, take into account. The object feature may further be a light signal of an LED or an extent of a homogeneous pixel value area in the image. The identification of the object to be tracked in the first image can take place with a software system which provides information about the position of the object in the first image, in particular by means of an object information area superimposed on the observation area. For the object to be tracked, the associated processing plan is also determined (step 55).

The tracking (step 57) of the pixel structure in pictures 39 of the picture sequences 41 can now be carried out by means of simple algorithms and without elaborate object recognition. In this case, firstly the positions of the object feature in pictures 39 of the first picture sequence 41, which were recorded temporally following the first picture, are determined (step 57A).

An object to be tracked can leave the observation area of the first image acquisition device. In this case, in a step 57B, the positions of the object feature are determined in images of the second image sequence. The localization of the object feature includes z. B. determining an object feature vector describing the motion of the object feature between images of the first of the image sequences. The object feature vector may be a directional or velocity vector of the pixel structure. These can affect the movement in the picture or the movement in the room, d. h., the production hall, relate. The object feature vector may be projected into an image of the second of the image sequences to continue tracking the pixel structure which may be to be adjusted for the second image sequence due to the viewing direction, etc. In general, when an object feature leaves an observation area associated with the image sequence 39 in the production hall, it is checked, in particular with the aid of suitable analytics tools and routines, whether an object feature corresponding to the object feature and / or object feature vector is in an appropriate one Place / time window of another image sequence, in particular with sufficiently high probability, is identifiable.

As indicated in FIG. 4, observation regions of image sequences 39 may overlap (overlap region 17 '), and a transfer of the object features from the first image sequence to the second image sequence may be based on pixel values in the overlap region 17' of substantially simultaneous images.

If an exact position of the object to be tracked is to be output (step 59), the position of the pixel structure in a current image is projected onto the associated observation area. For this purpose, for example, a comparison with reference points can be made (step 59A), or a corresponding size of the pixel structure can be taken into account (Step 59B) etc. In this case, data from the processing plan that belongs to the object can also be accessed.

Information from the process of determining the current position can be used in the context of self-learning machines (machine learning) in the tracking of the pixel structure (dashed arrow 61).

The particular position of the object is now incorporated into the control of the industrial plant for manufacturing the final product (step 63). Depending on the object, position information of a tool, a person, a means of transport, a machine tool and / or a workpiece collection unit is thus available for the control of the industrial production plant.

The integration into the control of the industrial production plant can, for example, define (step 63A) zones 23 and / or physical barriers 29 in the production hall, in particular in a site plan 25, 25 'of the production hall, and a comparison (step 63B) of FIG certain position with regard to the zones 27 and / or the spatial barriers 29. In the floor plan of the production hall can be found in step 63A z. B. a zone (machine tool zone) to machine tools / processing stations, for example, to the bending machine created. In this case, this zone can be defined as a solid (3D zone), which extends, for example, up to a height of 1.5 m above the hall floor. If a trolley with workpieces as objects to be tracked, which belong to a job, is pushed into this zone, the production control system registers this in step 63B.

The support of the production control of process sequences may include the integration possibilities discussed herein. For example, signals from a mobile unit 20 signal output device - e.g. As a display unit 17, an LED or a speaker - are generated.

Furthermore, the support of the production control of process sequences via the production control system 1 can control processing parameters on the machine tools 7 or generally set manufacturing parameters (step 63C), the z. B. can also refer to the production hall or to a subsequent data analysis.

As a further example of the integration into the production control, using the digital assignment of step 55, the production control system can register the associated processing order at the processing station (for example on the bending machine). Furthermore, additional secondary actions can be initiated automatically. Thus, an associated machining program can be loaded automatically in the machine tool. This may allow the machine tool (for example, a bending machine) to be automatically prepared via a tool master. On an assigned screen, information can be displayed to an operator for the pending machining process (operation). For example, an image of the original shape of the workpiece as well as the curved shape of the workpiece, the number of workpieces to be bent and / or the subsequent subsequent machining operation, etc. can be displayed.

An advantage of working with defined zones and barriers is that the operator has nothing more to do than to bring the tracked objects into a machine tool zone, whereby the various preparatory measures were automatically initiated. As mentioned, the machine tool z. B. automatically be set immediately to the new job to be processed. This can save considerable time and mistakes can be avoided.

Further, when a mobile unit interacts with the shop floor control system, or an operator operates additional functions (input keys, etc.) of a mobile unit, the operator can obtain feedback or notifications from the mobile units about output means such as RGB LED, vibration, displayed text or sound. For example, the status of an order can be visualized, for example, by an LED glows green, as long as the order is in the state of processing. Furthermore, feedback or a message to subsequent processing stations can be given. For example, automatically posting a completed edit can alert the subsequent process that the parts are now ready and where they are. In general, a triggering of actions such as booking over zones can be further enriched so that, for example, workpieces can be monitored over time during the various processing operations. Mobile units may also be part of an indoor location system that supports image-based tracking. In FIG. 6, this is illustrated by the steps 71 to 75:

 - Providing (step 71) of the indoor location system with several fixed in the

Production hall installed transceiver units, at least one mobile unit and an analysis unit, wherein the transceiver units and the mobile unit is designed for transmitting and receiving electromagnetic signals and the analysis unit is formed, from running times of the electromagnetic signals between the transceiver units and the mobile unit To determine positions of the mobile unit in the production hall.

 - associating (step 73) the mobile unit to at least one, in particular traceable object (15, 21, 23, 31).

 - determining (step 75) the position of the at least one associated object (15, 21, 23, 31) by locating the associated mobile unit with the indoor location system.

 - integrating (step 63) the determined position into the assignment process in which an object to be tracked determines a pixel structure in the image (step 53) and / or in the manufacturing control of the industrial production facility for manufacturing the final product, in particular into the image acquisition device -based tracking.

To the details of a z. B. UWB-based positioning system is referred to the patent applications mentioned above. The UWB based location systems disclosed herein for integration into industrial manufacturing are based e.g. On mobile units (often referred to as "tags") and stationary transmit receivers (often referred to as "anchors" or "anchors"). In integration into industrial manufacturing, to position a workpiece, generally an asset, it may each be provided with, or functionally or spatially referenced with, at least one mobile unit (also referred to herein as physical or spatial association) ). The mobile units are generally electronic components which are capable of being combined with the transceivers, in particular by means of UWB communication technology. cate. Each mobile unit can have its own time determination unit ("clock") to determine runtimes.

Spatial allocation may be performed by positioning a mobile unit near an associated workpiece or on the workpiece itself, or by placing the workpiece on a workpiece collection unit on which a mobile unit is provided, such as a trolley, sump, or pallet. The mobile unit may be fixedly mounted there (or even to a person) or attached to or deposited on the workpiece / workpiece collection unit. For an attachment, the mobile unit may, for. Example, have a holding mechanism, such as a magnet or a clamping, screw, clip, bayonet or suction device, with which it can be connected to the workpiece or on the workpiece collection unit so that they can not solve uncontrollably from this , In addition to the spatial assignment of a z. B. workpiece to the mobile unit can also be made an assignment of the mobile unit (and thus the spatially associated workpiece) to the associated production order of the workpiece (also referred to herein as digital allocation of the manufacturing process or short processing plan assignment).

The use of a UWB-based location system can be the starting point for location-dependent information processing. The positioning system can optionally be equipped with further sensors, for example with acceleration and / or position sensors, and thus also serve as the basis for position-dependent information processing. As a result, in particular a location-dependent (and event-dependent) interaction in the context of the UWB-based location system in production control as well as an optimization of production processes becomes possible. For example, the virtual gates and zones can also be used here in order to automatically monitor and control a production process and subsequent production steps. This can be done in particular in real time.

The UWB-based location system may include multiple transceiver units and at least one mobile unit. The location system can also be combined with the MES 3. ken. For example, a UWB analysis unit of the UWB-based location system may be formed as part of the MES 3.

The transceiver units may be configured to transmit to the mobile units UWB radio signals and to receive from these UWB radio signals. The transceiver units can be distributed in the production hall for a sufficient coverage of the production hall, for example similar to the image capturing devices 13. The distance between a locally movable mobile unit and a z. B. permanently installed transceiver unit, can be determined by the time required for the signal to overcome the distance between the two units. If the distances are determined by a plurality of transceiver units whose location is known in each case, the spatial location of the mobile unit can be compared with the transceiver units z. B. be determined by triangulation.

For transit time determination, the transceiver and the mobile unit (s) may have high precision clocks that can accurately determine the time to a few or even fractions of ns. Even if the clocks in the transceiver unit and in the mobile unit are highly accurate, the clocks are not necessarily synchronized. Different methods of synchronization of clocks or the elimination of errors following from the asynchronous clock history can be used. For example, one of the transceiver units can send a signal at a first time T1 and a second signal at a second time T2. The mobile unit may be aware of the time difference T2-T1 or transmitted along with the signals so that it can synchronize to the time of the transceiver units. Alternatively, the mobile unit may transmit two signals at a known time interval Ta. In this case, the sender-receiver unit can determine the synchronization deviation based on its own time measurement with its own clock from the reception of the first signal to the reception of the second signal and calculate out of the distance measurement. The time interval between the first signal and the second signal should be low, so that the mobile unit has not moved significantly locally during this time. The time interval may be selected by the mobile unit to be a predetermined multiple or a predetermined fraction of the time is that the mobile unit requires from receiving a signal to respond to until the first signal is output.

In the combined use of the image-based tracking disclosed herein and a z. B. UWB-based indoor location system tracking and tracking can be used in a smooth transition. For example, in a fluid transition between UWB tracking and image-based tracking (camera tracking), an area that is difficult to access for an optical system (eg due to unfavorable lighting conditions, angled buildings, etc.) or a sensitive area (eg due to person monitoring not permitted) with UWB technology. In suitable areas of the manufacturing plant with good optical conditions, for example in the area of the exit gates and entrance gates, e.g. for goods receipt, the image-based tracking can be provided. In addition, the production control can provide additional information that is taken into account in the image-based tracking. If z. B. a semi-finished during a manufacturing process, z. For example, from a 2D shape to a bent part and to a weld assembly, this image-based tracking information may be provided based on machining plan data. As a result, the optical detection can be facilitated. Furthermore, the image-based tracking can provide additional information that can be considered in the production control. This can be used in particular in quality assurance. Assuming that a resulting design of a workpiece deviates greatly from the expected shape or obviously lacking features, this can be visually recognized and communicated to the production control system. Furthermore, it can be determined as part of image-based tracking whether a workpiece or an assembly is primed or painted or has undergone another (eg in the visible frequency range) recognizable surface treatment. Likewise, in an image recognition in the IR Frequenzberiech z. B. a specific temperature distribution or a temporal temperature profile is detected and communicated to the production control system.

Further, it is possible to trigger events or indicate upcoming events based on the number of tracked objects located in one or more zones. For example, picking operations or transport tasks can be triggered. In addition to stationary zones, additional zones can migrate dynamically with one or more objects. This allows, for example, the transport of multiple carriers (transport trucks), and the entrained orders can be treated together as clusters from the manufacturing control system.

Further, an object may, for example, be manipulated on a hand tool, and with the help of the frame grabber-based tracking, it may be determined when and / or how the hand tool is being used.

By tracking tools, it is also possible to detect the movement of the tool. In this way, information can be generated on how many components have been processed or whether a processing step has been forgotten etc. Furthermore, further data can be transmitted via the image capture device-based tracking system, for example, error messages by corresponding movement pattern of an object z. In a defined error zone.

Another usage scenario relates to the recording of process states which are characterized by the positions of workpieces, people, machines and other equipment and which can be detected by a cognitive evaluation of these measured positions. In general, the image acquisition device-based tracking data as well as the information regarding zones and barriers allow for a variety of evaluation options. It is thus possible, for example, to generate key figures such as key performance indicators (KPIs) using such raw data and to carry out detailed analyzes to optimize production processes. These analyzes (eg KPIs) can be displayed in the form of "heat maps", as a live view or aggregated, so further evaluation diagrams such as spaghetti diagrams can be called up immediately for different processing operations generate a large effort in the survey, make it available at the touch of a button, such as throughput time, value stream analysis, etc. In addition, the processes in production based on the obtained location information can be improved using numerical optimization methods. The use of the imaging device-based tracking system also allows people or body parts of people to be located when they are determined to be tracked. In addition to workpieces and tools, the locating of persons (as a whole or even local location of leg, arm and hand) provides valuable information about processes in production. Usage scenarios for this concern, for example, the monitoring of security-critical areas for the protection of persons, in particular the processor. Furthermore, motion patterns can be generated, which in turn z. B. for process or Ergonomie- improvement of jobs the operator can be evaluated. In particular, the synchronous evaluation of both hands of a person, in particular an operator or a worker, detailed information about the manufacturing process and the workpiece can be detected. So can be detected that

 - an editor has accessed at a position X;

 - An operator has transported a specific workpiece from A to B;

 an operator has deposited a specific workpiece at a position Y;

 - a manufacturing process such as drilling, pressing, ... was performed x times;

 - a manufacturing process such as deburring, welding, ... was performed on a specific trajektory on a workpiece;

 - a joining process has been carried out at a specific position. Different objects to follow can have specific relationships to each other. For example, within a specific manufacturing process, they can be grouped into families to define basic (behavior / movement) patterns. For example, families can be assigned to a job, an assembly, a subsequent process of workpieces or an associated load carrier (transport cart, pallet, collection container). The family affiliation can be changed dynamically in the current editing process. Objects can belong to different families at the same time. Furthermore, families of objects may relate to a specific link, for example all carriers, all means of transport, all agents, all workpieces, all machines, etc., or a family of objects may relate to a specific status of the object, for example a charge status of the mobile units.

Accordingly, the analysis, like the detection of process states, may be based on the evaluation of such families. Through the enhancement of a manufacturing facility with imager-based tracking and interfaces to the production control system disclosed herein, the location of a workpiece collection unit can be determined and / or the movement of an operator's hand captured, for example, by infrared imaging.

An operator who monitors and controls the machining process in a control center can see on his monitoring monitor where a specific job is currently in the process chain and how its status is currently. Accordingly, it can also access a display unit directly in order to adapt displayed data (workpiece information) such as preferences, work steps, etc., and display it to a processor on site. Alternatively or additionally, this can also be done locally with an input device on the workpiece collection point unit (eg, push button, switch, touchpad) or via a data interface which is connected to an external z. B. mobile input unit provides access (smartphone, ipad, smartwatch, etc.). Accordingly, the workpiece collection unit z. B. have a near field radio network (Bluetooth, NFC). For example, the workpiece collection unit is selectively controlled so that the signaling device (eg a bright LED) is activated.

The image-based tracking can also be used for example in the sorting by z. B. the location of a hand (especially a clearly visible in pictures glove) is traced. If the "hand" of an operator removes a component from the skeleton, the component location in the MES is posted from the skeleton to the hand If the hand moves near a workpiece collection unit, the MES posts that this part has been deposited on the associated workpiece collection unit On the one hand, the tracking system can see that the hand came close to the workpiece, and on the other hand, a higher level system (eg, the MES) can link the workpiece collection unit and the hand position.

FIG. 7 shows by way of example process steps of an industrial production of a final product which can be carried out with a production control system, in particular the MES 3, disclosed herein.

In a first step 80, the receipt of a production order (with a processing plan 37) for the production of a final product from a workpiece 23 with the MES 3, the z. In a data processing device 30A. In a following Step 81, the selection of individual processing steps by means of the MES 3. In a further step 82, the selection of an order from the MES 3 (or an operator), in which the processing steps are to be performed. The processing steps can be one or more of the following operations: cutting, in particular laser cutting, punching, bending, drilling, tapping, grinding, joining, welding, riveting, screwing, pressing, treating the edges and surfaces.

In a further step 83, the data assignment of each of the processing steps to a machine 7 or a workstation unit takes place. The workstation unit may be a workstation 26 as described above, in particular a manual workstation.

In a further step 84, the data assignment of the production order to a tracked object data record is stored in the MES 3 for a mobile unit. This step 84 may correspond to step 53 shown in FIG. Step 84 can in particular also earlier, z. B. after one or more of the method steps described above, take place.

In a further step 85, the production of a workpiece 23, which is at least partially a part of the final product, in particular after a first of the processing steps at the machine 7 assigned this processing step or workstation unit. For example, part of the production order is cut from a sheet metal here. Since milling or punching may also be necessary as a subsequent processing step, this workpiece 23 can form even more material than the end product, ie only partially the end product or be part of it.

In a further step 86, the spatial assignment of the manufactured workpiece 23 to a position in an observation area takes place. This step 86 may correspond to step 59 shown in FIG. In a further step 87, a change is made to the status of the production order in the MES 3.

In an optional step 88, the position of the workpiece / tracked object is stored to the production order. In a further step 89, the transport of the workpiece 23 according to the production order to the next machine 7 or the next workstation unit in the predetermined order. This can be done as a result of an instruction of the MES 3 by a person or by an automated transport process.

In a further step 90, this processing step is carried out on the machine 7 or workstation unit assigned to it.

In an optional step 91, the position of the workpiece / tracked object is stored for this production step in the production order.

In a further step 92, the renewed storage of a change in the status of the production order in the MES 3 takes place. In a further step 93, a decision is made as to whether the method step 89, d. h., a transport to another processing step, to continue or whether the production is completed.

During these production steps, it is always possible to determine the position of tracked objects by the MES 3. Thus, the MES 3 can at any time have data such as current status data and current position data of the workpiece 23. In particular, MES 3 and the imaging device-based tracking system 5 may be designed as described above. All the above-described method steps carried out by a production control, a production control system, an image acquisition device-based tracking system or by the MES 3 can also be implemented by means of one or more data processing devices having means for performing the method steps.

It is explicitly stated that all features disclosed in the description and / or the claims are considered separate and independent of each other for the purpose of original disclosure as well as for the purpose of limiting the claimed invention independently of the feature combinations in the embodiments and / or the claims should be. It is explicitly stated that all ranges or indications of groups of units disclose every possible intermediate value or subgroup of units for the purpose of original disclosure as well as for the purpose of limiting the claimed invention, in particular as a limit of range.

Claims

claims

A method for manufacturing control of process sequences in the industrial machining of workpieces (23) according to machining plans (37), in particular in the metal and / or sheet metal processing, in a production hall for producing a final product, comprising the steps:

 Providing (step 51) a plurality of image pickup devices (13) arranged in the production hall and designed to record image sequences (41) of assigned observation areas (17) in the production hall,

Identifying (step 53) an object to be tracked (15, 21, 23, 31) in a first image of a first one of the image sequences (41) using at least one object feature which is present as a pixel structure in the pixels of the first image,

 Determining (step 55) the machining plan relating to the object to be tracked (15, 21, 23, 31),

Determining (step 55A) the positions of the object feature in pictures of the first of the picture sequences (41) which have been subsequently recorded in the first picture,

 Determining (step 57) a current position of the object to be tracked (15, 21, 23, 31) by locating the object feature in a temporally current image of one of the image sequences (41) and associating (step 59) the location of the object feature in a temporally current image to a current position in the observation region of the image sequence (41), which includes the temporally current image, and

 Incorporating (step 63) the determined current position of the object to be tracked (15, 21, 23, 31) into the manufacturing control of the industrial production facility for manufacturing the final product.

2. The method of claim 1, wherein the object to be tracked (15, 21, 23, 31) within the provided images (39) is tracked solely by means of typical pixel changes associated with an object feature. 3. The method of claim 1 or 2, wherein the identification of the object to be tracked (15, 21, 23, 31) in the first image (39) is carried out with a software system, the information about the position of the object (15, 21, 23, 31) in the first image (39).

4. The method of claim 1, wherein locating the object feature comprises:

 Determining an object feature vector describing the motion of the object feature between images (39) of the first of the image sequences (41), and

 Projecting the object feature vector into an image (39) of a second one of the image sequences (41).

5. The method according to any one of the preceding claims, wherein when leaving the object feature of the image sequence (41) associated observation area (17) is checked in the production hall, in particular with the help of suitable image evaluation and Bildauswerte- procedures / programs, whether the object Feature and / or object feature vector corresponding object feature in a suitable place / time window of another image sequence (41), in particular with sufficiently high probability, is identifiable.

6. The method according to any one of the preceding claims, wherein the object to be tracked (15, 21, 23, 31) is a workpiece (23), a transport (21), a tool, a mobile unit (15) or a person (31) and or

 wherein the image sequences (41) of the image recording devices (13) have a unique time reference for position determination and with one another.

A method according to any one of the preceding claims, wherein the object feature comprises a specific geometric shape, a color, a light signal of an LED, or an extent of a region in the image (39), and / or

 wherein an object feature vector is a direction or velocity vector.

8. The method according to any one of the preceding claims, wherein the observation areas (17) of the first image sequence (41) and a second image sequence (41) have an overlap area (17 ') and a transfer of the object features of the first image sequence (41) in the second image sequence (41) takes place on the basis of pixel values in the overlapping region (17 ') of substantially simultaneous images (39).

9. Method according to one of the preceding claims, wherein the assignment of the position of the object feature in the temporally current image (39) to a current position in the observation area (17) of the image sequence (41) comprises: Providing reference points (19) in the observation areas (17), and calculating an exact location / space time of the object to be tracked (15, 21, 23, 31) based on at least one reference point (19).

10. The method according to any one of the preceding claims, further with

 Record and / or provide the specific position via an appropriate

Interface and / or

 Incorporating the determined position in a motion analysis of the object to be tracked (15, 21, 23, 31).

11. The method according to any one of the preceding claims, further with

 Defining zones (27) and / or physical barriers (29) in the production hall, in particular in a site plan (25) of the production hall, and

 Matching the determined position of the object to be tracked (15, 21, 23, 31) with respect to the zones (27) and / or the spatial barriers (29) and controlling the manufacturing based on the alignment.

12. The method according to claim 11, wherein the matching of the determined position shows that the object to be tracked (15, 21, 23, 31) is located in or has left a zone (7) or that the object to be tracked (15, 21, 23, 31) has passed through a spatial barrier (29), and / or

 wherein the zones (27) and / or the spatial barriers (29) are defined in two or three dimensions.

13. The method according to any one of the preceding claims, wherein the incorporation of the particular position of the object to be tracked (15, 21, 23, 31) in the control for manufacturing a final product comprises one or more of the following steps:

 Outputting signals in support of locating the object to be tracked (15, 21, 23, 31);

 Displaying information on the position and / or the manufacturing state of the object to be tracked (15, 21, 23, 31), in particular on a mobile output device such as a tablet, smartphone or a monitoring monitor;

Setting operating parameters of a machine tool (7); Updating a log that logs the manufacturing, in particular processing steps;

 Assigning the obj ect to be followed (15, 21, 23, 31) to a contract for the manufacture of one or more end products in several different steps.

14. The method according to any one of the preceding claims, wherein, if the object to be tracked (15, 21, 23, 31) is a workpiece (23), the incorporation of the specific position of the object to be tracked (15, 21, 23, 31) controlling and / or monitoring in the controller for manufacturing a final product

- A processing of the workpiece (23) in several different steps, which are performed at different locations, but especially within the production hall;

 a transport of the workpiece (23) between different steps, which are carried out at different locations, but in particular within the production hall;

 machining the workpiece (23) at workplaces that are networked or integrated with the production control; and or

 a machining of the workpiece (23) at workplaces that are not networked with the production control or not integrated into this

includes.

15. The method according to any one of the preceding claims, wherein the production control is supported by an indoor location, by means of the following steps:

 Providing (step 71) an indoor location system having a plurality of transceiver units fixedly installed in the production hall, at least one mobile unit (20) and a location analysis unit, wherein the transceiver units and the mobile unit (20) are adapted to transmit and receive electromagnetic signals and the location analysis unit is designed to determine the positions of the mobile unit (20) in the production hall from transit times of the electromagnetic signals between the transceiver units and the mobile unit (20);

Assigning (step 73) the mobile unit (20) to at least one object, in particular to be tracked (15, 21, 23, 31), Determining (step 75) the position of the at least one associated object (15, 21, 23, 31) by locating the associated mobile unit (20) with the indoor location system and

 Incorporating (step 63) the determined position in the mapping process, wherein a pixel structure in the image is determined for an object to be tracked (step 53), and / or in the manufacturing control (step 63) of the industrial manufacturing facility for manufacturing the final product, in particular Image-based tracking.

16. Fertigungssteuerungssy system (1) for controlling manufacturing processes in a production hall, in particular a metal and / or sheet metal processing industrial production plant, with

 a plurality of image acquisition devices (13) arranged fixedly in the production hall, wherein the observation regions (17) of at least two image acquisition devices (13) overlap and / or wherein the observation regions (17) of at least two image acquisition devices (13) are spaced apart spatially, and

 an analysis unit (11), which is designed to determine, exchange and provide data for the position of an object to be tracked (15, 21, 23, 31) in the production hall and for carrying out a method according to one of the preceding claims. 17. A production control system (1) according to claim 16, further comprising

 a display unit which is designed to display the observation areas and / or a position of an object to be tracked (15, 21, 23, 31) and / or at least one mobile unit (20) in a site plan (25) of the production hall, and / or

 wherein at least one of the image pickup devices (13) is fixedly installed and / or movable, in particular freely movable about a pivot axis and / or along a displacement axis, and / or carried by a drone, and / or

 wherein at least one of the image recording devices (13) records images in the visible region of the light and / or in the non-visible region as in the IR or UV region and optionally additionally has its own light sources for additional illumination.

18. A production control system (1) according to any one of claims 16 or 17, further comprising an indoor location system with a plurality of permanently installed in the production hall transceiver units, at least one mobile unit and an analysis unit, wherein the Transceiver units and the mobile unit for transmitting and receiving electromagnetic signals is formed and the analysis unit is formed, from running times of the electromagnetic signals between the transceiver units and the mobile unit, the positions of the mobile unit and thus the position of a mobile unit associated nachzusfolgenden object (15, 21, 23, 31) in the production hall.

19. A method for the industrial production of a final product by means of a production control system (1), comprising the steps of:

 Receiving a production order for manufacturing the end product from a workpiece (23) with an MES (3) of the production control system (1) implemented in a data processing device,

 Selecting individual processing steps with the MES (3),

 Determining an order of the processing steps with the MES (3), wherein the processing steps comprise one or more of the following operations: cutting, in particular laser cutting, punching, bending, drilling, tapping, grinding, joining, welding, riveting, screwing, pressing, treating the Edges and surfaces;

 Data-technical assignment of the processing steps to a machine (7) or a workstation unit (26),

- data-technical assignment of the production order to an object to be tracked (15, 21, 23, 31) in the MES (3),

 Manufacturing a workpiece (23) for the end product, wherein it is processed in particular to a part of the final product after a first of the processing steps at the machine (7) or workstation unit (26) assigned to the processing step,

Optional spatial assignment of a mobile unit (20) assigned to the production order to the finished workpiece (23),

 Saving a status change of the production order in the MES (3),

 Transporting the finished workpiece (23), optionally together with the mobile unit (20), according to the production order to the next machine (7) or workstation unit (26) in the predetermined order,

 Perform the machining step on this machine (7) or workstation unit

(26)

 Saving a status change of the production order in the MES (3) and

Performing the processing steps of the production order with the MES (3), wherein the position of the object to be tracked (15, 21, 23, 31) is determinable by the MES (3) according to the method of any one of claims 1 to 15 and the MES (3) at any time via data on the current status and current position of the object to be tracked (15, 21, 23, 31) has.