CH718327A1 - Method for identifying the operational status of an industrial machinery and the activities that take place there. - Google Patents

Abstract

A method for identifying the operational status of an industrial machinery, where the aforesaid method comprises at least the following steps: use of at least one video camera and / or at least one sensor to monitor activities relating to the operational status of an industrial machinery; identification of one or more operating states of the aforementioned industrial machinery; where the aforementioned phase of identification of one or more operating states of the aforementioned industrial machinery is carried out by analyzing the images of the aforementioned video camera and / or the data detected by the aforementioned sensor through the use of Machine Learning techniques.

Description

FIELD OF THE INVENTION

[0001] The present invention relates to a method for identifying the operational state of an industrial machinery and the activities that take place there.

[0002] In particular, the present invention relates to a method for identifying the operating state of an industrial machinery, applicable for example, but not limited to, to printing, laminating, extrusion, cutting machines, machines for producing bags and flexible packaging materials and more.

[0003] In the present description, the expression operating state means the activity that a machine is carrying out or in which it is engaged. For example, but not limited to, the production of a certain product, a change (or retooling) between two different productions, adjustments of machine parts or production parameters, with the machine inactive, to correct the output of the machine

BACKGROUND ART NOTE

[0004] As known, industrial machinery is generally in full operation only for a part of the time. A remaining part of the time said machines are in different states, for example re-tooling or change over, or in the phase of adjustment of parts or production parameters (for example to overcome defects in quality problems), or are stopped for control of quality on the materials produced or are not in operation or stopped for lunch break and more.

[0005] These macro activities in turn can be broken down in greater detail.

[0006] The change over (retooling, job change) is a process composed of a series of activities that must be carried out on an industrial machine that produced a product A to make it suitable for producing a product B different from product A.

[0007] For example, the retooling of a rotary flexographic printing machine provides for the preparation of the colors, any cleaning, the change of printing cylinders / sleeves, the feeding of the material, the start-up and adjustment of the pressure and print register, the verification print result and so on until the start of production.

[0008] The activities to be performed during a retooling differ for each type of machine and depend on the structure and technology of the machine. Generally their complexity and duration also depends on the 2 products (or productions) between which the machine undergoes the change over process. In other words, the time and effort required for the changeover depend on many factors, such as the specific industrial machine used, the 2 products for which the machine undergoes the changeover process, the organization and the quantity of manpower. employed for work

[0009] Industrial machines, the operators who work on the machines and their productions are a link in a wider value chain with which they must be coordinated and connected. This leaves a number of technical problems open. The collection of reliable and real-time information on the time and effort used in each production, changes, adjustments etc. they are a key input for production planning, real-time updates and control.

[0010] Such information is also key information for an adequate price of the products, which takes into account its production efforts / costs.

[0011] This information is also a valid support for focusing efficiency improvement initiatives and, subsequently, a basis for their evaluation.

[0012] The collection of such information must be simultaneously: 1) reliable, 2) in real time, 3) detailed, showing not only the status of the macro machine but also all the activities carried out within that phase, 4) collection at competitive costs. In many areas this is still an unresolved challenge.

[0013] Systems are known which evaluate the state of wear or the operating state of an industrial machinery.

[0014] Such known systems are aimed at the predictive maintenance of industrial machines and can include a structure for analyzing the data of industrial machines which guarantees data flows for monitoring the state of wear or health of such industrial machines.

[0015] However systems of this type are not sufficient to tackle the much more complex problems illustrated above.

[0016] The object of the present invention is to solve the aforementioned technical problems by means of a method and a system which allows the operational status of one or more industrial machines and the activities performed by them to be monitored in real time.

[0017] A further object of the invention is to provide a practical and economical solution to the aforementioned problems.

[0018] The invention also has the purpose of providing a basis for focusing analyzes on how to improve the efficiency of production processes and to provide a basis for evaluating the effectiveness of production improvement initiatives, in order to maximize the time in which a machine it is in function with respect to the total time available.

BRIEF DESCRIPTION OF THE INVENTION

[0019] These purposes are achieved by a method for the automatic identification of the operating state of an industrial machinery and of the activities that take place there, where the aforementioned method is implemented by means of a computerized supervisor system of the aforementioned machinery (PLC), at least one video camera and at least one sensor to monitor activities relating to the operating status of the aforementioned industrial machinery and one or more computers, possibly connected in a local network and / or via the Internet, where the aforementioned method includes at least the following steps: a phase of creation of a first timeline on which to represent the operational states or activities carried out in relation to the machinery; a phase of creation of a second timeline derived from the aforementioned first timeline, where for each operational state belonging to the first timeline, intermediate phases of each operational state or intermediate activities carried out in relation to each of the aforementioned intermediate phases of each operational state are represented ; a phase of analysis of the operating conditions of the aforementioned machinery by using one or more sensors; a phase of analysis of the activities of the operators assigned to the aforementioned machinery carried out within the aforementioned second timeline, through the use of the aforementioned video cameras combined with machine learning techniques to interpret the videos collected, where the aforementioned analysis phases are combined with each other to determine the operational status of the industrial machinery and the activities that take place there within the aforementioned second timeline, based on the actions of the operators in charge and / or on the basis of the operating conditions detected of the aforementioned machinery.

[0020] This embodiment has the following advantages and particularities.

[0021] The method of the invention is configured in such a way that the computer recognizes all the different operating states (and sub-states) of the industrial machinery.

[0022] This is achieved by analyzing information from various sources.

[0023] A first source are the sensors mounted on the machine, as well as the machine supervisor (PLC).

These elements typically provide a combination of quantitative or Boolean information.

Their implementation is relatively simple and straightforward, but their application must be corroborated by other sources of information.

[0026] In particular, this information allows us to have a first level of understanding of the moment in which an industrial machine was stopped, for how long, but usually it does not provide us with any information on why it was stopped, and in which operational state is found while stopped.

For example, it is generally not immediately apparent whether the machine has been stopped for a change, whether it has been stopped to adjust some part of the machine or some parameters or whether it has been stopped for a repair. This is all information that usually cannot be inferred from the sensors on the machine alone. To evaluate these operating states it is necessary to understand what activities are performed on the machine and by the machine. Generally this requires seeing which parts of the machine are being changed, what activities are performed by operators, if there are personnel working in certain areas of the machine, which tools they are using (if any), if certain parts of the machine are open or in certain positions and so on.

[0028] All this information becomes triggers that are interpreted together with their timeline to deduce the operating state in which an industrial machine is located and the activities that take place there.

As stated above, the analysis also requires the analysis of the operator's activities on the machine. This is analyzed through the use of cameras and machine learning techniques to interpret the collected videos.

[0030] In order to make video analysis manageable by techniques based on neural networks, the activities can be divided into the single actions (sub-operations) of which they are composed.

[0031] For example: the hand washing activity can be divided into its action of soaping the hands, putting the hands under tap water and drying them.

[0032] Examples of actions: walking, pointing, putting, holding hands under the tap, picking up a tool, turning a tool, etc.

[0033] Examples of activities: washing hands, talking on the phone, drinking tea, replacing parts of an industrial machinery.

[0034] Examples of events: a football match, a job change of an industrial machinery.

[0035] Furthermore, the method of the invention allows a better control and a more complete collection of information relating to the activities of industrial machinery.

[0036] In particular, the method of the invention makes it possible to avoid the operators having to manually enter data relating or representative of one or more operating states of the monitored industrial machinery.

The method also allows for a greater level of detail than manual entry, as well as greater accuracy than manual entry.

[0038] Finally, the method of the invention can be used as a tool for programming production and for processing times and methods.

[0039] According to an embodiment of the invention, the Machine Learning techniques applicable to carry out the aforementioned identification step of one or more operating states of the aforementioned industrial machinery may include computational statistics, pattern recognition, artificial neural networks, filtering, convolutional neural networks and combinations of these.

[0040] According to a further embodiment of the invention, the video camera, in order to identify at least one operating state of the industrial machinery, can record at least one event selected from the position of the operators, the number of operators, the actions of the operators, the components of industrial machinery and more.

[0041] According to an embodiment of the invention, through the use of at least one video camera combined with Machine Learning techniques it is possible to analyze the position, number of operators and actions performed by the operators and through the use of sensors and information coming from the machinery that can be used to identify operating conditions of the machinery, it is possible to combine this information, also taking into account their time sequence and using specific algorithms for each type of machinery, in order to identify the state of the machinery.

[0042] Further characteristics of the invention can be deduced from the dependent claims.

BRIEF DESCRIPTION OF THE FIGURES

[0043] Further characteristics and advantages of the invention will become evident from reading the following description provided by way of non-limiting example, with the aid of the figures illustrated in the attached tables, in which: Figure 1 illustrates a diagram of the main components of the system suitable for implementing the method according to an embodiment of the invention; Figure 2 illustrates a plurality of operating states of an industrial machinery detected according to a first time line in accordance with the method of the invention; Figure 3 illustrates an example of decomposition of an event into activity and of the activities into actions, in accordance with the method of the invention; Figure 4 illustrates steps for collecting and interpreting information related to the operating state of an industrial machinery and the activities that take place there in accordance with the method of the invention; And Figure 5 illustrates in more detail further information related to the operational status of an industrial machinery and the activities that take place there.

DETAILED DESCRIPTION OF THE FIGURES

[0044] The present invention will now be described with particular reference to the attached figures.

[0045] In particular, Figure 1 illustrates a diagram of the main components of the system, where for the application of the method the use of at least one video camera 10 suitable for recording events concerning or occurring in the vicinity of an industrial machinery 20 is envisaged .

[0046] In the diagram of Figure 1 there is also a block (globally indicated with the numerical reference 30) which represents a set of Machine Learning techniques used by the method of the invention in order to determine or identify one or more operating states of the aforementioned industrial machinery 20.

[0047] For the purposes of the present description it is noted that the term Machine Learning refers to a branch of artificial intelligence that studies computer algorithms that improve automatically through experience or data sets. Such methods include artificial neural networks and convolutional neural networks. The latter are a class of multilayer artificial neural networks whose architecture is not completely connected. This means that not every neural link is connected with all the links of the next level. This architecture allows to build convolution and subsampling layers that are commonly identified as feature extraction layers and a fully connected final layer typically identified as a classification layer.

[0048] There is a plurality of approaches to learning using Machine Learning including: supervised learning: inputs and outputs provided as sets for training; unsupervised learning: linked to autonomous recognition of patterns in data; reinforcement learning: learning technique in which the machine learns from the consequences of its actions. Essentially the machine learns by trial and error.

[0049] In more discursive terms, Machine Learning is the science that allows computers to learn and act as humans do, and improve their learning over time autonomously, providing them with data and information in the form of observations and interactions with the real world.

[0050] In the diagram of Figure 1, therefore, block 30 represents a set of Machine Learning techniques used according to the method of the invention in order to determine or identify one or more operating states of the aforementioned industrial machinery 20 to configure, using these techniques of artificial intelligence, a production status reporting system.

[0051] These techniques are applied to the sequence of production steps performed by an industrial machinery and indicated with the arrow 50 in Figure 1 in order to obtain a report or a set of data representative of the various production steps (block 60).

[0052] The system described can also be integrated with the company ERP (Enterprise Resource Planning) system which as known is a type of management software that integrates all the relevant business processes of a company (sales, purchases, warehouse management, accounting etc. - see block 40 of figure 1).

[0053] In general, therefore, the invention relates to a method for identifying the operating state of an industrial machinery, where the aforementioned method comprises at least the following steps: use of at least one video camera and / or at least one sensor to monitor activities relating to the operational status of an industrial machinery; identification of one or more operating states of the aforementioned industrial machinery; where the identification phase of one or more operating states of the aforementioned industrial machinery is carried out by analyzing the images of the aforementioned video camera and / or the data detected by the aforementioned sensor by using Machine Learning techniques.

[0054] According to an aspect of the invention, the video camera 10, in order to identify at least one operating state of the industrial machinery analyzed, can record at least one event selected from the position of the operators, the number of operators, the actions of the operators, state of the components of the aforementioned industrial machinery and more.

[0055] By way of example, the operator actions recorded by the video camera may include the fact that the operator is inserting new printing cylinders or is starting or stopping the operation of the machine.

[0056] Furthermore, examples of the state of the analyzed machine can include the fact that the electrical panels are open, or that the safety doors to be opened during the retooling are open or other.

[0057] According to an embodiment of the invention, the images and videos received from the cameras would be interpreted and analyzed by means of Machine Learning techniques, including, for example, Artificial Neural Networks (ANN) and Convolutional Neural Networks (CNN) or others.

[0058] This requires the machines to be "trained" in advance. This is essentially a process in which the ANN, CNN or other Machine Learning tool is provided with a set of inputs and outputs.

[0059] The supervised training takes place by presenting to the neural network patterns of which the class is known, for example taken from the previous examples, such as the fact that the operator is inserting new printing cylinders or loading a new reel of material, or preparing the ink or by running the material into the machine or other.

[0060] The method of the invention can also provide a step of using at least one sensor to monitor activities relating to the operational state of the industrial machinery.

[0061] For example, the sensors provide information relating to the operation of the machinery, such as the speed of parts of the machinery, temperatures, pressures or other.

[0062] This system can also be put in communication with the HMI system (operator interface) or PLC of the machinery, in order to receive the necessary information directly from these.

More generally, it is noted that the use of machine learning and video cameras to understand operator actions constitutes a salient aspect of the invention.

[0064] This technique is added to an analysis already carried out in different contexts through the use of sensors and information obtained from the machinery processed by means of algorithms. These techniques often have limited results, but combined with the system of the invention used to understand the meaning of the operator actions they can become very effective.

Figure 2 illustrates a plurality of operating states of an industrial machinery detected according to the method of an embodiment of the invention.

[0066] The example of figure 2 relates to a flexographic printing machine.

[0067] The techniques described above are applied to the sequence of production steps performed by an industrial machinery and indicated with the arrow 50 in Figure 1 in order to obtain a report or a set of data representative of the various production steps (block 60).

[0068] In particular, the primary state of the machinery can be detected, that is: in operation, in retooling, in maintenance, or in another operating state.

[0069] The state or the detailed activity carried out by the machinery can also be detected, as well as the activities carried out by operators and their position.

[0070] The whole is collected in order to obtain information with various degrees of detail regarding the machinery, the activities relating thereto, possible efficiency initiatives, improve estimates and programming.

[0071] Figure 3 illustrates a plurality of operational states of an industrial machinery detected ("events" level) and how this "events" line is broken down into the activities that compose them. In turn, the activities are broken down into the basic actions that constitute them.

[0072] In particular, this second timeline refers, in the example of Figure 3, to the change of printing sleeves in the flexographic printing machine where this activity can be divided into a plurality of typical and necessary actions and which are carried out by one or more operators for carrying out the aforementioned change of print sleeves.

[0073] The actions that can be carried out in sequence to carry out the aforementioned change of printing sleeves can be, for example, the extraction of the previous printing sleeve, placing it on the floor of the same, gripping the subsequent printing sleeve and its insertion inside the machine, which can be repeated for each color of the printing machine.

[0074] Figure 4 illustrates steps for collecting and interpreting information related to the operating state of an industrial machinery and the activities that take place there according to the method of the invention.

[0075] In particular, the information gathering step can be carried out, for example, with the aid of one or more video cameras which function as input tools for storing images. The cameras are coupled to a system based on convolutional neural networks (CNN) aimed at recognizing the activities that take place in the vicinity of the machinery, for example the activities indicated in the second timeline illustrated in Figure 3.

[0076] In particular, for this recognition it is possible to proceed with machine learning techniques, such as those indicated above, which serve to identify first of all the position of the person with respect to the machine and the activities carried out in the sample period, for example the activities of the second timeline described above.

[0077] In parallel, the sensors applied to the machine provide the system with a plurality of information on the operation and status of the machine.

[0078] Furthermore, the system manages a memory that is based on the knowledge of the relevant timelines: for example the timeline with the list of the previous operating states of the machine, the timeline with the activities already carried out by the staff during the actual operating states and the timeline of the previous steps activated during the current operational state.

[0079] As regards the stage of interpreting the information collected, it is possible to proceed according to two options.

[0080] The first option consists in carrying out the interpretation of the images by means of hand-coded algorithms.

For example, a precoded algorithm that recognizes a specific state of the machine based on a precoded set of conditions.

[0082] Such an algorithm compares the information collected in the previous phase with a defined set of conditions: if this comparison gives a negative result, the recognition of the operating state of the machine is not modified; vice versa, if this comparison gives a positive result, the start of the recognized status is set in the timeline.

[0083] The second option consists in the interpretation of the set of information using machine learning techniques or using one or more neural networks trained with a set of input conditions (condition patterns) in order to produce as output labels indicating the various recognized operating states,

[0084] Figure 5 illustrates in more detail stages of interpretation of information related to the operating status of an industrial machinery and the activities that take place there.

[0085] For example, starting from a video sample of fixed length, one proceeds by extracting single frames from a sequence of frames. Using a first pre-trained neural network, the position of the people present in the image is identified.

[0086] The neural network e produces as output a set of image coordinates for each person present in it.

[0087] One or more pre-trained neural networks are able to proceed with the recognition of actions starting from an input consisting of a sequence of photos and the positions of the persons analyzed and producing as an output a label that identifies a particular action performed by the aforementioned person.

[0088] When an action is recognized, it is memorized and a timeline is then constructed in which all the previous recognized actions are contained.

[0089] This set of conditions is supplied to a neural network trained to produce as output the activity performed by an operator on the basis of the sequence of recognized actions: the recognized activities are then stored and a time line with all the activities is then constructed performed.

[0090] Obviously, modifications or improvements may be made to the invention as described, dictated by contingent or particular reasons, without thereby departing from the scope of the invention.

Claims (10)

1. Method for the automatic identification of the operating status of an industrial machinery and of the activities that take place there, where the aforementioned method is implemented by means of a computerized supervisor system of the aforementioned machinery (PLC), at least one video camera and at least one sensor to monitor activities relating to the operating status of the aforementioned industrial machinery and one or more computers, possibly connected to a local network and / or via the Internet, where the aforementioned method includes at least the following steps: - a phase of creation of a first timeline on which to represent the operational states or activities carried out in relation to the machinery; - a phase of creation of a second timeline derived from the aforementioned first timeline, where for each operational state belonging to the first timeline, intermediate phases of each operational state or intermediate activities carried out in relation to each of the aforementioned intermediate phases of each state are represented operating; - a phase of analysis of the operating conditions of the aforementioned machinery by using one or more sensors; - a phase of analysis of the activities of the operators assigned to the aforementioned machinery carried out within the aforementioned second timeline, through the use of the aforementioned video cameras combined with machine learning techniques to interpret the videos collected, where the aforementioned analysis phases are combined with each other to determine the operational status of the industrial machinery and the activities that take place there within the aforementioned second time line, on the basis of the actions of the operators in charge and / or on the basis of the operating conditions of the aforementioned machinery. 2. Method as in claim 1, in which through the use of at least one video camera combined with Machine Learning techniques it is possible to analyze position, number of operators and actions performed by operators and through the use of sensors and information coming from the machinery can be used to identify operating conditions of the machinery, where by combining the above information, also taking into account their time sequence and using specific algorithms for each type of machinery, it is possible to identify the state of the machinery. 3. Method as in claim 2, in which the algorithms that recognize a specific operating state of the machine are based on a set of precoded conditions and operate by comparing the information collected by the aforementioned video cameras and sensors on the industrial machinery with a predefined set of conditions . 4. Method as in claim 2, in which the algorithms that recognize an operating state of the industrial machinery are able to interpret the set of information collected in the previous steps (position of operators, activities performed, information from sensors, time sequences) using a or more neural networks trained with a set of input conditions in order to produce as output labels that indicate the various recognized operating states. 5. Method as in claim 2, in which starting from single frames from a sequence of frames, using pre-trained neural networks, features are extracted that allow a classifier of the aforementioned pre-trained neural network to identify the position of the people present in the image and produce a set of coordinates of the image for each person present in it and on the basis of the aforementioned sequence of photos and from the positions of the persons analyzed to produce as an output a label that identifies a particular action performed by said person. The sequence of actions performed and positions are subsequently analyzed by means of further neural networks to identify the activities they make up 6. Method as per the previous claims, characterized by the fact that the aforementioned Machine Learning techniques applicable to carry out the aforementioned phase of identification of one or more operating states of the aforementioned industrial machinery may include computational statistics, pattern recognition, artificial neural networks, filtering , convolutional neural networks and combinations of these. 7. Method as in claim 1, characterized by the fact that the aforementioned video camera, in order to identify at least one operating state of the aforementioned industrial machinery, can record at least one event chosen from the position of the operators, the number of operators, the actions of the operators , the state of the components of the aforementioned industrial machinery and more. 8. Method as in claim 6, characterized in that the aforementioned step of identifying one or more operating states of the aforementioned industrial machinery on the basis of the images of the aforementioned video camera provides for a preliminary phase of supervised training of the neural network or networks used. 9. Method as in claim 1, characterized by the fact that the step of using at least one sensor to monitor activities relating to the operating state of the industrial machinery comprises the use of sensors configured to record information relating to the operation of the aforementioned industrial machinery, selected between speeds of machinery organs, temperatures, pressures or other. 10. Method as in claim 1, characterized in that it provides for a phase of integration with the company ERP (Enterprise Resource Planning) system of the information relating to the operating states of the aforementioned industrial machinery determined during the aforementioned identification phase.