EP3977222A1

EP3977222A1 - Method for controlling operation of a packaging machine and related control unit

Info

Publication number: EP3977222A1
Application number: EP20742875.6A
Authority: EP
Inventors: Luca Cerati; Claudia DE MARIA; Andrea Biondi
Original assignee: GD SpA
Current assignee: GD SpA
Priority date: 2019-05-24
Filing date: 2020-05-22
Publication date: 2022-04-06
Also published as: WO2020240379A1; IT201900007205A1

Abstract

Shown herein is a method for controlling operation of a packaging machine (1) for consumer products (P) comprising the steps of: - acquiring a data flow in the form of video, or audio video, streaming comprising a sequence of frames acquired in sequence, relating to the products (P) being processed in the machine and/or to the materials (M) used to make those products (P); - comparing the acquired data flow with a reference data flow, for detecting any discontinuity in the acquired data flow; - undertaking corrective actions based on the discontinuity detected and/or saving the acquired data flow in a long-term memory (503) following detection of the discontinuity for a subsequent assessment. Also shown herein is a respective control unit (5) for controlling operation of a packaging machine (1) for consumer products (P) comprising: - an acquisition device (501), configured for acquiring a data flow in the form of video, or audio video, streaming comprising a sequence of frames acquired in sequence, relating to the products (P) being processed in the packaging machine (1) and/or to the materials (M) used to make those products (P); - a processing device (502) configured for comparing the data flow acquired by the acquisition device (501) with a reference data flow, for detecting any discontinuity in the acquired data flow; - a long-term memory (503); wherein the processing device (502) is additionally configured for undertaking corrective actions based on the discontinuity detected and/or saving the acquired data flow in the long-term memory (503) following detection of the discontinuity for a subsequent assessment.

Description

METHOD FOR CONTROLLING OPERATION OF A PACKAGING MACHINE AND RELATED CONTROL UNIT

This invention relates to a method for controlling operation of a packaging machine for consumer products and a related control unit.

In particular, this invention relates to a method for controlling operation and the related control unit of an automatic packaging machine for consumer products such as, for example, products for pourable food preparations which are liquid, semi-liquid, powdery, semi-solid, or smoking articles.

An automatic packaging machine for consumer products comprises a plurality of operating units connected to each other and a control apparatus configured for controlling those operating units.

For example, in an automatic packaging machine for products for pourable preparations the operating units may be at least one unwinding unit for unwinding a multi-layer web from a reel; a sterilising unit for sterilising the web unwound; a longitudinal sealing unit for the sterilised web, which forms a continuous tube; a filling unit thanks to which the food product is poured into the continuous tube formed; a forming and separating unit configured for sealing the filled tube along transversal seals which are opposite each other relative to the longitudinal axis of the tube so as to obtain a product and separating the product from the tube by means of a transversal cut; a folding unit, configured for receiving the products and for folding parts of the opposite transversal seals to obtain folded triangular tabs in the product.

In an automatic packaging machine for smoking articles, for example, the operating units are configured for carrying out successive processing operations on articles containing tobacco, starting from a paper web intended to be wrapped around tobacco deposited on the paper web itself, or on disposable cartridges usable for electronic cigarettes.

By way of example, an automatic packaging machine for smoking articles may be: a machine for making cigarette filters; a machine for making cigarettes, starting from tobacco deposited in a continuous paper web, connected to the filter making machine for receiving the filters made from the latter; a storage machine, for storing the cigarettes received from the cigarette making machine; a cigarette packet making machine, connected to the storage machine or to the cigarette making machine, for receiving the cigarettes made; an overwrapping machine, connected to the cigarette packet packaging machine in order to wrap them in a transparent plastic film, also called cellophane; a cartoning machine, connected to the overwrapper in order to form cartons of cigarette packets.

The control apparatus runs a control program for such operating units, in order to carry out automated processes intended for making the consumer products described above and/or for managing the wrapping materials, setting new production formats, supervising production or scheduled maintenance of the operating units themselves.

In order to do that, the control apparatus runs the control program by reading the input data supplied by the field devices and by the sensors connected to each operating unit, analysing the packaging machine status resulting from those input data read and generating output data, based on these input data and on a control program logic, so as to make the operating units of the packaging machine operate.

Over the years increasingly complex industrial automation systems have been introduced in production plant comprising packaging machines, in order to reduce to the minimum indispensable any maintenance work, machine down times, and the number of products rejected in each packaging machine present in the production plant, so as to improve the production efficiency of the packaging machines connected to each other. For example, in a packaging machine for products for pourable preparations, loss of product sterility considerably reduces production efficiency due to the high number of products rejected but, above all, due to the lengthy machine down time necessary in order to carry out a cycle for restoring sterility and making the packaging machine operational again.

For that purpose, often, in addition to the control apparatus, there are also product quality analysis devices present which, locally, carry out parts of the whole industrial process or off-line test procedures and many automation components which, reading the data arriving from the analysis devices and from the control unit input data, modify, during packaging machine operation, some operating parameters of one or more operating units, with the aim of modifying the packaging machine functions and attempting to minimise malfunctions, consequently increasing the production efficiency of the packaging machine itself.

Despite the various product quality analysis devices or the various automation components described above, it might not be possible to identify production plant criticalities which risk a machine stoppage, or a lack of wrapping material, or a high level of defectiveness in the products, or other risk factors, if such criticalities are not known in advance and there has been no identification of operating parameters to be modified in order to reduce or eliminate those criticalities.

In addition, although reading the data arriving from the analysis devices and from the control unit input data, there is often only a partial view of the criticalities and risk factors, given the large number of interdependencies and correlations between the input data of each packaging machine and the possibility that there may be hidden interdependencies present, which are difficult to identify even for an expert operator and designer.

The aim of this invention is to provide a method for controlling operation of a packaging machine for consumer products and a related control unit which is free of the disadvantages described above and, at the same time, is easy and inexpensive to implement.

Accordingly, this invention supplies a method for controlling operation of a packaging machine for consumer products and a related control unit as set out in the appended independent claims.

This invention will now be described with reference to the accompanying drawings, which illustrate an example, non-limiting embodiment of it, in which: - Figure 1 is a schematic view of a packaging machine for consumer products comprising a plurality of operating units for processing the products being processed and the materials used to make the products, a control apparatus for the operating units of the packaging machine, a control unit for controlling packaging machine operation, in accordance with this invention, and an acquisition device configured for acquiring a data flow in the form of video or audio video streaming;

- Figure 2 is a schematic view of the acquisition device of Figure 1 and of a processing device configured for comparing the acquired data flow with a reference data flow, for detecting any discontinuity in the acquired data flow. In Figure 1 , the numeral 1 denotes in its entirety an automatic packaging machine for consumer products P, for example a packaging machine 1 for products P for pourable preparations which may be liquid, or semi-liquid, powdery, granular or semi-solid.

The packaging machine 1 comprises a plurality of operating units configured for processing the products P and the material M used to make those products P. For example, the packaging machine 1 comprises an infeed part 2 comprising a plurality of operating units, made as drum conveyors 201 and placed in sequence, configured for feeding and processing a web of material M, used to make the products P. The web of material M is, for example, fed from a reel (not illustrated).

Although in Figure 1 the material M used to make the products has been shown as a web, for example of a paper material, it should be noticed that the material used to make the products may be, for example, a paperboard blank intended to be folded.

The packaging machine 1 also comprises, by way of example, an outfeed part 3 configured for supplying the packaged products P, which comprises a respective outfeed operating unit, for example made as a linear conveyor 301 configured for feeding the products P, which comprises a planar conveying branch 302 on which the products P are placed resting on it and are conveyed towards a supplying outfeed (not illustrated). Between the infeed part 2 for feeding the material M used to make the products P and the outfeed part 3 for supplying the packaged products P, the packaging machine 1 comprises a packaging part for packaging the products P starting from the material M, not illustrated.

The packaging machine 1 comprises a control apparatus 4 configured for controlling the packaging machine 1 , in particular the operating units 201 , 301 of the packaging machine 1 which are indicated above. The control apparatus 4 is for example, a PLC, which runs a software program installed in the control apparatus 4 itself. That software program is configured for cyclically reading the input data supplied by field devices and by the sensors connected to each operating unit 201 , 301 , for analysing the packaging machine 1 status resulting from those input data read and for generating output data, based on these input data, so as to make the operating units 201 , 301 of the packaging machine operate based on the control program. The packaging machine 1 also comprises a control unit 5 for controlling operation of the packaging machine which comprises an acquisition device 501 , configured for acquiring a data flow in the form of video, or audio video, streaming, that is to say, multi-media streaming, comprising a sequence of frames, that is to say, of images, acquired in sequence relative to the products P being processed in the packaging machine 1 and/or to the materials M used to make the products P.

The acquisition device 501 is, preferably, an image acquisition device, and optionally acquires sounds, at high speed and high resolution and the video, or audio video, flow acquired comprises a set of images, and optionally also of sounds, acquired in sequence which form the sequence of frames.

Hereinafter, for simplicity but without restricting the scope of the invention, the image acquisition device 501 shall be understood to be able to acquire either a video, or audio video, streaming data flow from an initial event E1 to a final event E2, as shown in Figure 2.

The initial event E1 , that is to say, the start of the acquisition, like the final event E2, that is to say, the end of the acquisition, may be commanded by the control apparatus 4 of the packaging machine 1 and correspond to successive moments.

The image acquisition device may comprise a body on which an electronic sensor is positioned, for example an array or arrangement of linear or two- dimensional matrix light-sensitive elements of the CCD or CMOS type, and special optical receiver devices fixed to the body, for example, a lens through which the sensor can receive the diffused light to be acquired. It should be noticed that an image with a resolution of [nxm] pixels may be acquired by means of a single shot using a two-dimensional sensor with two-dimensional matrix of light-sensitive elements [nxm] or by means of n consecutive shots using a linear sensor with m light-sensitive elements. Preferably, the acquisition device 501 is configured for acquiring images in a wavelength range of from 100 nm to 15 pm.

In detail, the acquisition device 501 is configured for acquiring images which are in the following ranges: ultraviolet UV (100 - 400 nm), and/or visible (380 - 750 nm), and/or NIR - Near Infrared (750 nm - 1 .4 pm), and/or SWIR - Short-wavelength infrared (1 .4 pm - 3 pm) and/or MWIR - Medium Wavelength Infrared (3 - 8 pm), and/or LWIR - Long Wavelength Infrared (8 -15 pm) including image thermography.

The acquisition device 501 may, optionally, be configured for acquiring hyperspectral images included in a spectral range of between 400 nm and 4000 nm, preferably between 950 nm and 1700 nm, more preferable still included between 1250 nm and 1600 nm.

In addition, the control unit 5 comprises: a processing device 502, which is configured for comparing the acquired data flow with a reference data flow, for detecting any discontinuity in the acquired data flow.

The term discontinuity refers to an anomalous variation or an unexpected irregularity compared with the reference data flow.

For example, the unexpected irregularity may relate to a reference multi- media signal, which may be uniform or cyclically repetitive (video, or audio video) and may relate to correct operation of the packaging machine 1 . However, the reference data flow may also be determined during a learning step, as described in detail below, as a whole and may not be linked to a specific correct operation of the packaging machine.

In other words, the discontinuity in the streaming data flow may highlight an anomalous operation of the packaging machine 1 relative to the products P being processed in the packaging machine 1 and/or to the materials M used to make the products P. For example, there is a discontinuity if a frame of the streaming data flow is not assessed to be acceptable relative to the reference data flow.

The control unit 5 also comprises a long-term memory 503. Indeed, the processing device 502 is configured for undertaking corrective actions based on the discontinuity detected and/or saving the acquired data flow in the long-term memory 503 following detection of the discontinuity for a subsequent assessment.

The control unit also comprises a temporary memory 504, for example a RAM memory, configured for saving the data flow acquired by the acquisition device 501 .

Thanks to detection of a discontinuity in an acquired data flow, for example in a video, or audio video, streaming relative to a reference data flow, an acquisition viewpoint is defined which is no longer dependent on the input data supplied by field devices and by the sensors connected to the operating units 201 , 301 , but instead is processed based on an external acquisition viewpoint, as if an expert machine operator were present, constantly observing the packaging machine itself, paying particular attention to the products P being processed in the packaging machine 1 and/or to the materials M used to make the products P.

The acquisition device 501 may be positioned facing a portion of packaging machine 1 comprising a plurality of operating units 201 , 301 , for example facing a specific operating unit 201 or 301 , in such a way as to frame a field of view having a“context” which is known beforehand. The term“context” refers to the fact that, given a portion of the packaging machine 1 framed, the scenes acquired and the content of the videos acquired are already known beforehand and do not change over time.

In this way, the processing device 502 can detect the discontinuity in the acquired data flow, for example in the acquired video or audio video streaming, without adding filters relating to the significance of the acquired data flow itself. The“context” being known beforehand, the whole data flow is entirely analysable and therefore the processing is simplified.

Advantageously, the acquisition device 501 may be positioned facing the portion of the packaging machine 1 from the front in an observation point in which a machine operator could be positioned for viewing or monitoring operation of the packaging machine 1 .

Otherwise, the acquisition device 501 may be positioned facing the portion of the packaging machine 1 , that is to say, facing the operating unit 201 or 301 , but from the side rather than from the front, if it is appropriate to acquire the data flow from an observation point different from an observation point of an operator.

The packaging machine 1 may also comprise multiple acquisition devices 501 , which are facing respective portions of packaging machine at respective observation points. It shall be understood that multiple acquisition devices 501 may be connected to the same processing device 502 or each acquisition device 501 may have a respective processing device 502 associated with it.

It should be noticed that the control unit 5 is connectable to the control apparatus 4 of the packaging machine 1 for receiving and/or supplying data and/or commands to the packaging machine 1 itself.

As already indicated, the acquisition device 501 is a video streaming image acquisition device, or an audio and video multi-media acquisition device, configured for acquiring the sequence of N frames, that is to say, of images, in sequence.

The sequence may be that of a continuous streaming or, alternatively, since it is connectable to the control apparatus 4 of the packaging machine 1 , the sequence may be that which the acquisition device 501 can select amongst those of the continuous streaming.

For example, the acquisition device 501 may acquire images, and optionally sounds:

- asynchronously relative to the packaging machine, that is to say, for example, at a periodic acquisition command;

- synchronously with the packaging machine 1 , that is to say, at packaging machine 1 significant events (for example, at one or more specific machine encoder degrees in each machine cycle, that is to say, “in phase” with machine cyclical operation).

In the case of asynchronous acquisition, if the periodic acquisition command has an acquisition frequency which is the maximum possible, this is continuous streaming (particular case of asynchronous acquisition with maximum acquisition frequency).

If the periodic acquisition command has a low acquisition frequency, a stroboscopic video, or audio video, acquisition is carried out, if it is at a high acquisition frequency an acquisition with slow motion effect is carried out (for example with a high number of fps“frames per second” - 100 fps rather than the usual 25 fps/30fps).

The number of frames per second acquired may range, for example, from a minimum of 15 fps with low quality video streaming, to a maximum of 300fps for extremely high quality video streaming with slow motion effect.

Moreover, also significant for video streaming quality is the number of pixels acquired per frame, which considering a pixel matrix [nxm] may range from 320x240 pixels/frame, for low quality video streaming, to 640x480 pixels/frame or 854x480 pixels per frame, for a medium quality video, up to 1280x720 (720p Standard HD) or 1920x1080 (1080p Full HD, also called 2K), in compliance with the most common proportions 4:3 and 16:9 (aspect ratio) of video formats.

The acquired data flow may therefore be produced using images, and optionally sounds, periodically acquired over time in sequence or using images, and optionally sounds, which are acquired in sequence and at packaging machine 1 significant events, as previously illustrated.

Alternatively, the data flow may be acquired starting from one of the significant events, if this is considered as the initial event E1 from which the streaming data flow is acquired, and is a continuous streaming.

For example, the video or audio video streaming data flow may be acquired at a start of a processing cycle for processing the products P, or processing of the materials M used to make those products P. For example, and as described in more detail below, the acquisition device 501 may be positioned facing from the front a filling unit for the pourable preparation and, for example, the streaming data flow may be activated immediately before the filling starts and may be deactivated when filling has ended.

The initial event E1 of the start of the acquisition, like the final event E2 of the end of the acquisition, may be commanded by the control apparatus 4 of the packaging machine 1 , which also controls a movement of the product P in front of the acquisition device 501 itself, relative to the movement itself. The control unit 5, when connected to the control apparatus 4, can modify a packaging machine 1 operation with respect to the discontinuity detected, for example to modify packaging machine 1 operating parameters associated with the operating units 201 , 301 .

Flowever, alternatively, the control unit 5 can activate a dedicated screen page of an operator interface (HMI - Fluman Machine Interface) of the control apparatus 4 of the packaging machine 1 to show, for example to a machine operator, the video, or audio video, streaming saved in the long term memory 503 after the discontinuity and highlighting the discontinuity detected, without modifying the operating parameters of the self-same packaging machine 1 .

In this case, the subsequent assessment indicated above, may be carried out by an expert operator who has the opportunity to perform an off-line analysis of the video, or audio video, flow showing the discontinuity detected. Similarly, the corrective actions which the processing device 502 may undertake, based on the discontinuity detected, may also relate to the detection logic itself. Indeed, the processing device 502 may be configured for correcting a discontinuity detection logic, for example correcting a mathematical model for said detection of the discontinuity.

That correction may take place by adapting the mathematical model used and/or selecting a different mathematical model, for example selected from a library of mathematical models available, based on metrics detected from the streaming data flow.

For the purposes of discontinuity detection, the processing device 502 comprises a software component configured for analysing the acquired data flow which is installed in the processing device 502 itself. The times for processing the data flow must be limited in order to modify packaging machine 1 operation and/or to correct the logic for detection of the discontinuity in the shortest possible time, so as to increase packaging machine 1 production efficiency.

According to a preferred embodiment, the data flow acquisition device 501 is separate from the processing device 502 in order to increase processing efficiency.

However, in a different embodiment, the acquisition device 501 and the processing device 502 may be integrated with each other but in any case their functions remain separate.

For that purpose, as illustrated in Figure 2, in order to compare the acquired data flow with a reference data flow the processing device 502 may comprise a first component 505 for carrying out, during a first analysing step, an analysis of the data flow, for detecting at least one metric of the data flow, and a second component 506 for carrying out, during a second detecting step, a detection of the discontinuity, analysing the metric detected by the first component, during the first analysing step, by means of the mathematical model for detection of the discontinuity.

The first analysis of the analogue or digital video, or audio video, streaming may be carried out on the streaming data flow saved in the temporary memory 504.

By adopting the first component 505 and the second component 506, the analysis of the video, or audio video, streaming data flow can be rendered more efficient and guarantee high performance.

Looking in more detail at the first component 505, it comprises a converting module 505’, configured for converting the video, or audio video, streaming data flow which comprises the sequence of N frames acquired in sequence from the initial event E1 to the final event E2, and for obtaining a vector field 505” indicating a direction of a movement and an intensity of the movement of a plurality of uniform zones of pixels in each frame relative to a base frame.

For an analysis of this type intended to obtain the intensity and direction of the movement, the base frame may correspond to the frame initially acquired at the initial event E1 . Therefore, in each frame after the initial one, for each uniform zone of pixels, the intensity and the direction of the movement relative to the base frame will be assessed.

The first component 505 also comprises a statistics module 505”’ for analysing, by means of a statistical analysis algorithm, during a statistical analysis step following the converting step, the vector field 505” and for identifying some significant parameters of each frame N, that is to say, the so-called metrics, obtaining a set of data having reduced dimensions, that is to say, a data matrix having the size B x N indicated with the number 507, wherein B indicates the number of metrics identified.

For example, significant metrics may be the kurtosis of the angle, or the average intensity of the vector field (that is to say, the average intensity of the modulus of a movement vector).

The converting module 505’ may, for example, be produced by means of an Optical Flow type algorithm, for example Dense Optical Flow, which is configured for considering the motion of an object in consecutive frames of a frequency of frames acquired in sequence and therefore to assess a movement over time between the object and the acquisition device 501 , assessing the movement of the position of a same pixel over time relative to the position of the self-same pixel in the base frame.

Since, as already indicated, the context of the frames framed does not change over time, advantageously, the converting module 505’ can identify in each frame the plurality of uniform zones of pixels, that is to say, closed zones inside which there are pixels present which meet the uniformity criteria. These closed zones are also called voxels.

For each of these zones, the Optical Flow type algorithm can consider the movement vector, defined by a pair of values given by the direction of the movement and by the intensity of the movement relative to the same zone of the reference frame for converting the frames, each defined by [nxm] pixels, and obtain the vector field 505” previously described.

It should be noticed that, if the number of the plurality of uniform zones (that is to say, the voxels) is V, then the vector field 505” has a number of values equal to V (number of voxels) x 2 (the direction and intensity of the movement of each voxel) x N (number of frames).

The statistical processing carried out by the statistics module 505’” on the vector field 505” allows a further reduction in the data associated with each frame in order to identify at least the significant metric of each frame N. Preferably, there are at least 2 metrics of each frame N and there may even be as many as 40.

Therefore, advantageously, following the processing carried out by the first component 505 during the first analysing step, it is possible to convert the video streaming data flow, comprising N frames each of which comprises a total number of pixels [nxm], into a matrix of metrics having a reduced size BxN, with more simple processing and analysis which can advantageously be analysed in shorter times by the second component 506, dedicated to detection of the discontinuity.

Therefore, thanks to the first analysing step carried out by the first component 505, the content of each frame is converted and is simplified, detecting at least one, or a plurality of metrics which summarise the information content of the frame itself.

For detection of the discontinuity, the second component 506 comprises a classifying module 506’ for classifying, during a classifying step, the metrics of each of the frames N of the video and/or audio video flow and an assessing module 506” for assessing, during an assessing step, whether each frame of the sequence should be considered acceptable, that is to say, “OK”, or unacceptable, that is to say,“NO”.

For classifying, the classifying module 506’ may comprise an algorithm of the Support Vector Machine type, for example, with which it is possible to associate with each frame a point in a Cartesian plane, if there are two metrics and they are for example the kurtosis of the angle and the average intensity of the field, or in a space having K dimensions if there are more than 2 metrics and they are equal to K.

Taking into consideration only two metrics, each frame of the sequence of frames N of the streaming data flow may therefore be represented in the Cartesian plane gradually as it is acquired, in accordance with the Support Vector Machine algorithm.

Thanks to the classifying module 506’, it is possible to consider multiple metrics simultaneously for each video frame, and to classify the video frame itself by means of complex criteria, which consider a combination of multiple metrics.

The result of the classifying module 506’ is then processed by the assessing module 506”, for the assessment of the acceptability of each frame, which considers the comparison with the reference data flow.

For that assessment, as the reference data flow, the assessing module 506” may take into consideration classifying maps, which identify sets of “OK” acceptable frames or sets of “NO” unacceptable frames, and assess whether the acquired frame is included in one of the classifying maps having acceptable frames, or in another of the maps having unacceptable frames. The discontinuity in the acquired data flow is detected if one of the frames of the sequence belongs to an unacceptable frames classifying map.

The classifying maps are defined during a learning step which precedes the operating step of packaging machine operation.

Therefore, in order for the second component 506 to be able to effectively detect the discontinuity in the video, or audio video, data flow acquired during an operating step of packaging machine operation, it is necessary to ensure that said operating step is preceded by the learning step.

During the learning step, the acquisition device 501 is configured for acquiring video, or audio video, streaming data flows in a controlled way, so that the processing device 502 can process them by means of the first component 505 and a configuring module (non illustrated) of the second component 506 prepared for configuring the classifying maps a posteriori. The classifying module can request the decision of an operator for each acquired streaming data flow during the learning step.

Indeed, the configuring module can submit each acquired data flow to an operator and receive indications about the acceptability, or lack of acceptability, of the data flow and therefore of the N frames acquired.

For each streaming data flow of the learning step, the operator can assess the presence or absence of a discontinuity capable of introducing a defectiveness in the products P being processed in the packaging machine 1 and/or in the use of the materials M used to make the products P.

Therefore, during the learning step, it may be the operator who assesses each streaming data flow and classifies it as“OK” or“NO”.

Alternatively, automatic methods may exist for configuring without a supervisor, which, during the learning step, are supplied with data flows that refer exclusively to correct operation of the packaging machine 1 and in which the deviation from them is assessed in statistical terms. Such automatic methods are faster, but less reliable.

Therefore, during the learning step, the configuring module can configure classifying criteria, identifying and aggregating sets of “OK” acceptable frames or sets of“NO” unacceptable frames, taking into consideration all of the frames of all of the streaming data flows, as acquired, and defining a reference data flow by aggregating the a posteriori assessment of all of the streaming data flows of the learning step, by the operator or by the automatic configuration methods.

Thanks to the learning step, the reference data flow is constructed a posteriori and may relate to packaging machine 1 operation which is correct, or incorrect if the decision of the operator is requested, or only correct if automatic configuration methods without a supervisor are used.

In the case of the classifying module in which the classifying algorithm is of the Support Vector Machine type, the classifying criteria are classifying maps which define boundaries between acceptance and non-acceptance spaces and which may be acceptance and non-acceptance maps.

At the end of the learning step, the classifying maps having been configured, the assessing module 506” is capable of assessing each frame, automatically, verifying which classifying map the frame belongs to.

By comparing each frame of the streaming data flow with the reference data flow, defined by means of the classifying maps, it is possible to identify the discontinuity in the acquired data flow. For example, there is discontinuity when a frame of the N frames of the streaming data flow belongs to a non- acceptance map.

It should be noticed that the classifying module 506’ of the second component 506 may even alternatively be produced using Neural Network or Random Forest type algorithms which do not necessitate classifying maps.

Flowever, even with algorithms of this type, the operating step of packaging machine operation must be preceded by the learning step, in which the operator assesses each streaming data flow and classifies it as“OK” or “NO”, so that the assessing module 506” can define the classifying criteria necessary so that subsequently, during the operating step of operation, the processing device 502 can classify each frame as acceptable, or not, in automatic mode. That is to say, even with different types of algorithms, it is necessary to define the reference data flow by means of the learning step in order to allow identification of the discontinuity in the acquired data flow.

If the processing device 502 is configured for correcting the discontinuity detection logic by correcting the mathematical model, for example, for that purpose the classifying module 506’ and/or the assessing module 506” may be modified in order to carry out an even more precise assessment of the streaming data flow.

As shown in Figure 1 , the acquisition device 501 , the processing device 502, the long-term memory 503 and the temporary memory 504 of the control unit 5 are connected to each other and are also connected to the control apparatus 4 by means of a communication network 6, for example the Internet or the factory LAN (Local Area Network).

Indeed, it should be noticed that, in this application, terms such as“unit”, “apparatus”,“device”,“component”,“module”, refer to one or more entities which are connected to, or form part of, an information technology apparatus with one or more specific functions, wherein such entities may be hardware, a combination of hardware and software, or exclusively software. For example, a component may be, without limitation, a process being executed on a processor, a processor, a hard disk unit, multiple storage units (optical or magnetic storage media) including a solid state storage unit; an information technology data item; a software program which can be run by a computer and/or a computer and/or a hardware calculation unit as a programmable hardware component or an application in the internet cloud. By way of example, both an application being run on an information technology server and the information technology server itself may be considered a component. One or more components may reside in an execution process and a component may be located on a computer and/or distributed between two or more computers. Moreover, the components as described herein may be executed by various computer-readable storage media having various data structures saved on them. For example, the long-term memory 503, usually made using a set of hard disks, even SSD, could be saved in an Internet cloud and not physically reside in the same information technology component in which the processing device 502 resides.

Similarly, the first component 505 and the second component 506 of the processing device 502 may reside on different hardware specially designed in order to obtain distributed processing for an efficient analysis of the video, or audio video, streaming or in the same hardware, but in processes which are separate from each other.

For example, consider a packaging machine for food products P for pourable preparations for which a general description has already been provided and therefore which will not be described again now.

If the product P is a food product, we have said that the packaging machine comprises the sterile processing unit in order to guarantee that the food product P is packaged in a sterile way in a protected atmosphere. The discontinuity of interest, for a machine of that type, may not be linked to a possible jamming of the materials with consequent packaging machine stop, as could occur in a packaging machine for smoking articles, but may be linked to events which could cause a loss of sterility in the packaging of the consumer food products P, for example a poor quality of the longitudinal and/or transversal seals.

In addition to the sterile processing unit, the packaging machine comprises the filling unit for filling the continuous tube. The discontinuity of interest detected may be linked to an incorrect filling due to excessive pressure used when pouring the pourable food product into the continuous tube.

By means of the processing by the first component 505 and the second component 506 of the processing device 502 there may be detection of a discontinuity in the video flow of the sterile processing unit, and/or of the filling unit, and/or of the longitudinal sealing unit, and/or of the forming and separating unit and therefore it may be possible to avoid a subsequent rejection of products packaged in an atmosphere which is no longer protected, with a significant increase in production efficiency.

In use, during the operating step of packaging machine operation, a method for controlling operation of a packaging machine 1 for consumer products P is proposed, comprising the steps of:

- acquiring a data flow, in particular in the form of video, or audio video, streaming relating to the products P being processed in the packaging machine 1 and/or to the materials M used to make those products;

- comparing the acquired data flow with a reference data flow, for detecting any discontinuity in the acquired data flow;

- undertaking corrective actions based on the discontinuity detected and/or saving the acquired data flow in a long-term memory 503 following detection of the discontinuity for a subsequent assessment.

The method also comprises the further step of saving the acquired data flow in a temporary memory 504, for example a RAM memory.

As regards setting the reference data flow, the control method may comprise the further step of defining the reference data flow by means of a data flow previously saved in the temporary memory 504, for example relating to correct operation of the packaging machine 1 , and/or receiving a reference start setting instruction from an operator and defining the reference data flow by means of the data flow acquired following that start setting instruction.

For example, the reference data flow may be that which, during a predetermined packaging machine 1 operating time period, corresponds to a maximum production efficiency of the packaging machine 1 . If the packaging machine 1 is kept under observation for a time period which is longer, for example, than the predetermined period, amongst all of the acquired data flows it may be possible to select as the reference data flow the acquired data flow which corresponds to the best performance of the packaging machine 1 itself.

However, considering the fact that the processing device 502 may comprise the second component 506 dedicated to executing the mathematical model for detecting the discontinuity which necessitates the learning step, the reference start setting instruction may even correspond to the start of the learning step, since the data flows acquired during the learning step all contribute to defining the reference data flow to be considered during the packaging machine operating step. During the learning step, each acquired data flow may be saved in the temporary memory 504.

During the operating step of packaging machine operation, the step of comparing the acquired data flow with a reference data flow comprises a first step of analysing the data flow, for detecting at least one metric of the data flow, preferably a plurality of metrics of the data flow, and a second step of detecting the discontinuity for analysing the metric detected during the first step and assessing the discontinuity in the acquired data flow relative to the reference data flow defined during the learning step, preceding the operating step of packaging machine operation.

The first analysing step comprises converting the video streaming data flow, comprising the sequence of N frames acquired in sequence, and obtaining a vector field 505” indicating, for each frame, a direction of a movement and an intensity of the movement of a plurality of uniform zones of pixels relative to a base frame. That converting step may be carried out by means of the Optical Flow algorithm.

The first analysing step additionally comprises a statistical analysis step, following the converting step, for analysing the vector field 505” and for identifying at least one significant parameter of each frame N, preferably a plurality B of significant parameters of each frame, that is to say, the so- called metrics.

Thanks to the first analysing step, the content of each frame is converted and simplified, by detecting a plurality of metrics which summarise the information content of the frame itself.

The second step of detecting the discontinuity comprises a classifying step, for classifying the metric, or the plurality B of metrics of each of the frames N of the streaming data flow and an assessing step, following the classifying step, for assessing whether each frame of the sequence should be considered acceptable, that is to say,“OK”, or unacceptable, that is to say, “NO” based on the classification of each metric.

The assessing step, during the packaging machine 1 operating step, comprises comparing each frame of the acquired streaming data flow with a reference data flow defined considering classifying maps, which identify sets of “OK” acceptable frames or sets of “NO” unacceptable frames, and assessing whether the frame is included in one of the classifying maps having acceptable frames, or in another of the maps having unacceptable frames.

The assessing step also comprises assessing the discontinuity in the acquired data flow if one of the frames of the sequence belongs to an unacceptable frames classifying map.

The method also comprises the step of defining the classifying maps during the learning step which precedes the operating step of packaging machine operation.

During the learning step, the method comprises the step of acquiring video, or audio video, streaming data flows, in a controlled way for allowing processing by means of the first analysing step and for obtaining the plurality of metrics and a configuring step for configuring the classifying maps a posteriori, after having obtained for all of the streaming data flows acquired during the learning step the decision from an operator, or from automatic configuration methods.

During the configuring step, the method according to this invention may comprise classifying the metrics of each of the frames N of the streaming data flow and, subsequently, submitting the acquired streaming data flow to an operator for receiving indications about the acceptability, or lack of acceptability, of the data flow and therefore of the N frames acquired.

Alternatively, during the configuring step, the method according to this invention may comprise acquiring streaming data flows which all relate to correct operation of the packaging machine 1 , classifying the metrics of each of the frames N of the streaming data flow and, subsequently, statistically analysing all of the acquired streaming data flows.

Therefore, the learning step comprises the further step of configuring the classifying maps, that is to say, the reference data flow, a posteriori, after the decision from the operator or the automatic configuration methods, identifying and aggregating sets of acceptable frames, or sets of unacceptable frames, taking into consideration all of the frames of all of the streaming data flows acquired during the learning step itself.

In detail, the step of undertaking corrective actions may comprise the step of modifying the operation of the packaging machine, for example modifying operating parameters of the packaging machine, or activating a correction of a mathematical model for detection of the discontinuity.

For example, if the acquisition of the video streaming relates to the filling unit for filling the continuous tube (or individual containers in other packaging machines), and the discontinuity detected is an incorrect filling due to excessive pressure used when pouring the pourable food product into the continuous tube, with respect to detection of the discontinuity in the streaming data flow, that is to say, with respect to detection of a non-optimal start of filling (splashes or diversion of the filling flow), there may be temporary reductions in the pourable product supply pressure and the packaging machine production speed.

As already indicated, the correction may take place by adapting the mathematical model used and/or selecting a different mathematical model, for example selected from a library of mathematical models available, based on the metrics detected.

For example, the classifying module 506’ of the second component 506 of the processing device 502 may be modified so as to deactivate classifying by means of Support Vector Machine, activating that using Neural Networks, if the discontinuity in the streaming data flow is more easily identifiable using the latter.

Indeed, there is the step of connecting to a network 6, for example the Internet, a control apparatus 4, for controlling the operating units 201 , 301 of the packaging machine 1 and a control unit 5 for controlling packaging machine 1 operation, and there is the step of preparing a communication software in the control apparatus 4 and in the control unit 5 so that the control apparatus 4 receives any warnings about video, or audio video, streaming data flow discontinuity from the control unit 5 and the control unit 5 receives warnings about events indicative of packaging machine 1 operation from the control apparatus 4, for example the initial event E1 of acquisition of the streaming data flow and the final event E2 of stopping the acquisition.

The step of acquiring a data flow comprises the step of acquiring the sequence of images from the initial event E1 to the final event E2, asynchronously relative to the packaging machine 1 , at a periodic acquisition command, or synchronously or in phase with events indicative of that operation, for example cyclical operating events.

Advantageously, the control method also comprises the further step of identifying at least one portion of packaging machine 1 comprising a plurality of operating units for packaging the products P and/or processing the materials M, and positioning an acquisition device, 501 , configured for acquiring the video, or audio video, streaming data flow, facing the portion of packaging machine 1 identified, for example one of the operating units 201 , 301 , for associating the discontinuity detected with said portion.

In particular the streaming data flow may be only video or audio video, that is to say, multi-media.

In other words, the control method may comprise the step of positioning the acquisition device 501 in the same observation point in which an expert machine operator could be positioned for viewing packaging machine 1 operation at the portion of machine identified.

The control method of this invention amounts to a smart supervising or monitoring of a complex operation such as that of a modern packaging machine with high production speed. The control method of this invention allows easy control of the operation of a packaging machine in a non-traditional way, by considering a viewpoint outside the packaging machine by means of acquisition of a video, or audio video, flow in which to detect anomalous discontinuities or irregularities. It should be noticed that, advantageously, the discontinuity detected may give rise to corrective actions on the packaging machine itself, but not only that, since the data flow may also advantageously be saved for an a posteriori control by an expert operator.

Claims

1 . Method for controlling operation of a packaging machine (1 ) for consumer products (P) comprising the steps of:

- acquiring a data flow in the form of video, or audio video streaming, comprising a sequence of frames acquired in sequence, relating to the products (P) being processed in the machine and/or to the materials (M) used to make those products (P);

- undertaking corrective actions based on the discontinuity detected and/or saving the acquired data flow in a long-term memory (503) following detection of the discontinuity for a subsequent assessment.

2. The method according to claim 1 , wherein it comprises the further step of saving the acquired data flow in a temporary memory (504), for example a RAM memory.

3. The method according to claim 2, wherein it comprises a further step of setting the reference data flow which comprises the step of: defining the reference data flow by means of a data flow previously saved in the temporary memory (504).

4. The method according to one of claims 1 , or 2, wherein it comprises the further step of receiving a start setting instruction and defining the reference data flow following the start setting instruction during a learning step preceding an operating step of packaging machine operation.

5. The method according to one of the preceding claims, wherein, during an operating step of packaging machine (1 ) operation, the step of comparing the acquired data flow with the reference data flow comprises a first step of analysing the data flow, for detecting at least one metric of the data flow, preferably a plurality of metrics, and a second step of detecting the discontinuity, for analysing the metric detected during the first step and assessing the discontinuity in the acquired data flow relative to the reference data flow, the method comprising the further step of defining the reference data flow during a learning step, preceding the operating step of packaging machine operation.

6. The method according to claim 5, wherein the first analysing step comprises a converting step for converting the video streaming data flow, and for obtaining a vector field (505”) indicating, for each frame, a direction of a movement and an intensity of the movement of a plurality of uniform zones of pixels relative to a base frame, for example by means of an Optical Flow algorithm.

7. The method according to claim 6, wherein the first analysing step additionally comprises a statistical analysis step, following the converting step, for analysing the vector field (505”) and for identifying at least the metric of each frame, preferably the plurality of metrics.

8. The method according to one of claims 5 to 7, wherein the second step of detecting the discontinuity comprises carrying out a classifying step, for classifying the metric, or the plurality of metrics, of each of the frames of the streaming data flow and an assessing step, following the classifying step, for assessing whether each frame of the sequence should be considered acceptable, or unacceptable, based on the classification of each metric.

9. The method according to claim 8, wherein the assessing step comprises comparing each frame of the acquired streaming data flow with the reference data flow defined considering classifying maps, which identify sets of “OK” acceptable frames or sets of “NO” unacceptable frames, assessing whether the acquired frame is included in one of the classifying maps having acceptable frames, or in another of the maps having unacceptable frames.

10. The method according to claim 9, wherein the assessing step also comprises assessing the discontinuity in the acquired data flow if one of the frames of the sequence belongs to an unacceptable frames classifying map.

1 1 . The method according to one of claims 5 to 10, and also comprising the step of defining, during the learning step which precedes the operating step of packaging machine operation, the reference data flow by means of classifying maps.

12. The method according to claim 1 1 , wherein during the learning step, the method comprises the step of acquiring video, or audio video, streaming data flows, in a controlled way and allowing processing by means of the first analysing step and, then, configuring of the classifying maps a posteriori, after having obtained for all of the streaming data flows acquired during the learning step an acceptability decision from an operator or from automatic configuration methods.

13. The method according to one or more of the preceding claims, wherein said subsequent assessment comprises modifying the operation of the packaging machine (1 ), for example modifying operating parameters of the packaging machine (1 ).

14. The method according to one of the preceding claims, wherein the step of carrying out the subsequent assessment comprises activating a correction of the detection of the discontinuity, for example correcting a mathematical model for said detection of the discontinuity.

15. The method according to one or more of the preceding claims, wherein said step of acquiring a data flow comprises the step of acquiring the sequence of frames from an initial event (E1 ) to a final event (E2), asynchronously relative to the packaging machine (1 ), at a periodic acquisition command, or synchronously or in phase with events indicative of packaging machine (1 ) operation.

16. The method according to one or more of the preceding claims, wherein it comprises the further step of identifying at least one portion of packaging machine (1 ) comprising a plurality of operating units (201 ; 301 ) for packaging the products (P) and/or processing the materials (M), and positioning an acquisition device (501 ), configured for acquiring said video, or audio video, streaming data flow, facing said portion for associating the discontinuity detected with said portion.

17. Control unit (5) for controlling operation of a packaging machine (1 ) for consumer products (P) comprising: - an acquisition device (501 ), configured for acquiring a data flow in the form of video, or audio video, streaming comprising a sequence of frames acquired in sequence, relating to the products (P) being processed in the packaging machine (1 ) and/or to the materials (M) used to make those products (P);

- a processing device (502) configured for comparing the data flow acquired by the acquisition device (501 ) with a reference data flow, for detecting any discontinuity in the acquired data flow;

- a long-term memory (503); wherein

- the processing device (502) is additionally configured for undertaking corrective actions based on the discontinuity detected and/or saving the acquired data flow in the long-term memory (503) following detection of the discontinuity for a subsequent assessment.

18. The control unit according to claim 17, wherein the processing device (502) comprises a first component (505) for carrying out an analysis of the streaming data flow, for detecting at least one metric of the data flow, preferably a plurality of metrics, and a second component (506) for carrying out a detection of the discontinuity, analysing the metric, or the plurality of metrics, detected by the first component (505).

19. The control unit according to claim 18, wherein the first component (505) comprises a converting module (505’), configured for converting the video, or audio video, streaming data flow, and for obtaining a vector field (505”) indicating a direction of a movement and an intensity of the movement of a plurality of uniform zones of pixels in each frame relative to a base frame and a statistics module (505”’) for statistically analysing the vector field (505”) and identifying the at least one metric, or preferably the plurality of metrics, for example a kurtosis of the angle and an average intensity of the field, of each frame obtaining a set of data having reduced dimensions (507).

20. The control unit according to claim 18, or 19, wherein the second component (506) comprises a classifying module (506’) for classifying the at least one metric of each of the frames of the video and/or audio video flow and an assessing module (506”) for assessing whether each frame of the sequence of frames should be considered acceptable, that is to say, “OK”, or unacceptable, that is to say, “NO”, which is configured for comparing each frame with classifying maps having acceptable frames and with maps having unacceptable frames which define the reference data flow.

21 . The control unit according to claim 20, wherein the second component (506) comprises a configuring module, which is prepared for configuring during a learning step which precedes an operating step of packaging machine (1 ) operation, the classifying maps which define the reference data flow.

22. The control unit according to claim 21 , wherein during the learning step, the configuring module is suitable for configuring the classifying maps a posteriori, after having obtained for all of the streaming data flows acquired during the learning step a decision from an operator or from automatic configuration methods.

23. The control unit according to one of claims 20 to 23, wherein the discontinuity in the acquired data flow is detected if one of the frames of the sequence belongs to an unacceptable frames classifying map.

24. The control unit according to one of claims 18 to 23, wherein the processing device (502) is configured for correcting a discontinuity detection logic, for example correcting a mathematical model for said discontinuity detection.

25. The control unit according to claim 24, when it is dependent on one of claims 20 to 23, wherein the processing device (506) is configured for modifying the classifying module (506’) and/or the assessing module (506”) in order to carry out an even more precise assessment of the streaming data flow.

26. The control unit according to one of claims 17 to 25, wherein the packaging machine (1 ) comprises a control apparatus (4), the control unit (5) being connectable to the control apparatus for modifying the operation of the packaging machine (1 ), for example modifying operating parameters of the self-same packaging machine (1 ).

27. The control unit according to claim 26, wherein the control apparatus (4) is configured for controlling an initial event (E1 ) of the start of acquisition of the data flow and a final event (E2) of the end of the acquisition, relative to a movement of a product (P) in front of the self-same acquisition device (501 ).

28. The control unit according to one of claims 17 to 27, wherein the acquisition device (501 ) is amongst: high speed and/or high resolution video cameras, which are configured for acquiring images in sequences in a wavelength range of from 100 nm to 15 pm and/or in a range of from 15 to 300 frames/sec.