WO2021110226A1

WO2021110226A1 - A method of monitoring a production area and a system thereof

Info

Publication number: WO2021110226A1
Application number: PCT/DK2020/050343
Authority: WO
Inventors: Kasper Koops Kratmann; Jason Stege
Original assignee: Claviate Aps
Priority date: 2019-12-02
Filing date: 2020-12-02
Publication date: 2021-06-10

Abstract

The invention relates to a method of monitoring an industrial site, a system, a computer program and a computer-readable medium thereof. A number of cameras are positioned relative to the industrial site and connected to a data processing apparatus. The data (5) processing apparatus comprises a processor configured to analyse the raw image data using an object detection algorithm. The object detection algorithm is configured to detect objects and remove them from the images, where the modified image data is stored in an anonymous image database. Validated objects are used to retrain the neural network, where the validation is done by a data controller or automatically based on (10) detection confidence level.

Description

A method of monitoring a production area and a system thereof Field of the Invention

The present invention relates to a method of monitoring an industrial site, where a num ber of cameras are positioned relative to the industrial site and capturing image data of the industrial site, the captured image data is transmitted to a data processing apparatus which applies an object identification algorithm to the captured image data.

The present invention also relates to a system for monitoring an industrial site.

Furthermore, the present invention relates to a computer program and a data carrying medium thereof.

Background of the Invention

It is known to use surveillance systems to monitor manufacturing facilities, assembly areas, factory floors, storage areas, loading docks, yards or other industrial sites in which objects are being manufactured, assembled, loaded for transport, moved to stor age or otherwise handled. The production and assembly of large objects often require much space and the operation of large, heavy equipment, such as cranes, fork lifts, cus tomised transport vehicles and the like. The components for such large objects often require the use of large or custom made production machinery.

Large structural objects, such as wind turbine components, aircraft components, vehicle components, building components or the like, are often assembled at large production facilities or at large assembly sites. The assembly site may in example be a harbour area, an assembly hall, an industrial yard or a temporary site. During assembly, the object may be moved from one station or area to another station or area for further assembly. This requires components to be delivered on time and in the right order, particularly if the assembled object is moved along a conveyor system. Different workers may perform different tasks on the large object at the same time, and some tasks cannot be initiated before other tasks are completed.

Thus there is a desire to monitor the performance and downtime of the production, as well as monitoring the flow of components and objects during assembly. For insurance reasons, it may be desired to monitor certain areas or the entire site. It is known to use cameras to capture images of the monitored area and link boxes, or data loggers, to gather operating data from the production machinery. In the event of an accident or downtime, the stored image data and operating data can then be analysed for insurance reasons or for identifying the root causes of the downtime or faults.

In some instances, the installation of large objects to form large structural units and the operation thereof may be monitored, particularly for offshore structures. It may be de sired to monitor the loading and unloading of equipment, goods, liquids or personal for insurance or safety reasons.

Tracking of objects may also be used to optimize the production or assembly process and create a more effective flow of objects. Statistical analysis of the process steps and the flow of components may be used to adjust or change the flow and how each step is performed, thereby increasing the production or assembly and saving costs. However, capturing and storing image data of the industrial site may require consents from the people on the images due to the data protection law, which can be a complex and time consuming process if many people are moving in and out of the monitored area. Fur thermore, data controllers may manually monitor the image data to follow objects of interest, but they may lose focus of the objects due to other events or people in the images.

It is known to use facial identification algorithms to identify persons or individuals in an image, wherein the parameters of the facial identification algorithm are typically determined based on training and testing. The training and testing are typically per- formed using simple models or standardised datasets, such as Coco or ImageNet which focus on everyday objects and people. Conventional object detection algorithms are able to calculate a confidence score for each individual detected in the image, which indicate the accuracy of the detection, and apply a bounding box around each detected individual. Object identification algorithms are commonly used in CCTV-surveillance of pedestrians in public sites or streets to track their movements and behaviour. Such person identification algorithms are also used in CCTV-surveillance of customers in large shopping centres to track their movement and shopping patterns. Machine learning, including deep learning, has been suggested for optimizing the de tection of people by applying learning algorithms to the captured image data. Further more, supervised and unsupervised learning algorithms have been proposed for improv ing the detection of multiple people within an image. Such algorithms can be trained and tested via various available programming platforms provided by software providers using either open-source datasets or custom datasets.

Therefore, there is a need for providing an improved monitoring method that allows objects to be tracked without data controllers loosing focus of the objects of interest.

Object of the Invention

An object of the invention is to overcome the drawbacks of the prior art mentioned above.

Another object of the invention is to provide a method and a system that allows data controllers to better track objects of interest within the image data for statistical pur poses.

Yet an object of the invention is to provide a method and a system that allows image data to be stored for later analysis without violating data protection laws.

Description of the Invention

The invention relates to a method of monitoring an industrial site, comprising the steps:

- capturing a set of raw image data of at least a part of the industrial site using a number of cameras,

- transmitting the raw image data to a data processing apparatus, the data processing apparatus comprising a processor,

- analysing the raw image data in the processor using an object detection algorithm, the object detection algorithm being configured to detect at least one object in the raw im age data, the object detection algorithm outputting a set of coordinates of at least one detected object within the image data and a score indicative of the accuracy of the de tection of the at least one object,

- altering the image data of the at least one detected object in the processor to anonymise the at least one object while leaving the other image data unaltered, thereby forming a set of modified image data, - transmitting the modified image data to an anonymous image database, and

- storing the modified image data in the anonymous image database.

The present method provides an improved method of monitoring an industrial site and the industrial process performed at the site. Further, the present method provides a mon itoring method that allows irrevocably anonymised image data to be stored for later analysis, thus the right to privacy is met and consent from the people in the images are not required. The present method can easy be adapted to monitor different industrial sites and different industrial process without any major modifications.

The term “industrial site” refers to an industrial area in which a dedicated industrial process is performed, such as manufacturing facilities, assembly floors, storage areas, loading areas, enclosed or open yards, or the like. The industrial site also includes har bour areas, container yards, shipyards, temporary assembly sites, gipsy facilities, or the like. Such industrial sites typically have restricted access and require the use of safety equipment and other equipment specific for that type of industry. Any industrial process may be performed at the industrial site, such as production, assembly, loading and un loading, storing, or the like. A number of cameras are positioned relative to the industrial site and capture a set of images of the monitored areas. Each camera may capture an image of the entire indus trial site, or a part thereof. The number of cameras and their individual positions may be determined based on layout and complexity of the industrial site. The industrial site may thus be monitored by a single camera or by a plurality of cameras. The cameras may thus capture images of the monitored area from different angles and/or different distances in order to provide the best coverage. Here, the term “raw image data” should be understood as the original image format in which the images are captured.

The processor of the data processing apparatus may be configured to perform an initial data processing of the raw image data received from each camera. This may be done in an image processing module implemented in the processor. The raw image data may be inputted to the processor, e.g. the image processing module, which may perform an image enhancement of each captured image. Alternatively or additionally, all or some of the individual images may be stitched together or overlapped to form a large image of the monitored areas. The processor may thus output pre-processed image data, e.g. a set of individually enhanced images and/or a large image. This allows for a more accu rate detection in the object detection algorithm. This may potentially reduce the amount of training needed to train the processor, e.g. the neural network.

In one embodiment, the object detection algorithm applies a region of interest relative to the at least one detected object, wherein the set of coordinates indicates the location of the region of interest. A region of interest may be applied over the image, e.g. the raw or pre-proceed image data, in the object detection algorithm. The region of interest may indicate a detected object. The size and shape of the region of interest may be determined by the object detection algorithm. The region of interest may in example be a bounding box. The object detection algorithm generates a set of coordinates indicating the location of a detected object, e.g. the location and height and width of the region of interest.

The object detection algorithm may also generate a score indicating the accuracy of the detection. The score may thus indicate the probability that the region of interest actually comprises a detected object. Alternatively or additionally, the score may be used during validation to sort the objects and thus increase the accuracy of the object detection al gorithm. The coordinates and the score of each detect object may be stored together with or linked to the raw image data, or pre-processed image data, in a database or memory unit for later validation. Alternatively or additionally, the coordinates and/or the score of each detect object may be inputted together with the raw image data, or pre- processed image data, to an anonymization module in the processor for further data processing.

The region of interest and score may be displayed together with the image on a screen to one or more data controllers authorised to evaluate the image data. The data control- lers may then validate that an object actually is present or not, as described later.

In one embodiment, the step of altering the image data of the at least one detected object comprises at least one of:

- blurring the at least one detected object in the image, - redacting the at least one detected object in the image, or

- removing the at least one detected object from the image.

The image data of each detected object may be altered in the anonymization module so that that object cannot be positively identified. If the object is a person, then the image data is altered so the identity of that person cannot be positively confirmed. In example, the image of the detected object may be blurred by applying a blurring function. In example, the detect object, or the region of interest, may be redacted with a solid block, e.g. a black box, or otherwise obscured. The detected object may also be removed from the image by deleting the object and blending the background of a previous image in a time series. Such blurring or replacement techniques are known and will not be de scribed in further details. This allows the present method to automatically respect the right to privacy in accordance with the data protection law. At least the coordinates of each detect object together with the raw image data, or pre- processed image data, may be inputted to the anonymization module. A background dataset may further be inputted to the anonymization module. The anonymization mod ule may output a set of modified image data which can be stored for later statistical analysis.

Optionally, the coordinates of the region of interest and/or the score may further be stored in the anonymous image database. These data may be used in any subsequent data analysis of the anonymous image data. In example, the modified regions in the stored images may affect any subsequent object objection and may thus be excluded from the data analysis.

In one embodiment, the processor further analyses the raw image data to further detect a background image of the same location as the location of the at least one detected object, wherein the image data of the at least one detected object is replaced in the pro- cessor with the corresponding image data of the background image.

The object detection algorithm may further detect a background of the image, where the detected image data of the background may be added to a background dataset. The background dataset may be continuously updated with backgrounds taken from multiple cameras, thereby forming a complete background of the industrial site. In example, the background dataset may comprise a set of previous image frames (n-x) preceding a current image frame (n). The number ‘x’ indicates the number of frames stored. Alter natively, the background may be determined dynamically during the data processing. This background may be used to remove objects from the images and thus allow data controller to better track other objects of interest.

The anonymization module may search the background dataset to identify a back ground, e.g. a previous image frame, of the same location as the detected object. The detected object, or the region of interest, may then be replaced with the background image of the same location, as mentioned above, in the anonymization module. The background may thus appear as unmodified to the data controller.

In one embodiment, the processor further compares the score to at least a first threshold value, and if the score is below the first threshold value then the image data of the at least one detect object is transmitted to further evaluation, and if the score is above the first threshold value then a verified image of the at least one detect objected is detected.

The score for each detected object may be compared in the processor, e.g. in the anon- ymization module, to one or more threshold values. The score may be compared to a minimum threshold value and/or a validation threshold value. If the score is below the minimum threshold value, then this may indicate that the detected object is located far away or in a position so that the identity of that object, e.g. person, cannot be positively identified.

If the score is below the validation threshold value, e.g. between the validation threshold value and the minimum threshold value, then further evaluation of the detected object may be needed, as described later, and the detected object, e.g. the image data of the object or that image frame, is isolated from the rest of the image data. This allows the object detection algorithm to be trained on only validated images of objects, thereby reducing the size of the training dataset. In one embodiment, the step of further evaluation comprises evaluating the image data of the at least one detected object by a data controller, wherein the data controller vali dates whether an object has been detected or not. When isolated, the image data of the detected object, or that image frame, may be tem porary stored in a quarantine or validation area of the database or memory unit. A re quest for validation may then be transmitted from the processor to the data controller.

The data controller may subsequently review the image of the detected object at a com- puter terminal and confirm or disconfirm that an object has been detected. The image data, or image frame, may be transmitted together with the request, or transmitted upon receiving a request from a local processor at the data controller.

The data controller may then validate that an actual object has been detected by gener- ating a ‘true positive’ signal which may be transmitted back to the data processing ap paratus. If the detected object is not an actual object, then a ‘false positive’ signal may be generated and transmitted back to the data processing apparatus. The phrase “not an actual object” should interpreted as an object that differs from what the object detection algorithm is programmed to search for. For example, if the object detection algorithm is searching for persons and the detected object is a dog, then a ‘false positive’ signal is generated.

The data controller may use the generated score, e.g. confidence score, to sort the de tected objects into different classes before retraining the neural network. Alternatively, an application running on the computer terminal may rank the received image data of the objects in a predetermined order, e.g. based on the score. The objects may then be displayed according to the ranked order. This may allow for a more efficient validation process. If the processor in the data processing apparatus receives the ‘false positive’ signal, then the image data, or that image phrase, may be deleted from the quarantine or validation area. Alternatively or additionally, the object detection algorithm may be instructed by the processor to ignore that object in the image data. This reduces the number of false positives. If the ‘true positive’ signal is received, then the image data is transmitted to the anony mization module where the detected object is masked, blurred or removed, as described earlier. Alternatively or additionally, the detected object may be added to the training dataset. The object detection algorithm may thus be trained using only validated objects, thereby reducing the amount of training and increasing the accuracy of the detection.

The validation of the detected objects may be performed on-edge, thereby reducing the amount of data transmission. Alternatively, the validation of the detected objects may be performed remotely from the industrial site, e.g. via a secured data link.

In one embodiment, the object detection algorithm comprises a number of dedicated classes used to detect the at least one object, wherein each dedicated class comprises a dataset indicative of:

- a person or a part of a person, - industry specific features associated with the industrial site,

- a motion of an object, or

- a triangulation of an object.

The object detection algorithm may be provided with one or more classes dedicated to detect one or more types of objects. Each class may further comprise one or more sub classes dedicated to detect specific features of that object. In one embodiment, the object detection algorithm may be dedicated to detect a single type of objects. In another em bodiment, the object detection algorithm may be dedicated to detect at least two types of objects. The object detection algorithm may preferably be configured to detect peo- pie, i.e. humans in general; industrial specific features; motions of an object; triangula tion of an object; or any combinations thereof.

A person dataset may be used by the object detection algorithm to detect people in the images, optionally one or more sub-datasets may be used to detect parts of the body, such as legs, arms, head, etc. Workers moving around such industrial sites typically have specific body poses or working positions when perform certain tasks. The object detection algorithm may thus be trained to detect such unique positions or poses. An industry specific feature dataset may be used by the object detection algorithm to detect these unique features, optionally one or more sub-datasets may be used to detect different features, such as personal protection equipment, working clothes, tools, vehi cles, etc. The personal protection equipment may comprise helmets, vests, goggles, ear- muff, etc. The object detection algorithm may thus be trained to detect such unique features.

The industry specific feature dataset may comprise a vehicle dataset with vehicle spe cific features, such as license plates, unique vehicle numbers, company names, logos, etc. Such features may be anonymised, e.g. removed, on the vehicle in the image by the anonymization module in the same manner as described above. Alternatively or addi tionally, the anonymization module may anonymise, e.g. remove, the entire vehicle from the image. This allows the data processing apparatus to also irrevocably anony mised vehicles or vehicle specific features.

An object motion dataset may be used by the object detection algorithm to detect unique movements of the object, such as moving up or down on ladders, entering and exiting openings in large objects, moving around on large objects, etc. The object detection algorithm may thus be trained to detect such unique motions.

According to one embodiment, the method comprises capturing a first image and at least a second image using two or more cameras and triangulating the position of an object by the object detection algorithm based on at least the first and second images. In a further embodiment, the method comprises altering the image data of said object in at least the first and second images in the processor.

An object triangulation dataset may be used by the object detection algorithm to trian gulate the position of objects based on the set of captured images. The object triangula tion dataset may comprise a 3D anchor for each object which is projected onto the image of two or more cameras. An object which cannot be validly detected in a first image may be detected in a second image, which may then be used to alter the image data of the detected object in at least the first and second images. Preferably, the detected object is removed from both the first and second images. The abovementioned datasets may be combined in a single neural network, or in sepa rate neural networks. The detection of different objects may thus be performed using one or more neural networks. This allows the configuration of the data processing ap paratus may thus be adapted to a particular industrial site. During training, two or more 5 neural networks might be used to validly detect objects. When full trained, a single neural network might be sufficient to validly detect objects. Thus the number of neural networks used may be adapted to the different phases of the monitoring.

Alternatively, the neural network(s) may be combined with one or more machine learn ing algorithms. The machine learning clustering algorithm may, in example but not lim it) ited to, be K-means clustering algorithm or another suitable clustering algorithm. This allows the types of objects, e.g. classes of objects, to be detected using different data processing techniques.

In one embodiment, the method further comprises steps of:

15 - transmitting the raw image data to a raw image database, and storing the raw image data in the raw image database prior to being transmitted to and analysed in the data processing apparatus,

- optionally, deleting the raw image data in the raw image database after a predeter mined time period. 0

The captured image data, or raw image data, may be transmitted directly to the data processing apparatus via the first data link. This allows for a real-time or near real-time data processing of the image data. This is advantageously if the data processing appa ratus is located on-edge, i.e. at the industrial site. 5

Alternatively, the captured image data, or raw image data, may be transmitted to a raw image database via the first data link, e.g. a secured data link. The raw image data may then be stored in the database for later data processing. The data processing apparatus may communicate with the raw image database via a third data link, e.g. a secured data 30 link. This is advantageously during training of the neural network. The raw image data may thus be stored on-edge or in a remote database.

Optionally, the raw image data may be deleted from the database after a predetermined time period. The predetermined time period may indicate a maximum storage time period. Alternatively, the raw image data may be deleted from the database once the data processing is complete, i.e. when the modified or unmodified image data is stored in the anonymous image database. The maximum storage time period may be selected in compliance with the data protection law.

In one embodiment, training process is applied to the object detection algorithm in the data processing apparatus to optimise the detection of the at least one object, wherein the stored raw image data is used to train one or more layers of the object detection algorithm using machine learning.

The present data processing may advantageously be implemented using artificial intel ligence (AI) with the purpose of analysing the captured image data and detecting objects in the images in real-time or near-time. Furthermore, the use of AI also enables the step of anonymising the detected objects to be performed in real-time or near-time. This minimises the need for storing the raw image data before processing the image data in the processor. This also reduces the time needed to train the neural network.

An off-the-shelf neural network may initially be implemented in the processor and a training dataset may be defined. The neural network may already be pre-trained to de- tect the desired objects. If not, the neural network may be pre-trained using a public available dataset, such as COCO or ImageNet, comprising images of the objects in tended for detection. However, such public available dataset typically provides a limited model of objects under different conditions and/or in situations. The raw image data stored in the raw image database may therefore be used as training dataset. The training dataset may comprise raw image data of a time period equal to the abovementioned maximum storage time period. Alternatively, the training dataset may comprise raw image data of another time period. The training dataset may be a copy of the stored raw image data, or link directly to the stored raw image data. This raw image data is thus used to optimise the structure of the neural network, such as network depth, width, resolution and layer weights. An example of the training process will now be described for detecting people. How ever, the training process may equally be applied for other types of object, such the ones mentioned earlier. The object detection algorithm may analyse the training dataset, e.g. the raw image data, to detect any persons. Each detected person, or the frames thereof, may then be isolated in the quarantine or validation area. The data controller may then confirm or disconfirm that a valid person has been detection by generating the ‘true positive’ or ‘false positive’ signal. If the detected person is validated by the data controller, then that person may be removed, e.g. anonymised, from the image in the anonymization module. The mod ified image data may subsequently be stored in the anonymous database. The validated person may further be added to the corresponding person dataset mentioned earlier.

If the detect object is not a valid person, then the object detection algorithm may be instructed to ignore the object during the data analysis. The person, or the frames thereof, may subsequently be deleted from the quarantine or validation area.

The weights of one or more layers of the neural network may afterwards be adjusted and the process may be repeated, e.g. instantly or at time intervals. The time intervals may in example, but not limited to, be daily, weekly, bi-weekly or monthly. The process may be repeated using the same or a new training dataset.

The training process may further comprise a validation check, where samples of the image data, e.g. frames thereof, are extracted from the stored modified image data. The samples may then be transmitted to the data controller for validation. If no persons are present in the images, then the data controller may generate a ‘no person present’ signal. This ‘no person present’ signal may then be transmitted back to the processor and that sample may be deleted. If a person is still present in the images, then the data controller may generate a ‘person present’ signal which may be transmitted back to the processor. A current version of the object detection algorithm, e.g. the neural network, may subsequently be applied to that sample, or an extended sample around that sample. Alternatively, the data controller may manually determine a set of coordinates of that person, e.g. of a region of interest applied to that person. The validated person may then be added to the corresponding person dataset, as mentioned earlier, and an updated version of the object detection al gorithm, e.g. the neural network, may subsequently be applied to that sample, or an extended sample around that sample.

A number of samples may be extracted on a regular basis, e.g. weekly, bi-weekly or monthly. The number of samples may be selected dependent on the amount of data. Each sample may comprise the image data of a short time period, e.g. minutes or hours. After a suitable amount of training, the object detection algorithm, e.g. the neural net work, may enter a validation phase where people are automatically detected and re moved in the processor and the modified image data is stored in the anonymous data base. The abovementioned validation check may be performed on the stored modified image data.

After a suitable amount of validation, the object detection algorithm, e.g. the neural network, may enter a stable operation where people are automatically detected and re moved in the processor and the modified image data is stored in the anonymous data base.

During the stable operation, and optionally the validation phase, the data controller or the owner of the industrial site may report back if people are present in the modified images stored in the anonymous image data. The operator of the present system may then re-activate the training process, or simply add the detected person to person dataset and update the object detection algorithm, e.g. the neural network.

The training process, the validation phase and stable operation may equally be applied for detection of other types of object, such the ones mentioned earlier. The present invention also relates to a system for monitoring an industrial site compris ing:

- a number of cameras positioned relative to an industrial site, each camera being con figured to capture raw image data of at least a part of the industrial site, - a data processing apparatus configured to communicate with the number of cameras via a first data link, the data processing apparatus comprising a processor configured to analysing the raw image data received from the number of cameras,

- an anonymous image database configured to communicate with the data processing apparatus via a second data link, the anonymous image database being further config ured to store modified image data received from the data processing apparatus, wherein the processor is configured to apply an object detection algorithm to the raw image data, the object detection algorithm being configured to detect at least one object in the raw image data, the object detection algorithm is configured to output a set of coordinates of at least one detected object in the image data and a score indicative of the accuracy of the detection of the at least one object, and wherein the processor is further configured to alter the image data of the at least one detected object to anonymise the at least one object while leaving the other image data unaltered, thereby forming a set of modified image data being transmitted to the anonymous image database.

The present system provides an improved monitoring of an industrial site and the in dustrial process performed at the site. Further, the present system advantageously anon ymises the captured image data so that it can be stored for later analysis. The system is configured to comply with the data protection law and thereby respect people’s right to privacy. The system can easy be adapted to different industrial sites and industrial pro cesses, as it does not require any significant modifications.

Any type of suitable cameras may be used to monitor the industrial site, preferably the camera is a surveillance or security camera. The size, resolution, exposure time and other relevant settings in the camera may be determined based on the conditions of the industrial site. The camera may optionally be adapted to capture images within one or more spectral bands of the electromagnetic spectrum, e.g. a multispectral image or a visual spectrum image or an infrared image. Alternatively, one or more dedicated filters may be applied to the captured images to achieve a certain spectrum image. The cameras may capture still images or moving images of the monitored area. In some applications, a single camera can be used. In other applications, a plurality of cameras may be used. Preferably, different types of cameras may be used to monitor the industrial site. In example, the computer terminal at the data controller may be a laptop, a desktop, tablet, a smartphone, or another suitable computer terminal. The data controller may alternatively evaluate the detected objects by accessing a dedicated website via the com puter terminal.

The present data processing steps may preferably implemented in at least one neural network in the processor. Preferably, the neural network may be a convolutional neural network. The neural network may comprise at least one input layer, a number of inter mediate layers and at least one output layer. The intermediate layers may comprise one or more convolution layers and one or more pooling layers arranged in alternating order. The convolutional neural network may in example, but not limited to, be a Region based convolutional neural network (R-CNN), a fast or faster R-CNN, YOLO or any another state-of-the-art neural network architecture. The present neural network is not limited to a traditional regional based convolutional network (R-CNN) as R-CNN networks over the recent years have been significantly surpassed both in efficiency and accuracy by one stage detectors, such as YOLOv3 and latest FastDet, which uses a bi-directional feature pyramid network architecture (BiFPN) on a fastnet backbone where all the di mensions (width, depth and resolution) of the neural network are optimized through compound scaling. Thus, no custom-made neural network is needed to implement the present data processing as many suitable neural network designs are available.

The processor may be an Al-accelerator, such as a central processing unit, a graphics processing unit, a field-programmable gate array or a dedicated ASIC (application spe cific integrated circuit) unit. Optionally, the processor comprises more than one AI ac- celerator. The Al-accelerator may be programmed using a standard programming tool, typically provided by the manufacture of the Al-accelerator.

In one embodiment, the data processing apparatus is arranged at the industrial site, wherein the analysis of the raw image data in the processor is performed on-edge.

The data processing apparatus may be a computer unit, e.g. a local server, arranged at the industrial site and connected directly to the cameras via a data link. Thus, transmis sion of the raw image data is done locally on site, i.e. on-edge. The data link, e.g. the first data link, may be an unencrypted data link. This configuration may be preferred, once the present object detection algorithm, or neural network, is in the validation phase or in the stable operation.

In one embodiment, the data processing apparatus is arranged at a centralised site, wherein the analysis of the raw image data in the processor is performed remotely from the industrial site.

The data processing apparatus may be a centralised computer unit, e.g. a remote server, which is arranged remotely from the industrial site. In example, the centralised com- puter unit may be a cloud based computer system. The data processing apparatus may be connected to the cameras via a data link. The data link, e.g. the first data link, may be encrypted to establish a secured data link. Any suitable encryption technique may be used. This allows images comprising personal information to be transmitted between different system components. In example, the data link may be a VPN-connection, but other data links may be used.

In this configuration, the raw image data may be transmitted outside the industrial site for data processing. This configuration may be preferred, when the present object de tection algorithm, or neural network, is in the training phase. In on embodiment, the anonymous image database and, optionally a raw image data base, is arranged at the industrial site or at a centralised site.

The anonymous image database may be a remote database, e.g. a database server, ar ranged outside the industrial site. In example, the remote database may be a cloud based database. The anonymous image database may communicate with the data processing apparatus via a data link. The data link, e.g. the second data link, may be a wired or wireless connection. The modified image data may be accessed by a customer or data controller via a dedicated website using dedicated unique logins. Similarly, the raw image data may be a remote database, e.g. a database server, arranged outside the industrial site. In example, the remote database may be a cloud based data base. The raw image database may communicate with the data processing apparatus via another data link. The data link, e.g. the first data link, may be a wired or wireless connection. The raw image data may be accessed by a data controller via a dedicated website using a dedicated unique login.

The computer terminal of the data controller may be a laptop, a tablet, a smartphone, a deck computer, or another suitable computer terminal. A dedicated application may be configured to run on that computer terminal, wherein the data controller may confirm or disconfirm the validity of a detected object. The dedicated application may be con figured to communicate with the processor of the data processing apparatus via another data link, e.g. a wired or wireless connection.

The application may be designed so that the data controller may generate the ‘true pos itive’ or ‘false positive’ signal, or the ‘no person present’ or ‘person present’ signal, by simply interacting with the user interface of the computer terminal.

The data controller may use the application to send a report back to the data processing apparatus indicating that objects are still present in the modified images. A customer may use another computer terminal to send a similar report back to the data processing apparatus or to the data controller. The data processing apparatus may then, upon re ceiving the report or an activation signal from the data controller, re-activate the training process to retrain the neural network to detect that object.

Alternatively, the processor in the data processing apparatus may be configured to au tomatically perform the validation process, where detected objects having a score above an upper threshold value are automatically added to the training dataset. The processor may further be configured to automatically retrain the neural network, e.g. at predeter- mined intervals or each time objects are added to the training dataset. This allows the training and retraining of the neural network to be performed automatically in the pro cessor, thus reducing the amount of man hours needed to train and retrain the neural network. The present invention further relates to a computer program comprising instructions which, when loaded and run on the system described earlier, causes the data processing apparatus to perform the corresponding steps described earlier. A computer program dedicated to perform the steps of the abovementioned method is implemented in the processor. The computer program may be an AI application, pref erably a neural network as described earlier, that is configured to run on a particular processor, e.g. a hardware accelerator. Preferably, the hardware accelerator is an AI accelerator. The neural network may be a software or hardware based neural network.

The present invention additionally relates to a computer-readable medium having stored thereon the computer program described earlier. The data processing apparatus further comprises a suitable computer-readable medium, e.g. a memory unit, on which the present computer program can be stored. The com puter-readable medium may be adapted to interact with a particular processor, e.g. an AI accelerator. The described embodiments can be combined in any combinations without going be yond the scope of the invention.

Description of the Drawing

The invention is described by example only and with reference to the drawings, wherein:

Fig. 1 shows a first embodiment of the method according to the invention; Fig. 2 shows a second embodiment of the method according to the invention; Fig. 3a-c show three images captured of an example of the industrial site; Fig. 4a-b show two examples of images of another example of the industrial site; Fig. 5 shows an example of a training phase of the neural network; Fig. 6 shows an example of a validation phase of the neural network; Fig. 7 shows an example of a stable operation of the neural network; and Fig. 8a-f show six examples of an industrial site according to the invention. In the following text, the figures will be described one by one and the different parts and positions seen in the figures will be numbered with the same numbers in the differ ent figures. Not all parts and positions indicated in a specific figure will necessarily be discussed together with that figure. Reference list

1 Industrial site

2 Cameras

3 Raw image database 4 First data link

4a Transmission of raw image data

5 Inputting raw image data to data processing apparatus

6 Data processing apparatus

7 Processor 7a-d Neural networks

8 Object

8a People

9 Outputting location and score of each detected object

10 Di sregarding detected obj ects 11 Validated objects

1 la-d Validated objects of each object detection algorithm

12 Modified image data

13 Transmission of modified image data

14 Anonymous image database 15 Capturing raw image data

16 Analysing the raw image data

17 Isolating detected objects

18 Adding valid obj ects to training dataset

19 Disregarding invalid objects 20 Removing objects from image data

21 Retraining the neural network

22 Evaluating of the image data

23 Releasing the modified image data

24 Isolating the image data for further processing 25 Isolating the image data for further processing

26 Reporting objects in modified image data Detailed Description of the Invention

Fig. 1 shows a first embodiment of the method of monitoring an industrial site 1 ac cording to the invention. A number of cameras 2 are positioned relative to the industrial site 1 to capture a set of raw image data of the industrial site 1. Each camera 2 monitors at least a part of the industrial site 1 , wherein each camera 2 is connected to a raw image database 3 via a first data link 4. Here, the captured image data of each camera 2 is transmitted (indicated by arrows 4a) to the raw image database 3. The raw image data is then stored in the raw image database 3. The raw image data is optionally encrypted before transmission.

The stored raw image data is inputted 5 to a data processing apparatus 6 comprising a processor 7 configured to analyse the captured image data. Here, the data processing apparatus 6 is a centralised computer unit. Here, the processor 7 is a neural network running on a hardware accelerator or a virtual cloud based processing instance.

An object detection algorithm is implemented in the processor 7 and configured to de tect at least one type of object 8. Here, the object detection algorithm is configured to detect people. The object detection algorithm is configured to output (indicated by ar row 9) a set of coordinates of a detect object in the image and a score indicating the accuracy of the detection.

One or more of the detected objects 8 is isolated for further evaluation by a data con troller, if the score is below a predetermined threshold value. Any objects 8 not vali dated by the data controller are disregarded (indicated by arrow 10) in the further data processing. Validated objects (indicated by arrow 11) are stored in a corresponding da taset which is used for retraining the neural network. If the score is above the predeter mined threshold value, then the object is optionally automatically regarded as a vali dated object. The processor 7 alters the image data of each detected object 8 so that the object cannot be positively identified in the image. Here, the detected people are irrevocably removed from the image data, thereby creating a set of modified image data 12. The modified image data 12 is transmitted (indicated by arrow 13) to an anonymous image database 14. The modified image data 12 is then stored for later statistical anal ysis. Fig. 2 shows a second embodiment of the method according to the invention, wherein the processor 7’ differs by comprising a multiple of object detection algorithms (indi cated by the four individual columns in the data processing apparatus 6’). In the first embodiment of Fig. 1, the processor 7 comprises a single object detection algorithm. Here, the processor 7’ of the data processing apparatus 6’ comprises four separate neural networks 7a, 7b, 7c, 7d each dedicated to detect a particular type of object. But the processor 7’ may comprise fewer or more neural networks. Each neural network 7a, 7b, 7c, 7d acts in parallel and each object detection algorithm outputs a number of detected objects 11a, 1 lb, 11c, lid. The detected objects 11a, 1 lb, 11c, lid may be validated in the manner as mentioned under Fig. 1.

Instead of using separate neural networks for each object type detection, the object type detection may be performed in a single neural network comprising a multiple of object detection algorithms.

Fig. 3a-c show three images captured of an example of the industrial site 1, where Fig. 3a shows a raw image, Fig. 3b shows an image outputted from the object detection algorithm and Fig. 3c shows the modified image after alteration of the image data. As illustrated in Fig. 3b, the object detection algorithm has detected six objects and added a region of interest around each object. The image data of each detect object is altered by the processor 7 by replacing the image of the object with a background image of the same location. The background is determined in the object detection algorithm and added continuously to a background dataset. The background dataset is then used in the processor 7 to re move objects from the image. Alternatively, the background may be determined dy namically during the data processing. Fig. 4a-b show two examples of images of another example of the industrial site 1, where Fig. 4a shows a raw image of a production line of the industrial site 1. As illus trated in Fig. 4a, a number of people 8a are moving around relative to the production line. The images are transmitted to the data processing apparatus 6 for processing.

Fig. 4b shows a modified image of the same production line after being processed in the processor 7. Here, the object detection algorithm is trained to detect person objects, or people 8a, using a dedicated person dataset. The detected people 8a are then removed using the anonymization module in the processor 7. The modified images are afterwards transmitted to and stored in the anonymous image database 14.

The modified image data, or images, may then be released by the data controller to the customer, e.g. the industrial site owner or operator. The modified image data may later be processed in the processor 7, or in another processor, to extract statistical data relat- ing to the flow of components, quantity, time and/or other relevant data.

Fig. 5 shows an example of a training phase of the neural network 7a, 7b, 7c, 7d of the processor 7’, wherein the raw image data is used to train the neural network. The cameras 2 are used to capture 15 raw images of the monitored industrial site 1 which is then analysed 16 in the data processing apparatus 6 using the object detection algorithm. If an object 8 is detected, then the detected object 8 is isolated 17 in a data base for further evaluation by the data controller. If the data controller confirms that a valid object has been detected, then the image data of the object is added 18 to the training dataset. If no valid object is detected, then the image data of the object is deleted from the database and the object is disregarded 19.

The detected valid objects are subsequently further processed in the processor 7 by re moving 20 the object 8 from the image data, e.g. by replacement or blurring. The mod- ified image data 12 is then stored in the anonymous image data 14. One or more of the neural networks 7a, 7b, 7c, 7d is then finally retrained 21 by adjusting the weights of one or more layers of that neural network 7a, 7b, 7c, 7d. If no objects 8 are detected by the object detection algorithm, then the data controller evaluates 22 the image data to confirm if all objects have been detected or not. If yes, then the modified image data is released 23. If not, then the image data or frame is stored 24 for further processing. The further processing 24 may be performed by the data con- troller manually identifies the object 8 and add the image data to the training dataset.

Fig. 6 shows an example of a validation phase of the neural network, where the training of the neural network is validated. The raw image data is analysed 15 in the processor 7. Detected objects 8 are removed from the image data and the modified image data 12 is stored in the anonymous database 14.

A number (marked by N) of samples are taken from the modified image data 12 and evaluated 22’ by the data controller. If all objects 8 are removed, then the modified image data is released 23.

If not all objects 8 have been removed, then image data or frame is stored 25 for further processing. The further processing 25 may be performed by the data controller manually identifies the object 8 and add the image data to the training dataset. The data controller or customer may report 26 if objects 8 are still present in the modi fied image data. When reported, the modified image data may be further processed 25 by the data controller manually identifying the object 8 and adding the image data to the training dataset. Fig. 7 shows an example of a stable operation of the neural network, where neural net work is trained to an acceptable level. The raw image data is analysed 15 in the proces sor 7. Detected objects 8 are removed from the image data and the modified image data 12 is stored in the anonymous database 14. Fig. 8a-f shows six examples of an industrial site 1 according to the invention. Fig. 8a shows an example of a full assembly line for assembling large ship components at Hyundai, such as ship motors. Here the delivery of components and movement between sub-stations needs to be monitored in order to ensure an optimal assembly. Fig. 8b shows an example of loading large wind turbine components onto an installation vessel at Bladt Industries. Here the order at which the large wind turbine components arrive and the arrangement of these need to be monitored in order to ensure an optimal loading process.

Figs 8c show an example of an assembly line for assembling large wind turbine com ponents at Enercon, such as generators. Fig. 8d shows an example of a manufacturing facility for manufacturing wind turbine blades at LM Wind Power. Here the individual tasks performed at each sub-station need to be monitored to ensure an optimal manu- facture.

Fig. 8e shows an example of an assembly line for assembling large wind turbine com ponents at Nordex, such as rotor hubs. Here the delivery of components and movement between sub-stations needs to be monitored in order to ensure an optimal assembly. The monitoring of the assembly line can be done using a plurality of cameras.

Fig. 8f shows an example of an offshore installation of a wind turbine using an instal lation vessel. Here the movement of individual wind turbine components needs to be monitored in order to ensure an optimal installation. The monitoring can be done using a single camera.

The present system and method can easily be adapted to monitor these industrial sites without any major modifications. The placement of each camera 2 is simply adapted to that industrial site, where each camera 2 is connected to the data processing apparatus 6. The neural networks 7a, 7b, 7c, 7d are subsequently trained to detect all persons, or other objects, using the raw image data from the cameras 2.

Claims

1. A method of monitoring an industrial site, comprising the steps:

- analysing the raw image data in the processor using an object detection algorithm, the object detection algorithm being configured to detect at least one object in the raw im- age data, the object detection algorithm outputting a set of coordinates of at least one detected object within the image data and a score indicative of the accuracy of the de tection of the at least one object,

- altering the image data of the at least one detected object in the processor to anonymise the at least one object while leaving the other image data unaltered, thereby forming a set of modified image data,

- transmitting the modified image data to an anonymous image database, and

- storing the modified image data in the anonymous image database.

2. The method according to claim 1, characterised in that the object detection algorithm applies a region of interest relative to the at least one detected object, wherein the set of coordinates indicates the location of the region of interest.

3. The method according to claim 1 or 2, characterised in that the step of altering the image data of the at least one detected object comprises at least one of: - blurring the at least one detected object in the image,

- masking the at least one detected object in the image, or

- removing the at least one detected object from the image.

4. The method according to claim 3, characterised in that the processor further analyses the raw image data to further detect a background image of the same location as the location of the at least one detected object, wherein the image data of the at least one detected object is replaced in the processor with the corresponding image data of the background image.

5. The method according to any one of claims 1 to 4, characterised in that the processor further compares the score to at least a first threshold value, and if the score is below the first threshold value then the image data of the at least one detect object is transmit ted to further evaluation, and if the score is above the first threshold value then a verified image of the at least one detect objected is detected.

6. The method according to claim 5, characterised in that the step of further evaluation comprises evaluating the image data of the at least one detected object by a data con troller, wherein the data controller validates whether an object has actually been de tected or not.

7. The method according to any one of claims 1 to 6, characterised in that the object detection algorithm comprises a number of dedicated classes used to detect the at least one object, wherein each dedicated class comprises a dataset indicative of:

- a person or a part of a person,

- industrial specific features associated with the industrial site,

- a motion of an object, or

- a triangulation of an object.

8. The method according to claim 7, characterised in that the method comprises captur ing a first image and at least a second image using two or more cameras, and triangu lating the position of an object by the object detection algorithm based on at least the first and second images.

9. The method according to claim 8, characterised in that the method comprises altering the image data of said object in at least the first and second images in the processor.

10. The method according to any one of claims 1 to 9, characterised in that the method further comprises steps of:

- transmitting the raw image data to a raw image database, and storing the raw image data in the raw image database prior to being transmitted to and analysed in the data processing apparatus,

- optionally, deleting the raw image data in the raw image database after a predeter mined time period.

11. The method according to any one of claims 7 to 10, characterised in that a training process is applied to the object detection algorithm in the data processing apparatus to optimise the detection of the at least one object, wherein the stored raw image data is used to train one or more layers of the object detection algorithm using machine leam- ing.

12. A system for monitoring an industrial site comprising:

- an anonymous image database configured to communicate with the data processing apparatus via a second data link, the anonymous image database being further config- ured to store modified image data received from the data processing apparatus, wherein the processor is configured to apply an object detection algorithm to the raw image data, the object detection algorithm being configured to detect at least one object in the raw image data, the object detection algorithm is configured to output a set of coordinates of at least one detected object in the image data and a score indicative of the accuracy of the detection of the at least one object, and wherein the processor is further configured to alter the image data of the at least one detected object to anonymise the at least one object while leaving the other image data unaltered, thereby forming a set of modified image data being transmitted to the anonymous image database.

13. The system according to claim 12, characterised in that the data processing appa ratus is arranged at the industrial site, wherein the analysis of the raw image data in the processor is performed on-edge.

14. The system according to claim 12, characterised in that the data processing appa- ratus is arranged at a centralised site, wherein the analysis of the raw image data in the processor is performed remotely from the industrial site.

15. The system according to claim 13 or 14, characterised in that the anonymous image database and, optionally a raw image database, is arranged at the industrial site or at a centralised site.

16. A computer program comprising instructions which, when loaded and run on the system of any one of claims 12 to 15, causes the data processing apparatus to perform the corresponding steps of any one of claims 1 to 11.

17. A computer-readable medium having stored thereon the computer program of claim 16.