NL2023653B1 - Method and system for detecting and gripping a corner of a flatwork item - Google Patents

Abstract

The invention relates to a method and a system for detecting and gripping a corner of a flatwork item, wherein images of flatwork items having corners are used to 5 determine the position and/or orientation of at least one corner of a provided flatwork item which can subsequently be gripped by a first gripper.

Description

Method and system for detecting and gripping a corner of a flatwork item The invention relates to a method for detecting and gripping a corner of a flatwork item. The invention also relates to a system for detecting and gripping a corner of a flatwork item. The invention also relates to the use of such system.

Industrial laundry services are specialized in providing a full service laundry solution for, for example, hotels, restaurant, gyms, spas, schools, care and nursing homes or any further. This means that the dirty washing of such party is provided to the industrial laundry service and will be returned once clean. Therefore, batches of used or soiled laundry need to be sorted, washed, dried, optionally ironed or pressed and subsequently folded such that they can be returned to their place of use. The washing and drying process is almost fully automated by making use of a sequence of washing machines, dryers and/or presses and automated transport systems. Also folding of laundry can be automated by making use of a folding device. However, for some steps the laundry item needs to be provided to the following processing unit in a rather specific arrangement. For example the sorting of a bunch of laundry is done manually. Another example is feeding of laundry to a feed conveyor or a mangle, or the insertion of textile in a folding machine. For the latter, it is of importance that for example a sheet (or towel) is positioned such that a first side of the typically rectangular or square sheet is perpendicular to a transport direction and that two corners of the sheet are parallel to each other when inserted in two vertically positioned rollers or on a conveyor. This specific alignment is a typical requirement of conventional folding devices and further machinery used in this technical field. Since this specific orientation of sheets it is difficult, or even unachievable, to automate this step. Hence, said steps are nowadays still done manually. The overall process still having steps to be done manually is undesired for several reasons, this is for example both labor intensive and costly. Further, it is physically demanding work and also relatively time consuming, and thus a limiting factor in the overall process. Hence, there is a need for further automating of the entire system. A difficulty for further automating abovementioned laundry process is that it is difficult to develop a device which can detect and grab a corner of a piece of laundry, such as a flatwork item, in a controlled and repeatable manner.

Distinguishing corners and edges appears to be difficult for automated system and this also applies to subsequent gripping of the detected edge or corner.

Hence, it is a goal of the invention to provide a method and/or a system which can contribute to further automating of an industrial laundry process.

The invention provides hereto a method for detecting and gripping a predetermined part of a flatwork item, such as a first corner of a flatwork item, comprising the steps of: a) providing a neural network trained to recognize at least one corner of a flatwork item in an image and to indicate the position and/or orientation of said at least one corner, b) providing at least one flatwork item, c) capturing at least one first image of at least part of said flatwork item, d) providing the first image to the neural network, e) receiving a possible position and/or orientation of at least one corner of the flatwork item (provided at step b) ) from the neural network; f) displacing a first gripper towards the determined corner of the flatwork item, and g) gripping of the corner by said first gripper.

Due to the provision of a neural network trained to recognize at least one corner of a flatwork item in an image and to indicate the position and/or orientation of said at least one corner which is configured to provide information of at least one first image made during step c) of the method according to the present invention, it is possible to determine the position, or location, and/or orientation of at least one corner of the flatwork item. The use of the captured first image within the neural network, results in receiving a possible position and/or orientation of at least one corner of the flatwork item (provided at step b) ) from the neural network. Basically, the method according to the present invention makes use of image recognition and/or classification in a practical manner. The neural network is hereby trained to determine the orientation and/or position of at least one corner of the flatwork item which is to be determined. Once information regarding the position and/or orientation of the corner is provided by the neural network, the first gripper is displaced and actuated to grip the corner of the flatwork item. A benefit of this method is that it enables determination of the position and/or orientation of a corner of a flatwork item without having to arrange the flatwork item upfront. Due to the use of the trained neural network, it is of minor importance how the flatwork item is arranged. Herewith, an additional arrangement step can be omitted. The method even enables detection of a corner of a flatwork item from a pile of multiple flatwork items. The specific gripping of the detected corner by the first gripper can ensure that only one flatwork item is gripped. Further, the method enables that it can be ensured that a corner is detected and not just an edge or an folded part of the flatwork item. The method according to the invention provides a reliable and repeatable way to detect and grip a corner of a flatwork item. Said corner is in particular a first corner, which means the first corner to be gripped. The method benefits over the prior art in several respects. Methods or systems according to the state of the art typically randomly grip a flatwork item which is then hung out, such that the lowest hanging corner can be detected and gripped. This is relatively cumbersome and cannot ensure reliable and repeatable gripping of the corner. Further, systems which make use of determination of surface profiles cannot ensure exact and replicable gripping of the corner itself. The method according to the present invention does not require that the flatwork items are marked or provided with identification means itself.

The steps of the method according to the present invention are typically successive steps. Capturing of at least one image can for example be done by visual imaging means or a visual imaging device. This can be any conventional visual imaging means, non-limiting examples thereof are a 2D camera and a 3D camera.

When it is referred to a flatwork item, this can be type of conventional flatwork item. A flatwork item is typically a flat workpiece. Non-limiting examples of flatwork items according to the present invention are towels, cloths, (bed) sheets, napkins, bed linen, duvets, tabletops and/or pillowcases. Such flatwork item is typically substantially rectangular or square. Typically, the flatwork item has at least one corner, preferably at least two corners, and more preferably the flatwork item has four corners. The flatwork item is typically at least partially made of a textile or fabric, which can be either woven or nonwoven. However, it is also conceivable that the flatwork item is a garment. Hence, further non-limiting examples of the flatwork item are a top, shirt, blouse, sweater, trousers and/or a dress.

With respect to the present invention, the term corner is just exemplary, this can also be any predetermined part or area of the flatwork item.

Non-limiting examples thereof are an edge, corner region, shoulder part, collar, sleeve, button and/or zipper.

Hence, the invention also relates to a method for detecting and gripping a predetermined part of a flatwork item, wherein the neural network is trained to recognize a predetermined part of a flatwork item in an image and to indicate the position and/or orientation of said predetermined part and wherein a possible position and/or orientation is received of said predetermined part of the flatwork item from the neural network, such that a first gripper can be displaced towards the determined corner of the flatwork item.

In a preferred embodiment it is conceivable that the method further comprises the steps of taking multiple images of at least one flatwork item, wherein each image shows at least one flatwork item, (manually) annotating each image with information about a position and/or orientation of at least one corner and preferably all corners of said flatwork item, and including the annotated images of flatwork items in a neural network.

Typically these step are performed prior to the neural network being provided in the method.

Hence, these steps are done prior to step a) of the method according to the present invention.

Since the annotating of the images, which may be done manually, only needs to be done once, this will not significantly limit the overall process in practice.

It is also conceivable that the annotating of the first image is computer-controlled.

It is conceivable that no corners are observed during annotating of the images.

This is also useful since it is conceivable that the physical flatwork item which is to be determined is in a position that no corners are visually observable.

It is also possible that the method comprises the steps of determining if the gripping of the determined corner by the first gripper was successful, annotating at least one first image made at step c) with information of the position and/or orientation of at least one corner, and adding of the annotated image to the neural network.

Annotating of the captured first image with information about the position and/or orientation and adding this annotated image to the neural network can contribute to the ability of the method to faster and/or better recognize a corner in a newly captured first image.

Including the annotated first image in the neural network may further enhance the reliability of the method.

The annotating of the first image may for example be done manually. However, it is also conceivable that the annotating of the first image is computer-controlled. Basically, a continuous method for determining a corner of a flatwork item can be obtained. The method may even qualify as a continuous deep learning system. Gripping of a (first) corner can for 5 example be classified as successful if the corner is gripped by the first gripper such that the first gripper can take along the flatwork item to a further processing step. If the corner was not sufficiently gripped the first gripper may lose the corner, and thus the flatwork item. If this is detected, for example by a further visual imaging device, the gripping cannot be classified as successful. It is conceivable that the image will not be annotated and/or added to the neural network if the gripping was not successful. Hence, the annotation and addition of an image is possibly only done when the gripping is classified as successful. The determined corner may be at least partly lifted before it is gripped, preferably via providing a flow of air. Possibly, the determined corner is gripped by providing a flow of air over the corner which is to be gripped. The first gripper may for example be a coanda gripper which makes use of the lifting force of air in order to (temporarily) lift the flatwork item, such that the item can be gripped. It is for example conceivable that the first gripper makes use of compressed air.

A conventional gripper making use of vacuum, or the suction of air, is not usable for gripping a corner of a flatwork item. Such gripper cannot ensure that only the top layer of the flatwork item is gripped, since multiple parts of the flatwork item may engage and/or stick together due to the suction.

Preferably, the determined corner is gripped by enclosing at least part of said corner of the flatwork item by the first gripper, in particular by a gripping mouth of said first gripper. Hence, the first gripper, and in particular the gripping mouth of the first gripper, can provide double-sided engagement of at least part of the corner.

Enclosing of at least part of the corner can provide a reliable and firm engagement of the corner, and thus the flatwork item, by the first gripper. Due to the determination of the position and/or orientation of the corner via the method according to the present invention it is also possible to grip a corner of a randomly arranged flatwork item, or even from a single flatwork item in a pile of flatwork items.

it is conceivable that the flatwork item is provided in a laying position at step b). This means for example that the flatwork item is provided on a substantially horizontal surface. The flatwork item can have a free orientation. It is also conceivable that the flatwork item is provided in a substantially horizontal orientation at step b). It is of importance that the flatwork item is in a non-fixed position. As described above, the method enables that pre-arrangement of the flatwork item can be omitted. In a possible embodiment, the method further comprises the step of displacing the flatwork item via the gripped corner in a substantially horizontal orientation. This is in particular done after step g). It is for example conceivable that the horizontal orientation is enabled by means of guiding the flatwork item through a guiding device. The flatwork item may be displaced through at least two adjacent guiding elements, such as guiding rollers. Therefore such guiding rollers can be positioned above another, such that the flatwork item can pass through in a substantially horizontal orientation. A benefit of displacing the flatwork item in a substantially horizontal position is that the flatwork item, and the movement thereof, can be more controlled compared to displacing the flatwork item in a vertical, or hanging, orientation.

A further possible embodiment of the method according to the present invention comprises the steps of receiving a score for each corner of at least one flatwork item {which is captured during step c) ) from the neural network, selecting a corner to be gripped which has a score which is higher than a predetermined threshold score, and/or discarding the flatwork item if the score is lower than the threshold score. The threshold score can for example be determined based upon the extend of resemblance between the captured first image and the images of the neural network. It is for example possible that a threshold score of at least 50% resemblance is set. However, this can also be another, preferably higher, threshold score. If a corner has a score below the threshold score, the corner can be discarded. If in the captured first image, no corners having a score above the threshold score are detected, than the flatwork item as such can be discarded. It is however also conceivable that the flatwork item is moved a bit, and that subsequently a new image is captured and the corner determination is started again. Moving of the flatwork item can for example be displacing and/or shaking of the flatwork item. it is possible that the first gripper is controlled to grip the nearest corner when multiple corners are determined, or detected. In particular, the first gripper can be controller to grip the nearest corner of the flatwork item with a score above the predetermined threshold score when multiple corners are detected. Gripping the nearest corner can enhance the efficiency of the overall process.

The invention also relates to a system for detecting and gripping a first corner of a flatwork item, preferably via a method as described above, the system comprising at least one visual imaging device configured for capturing at least one first image of at least part of a flatwork item, a neural network trained to recognize at least one corner of a flatwork item in an image and to indicate the position and/or orientation of said at least one corner, and a displaceable first gripper configured for gripping a corner of a flatwork item. The system according to the present invention can experience the same benefits and embodiments as described above for the corresponding method according to the present invention.

The first gripper preferably comprises a gripping mouth configured to enclose at least part of a corner of the flatwork item. The gripping mouth can be configured to clasp the corner of the flatwork item at a predetermined distance from at least one edge of the corner and preferably at a predetermined distance of both edges of the corner. Hence, a repeatable and reliable gripping of the corner of the flatwork item can be obtained. Preferably the gripping mouth is configured such that it does not damage the flatwork item during gripping and/or displacing. The first gripper may also be configured for providing a flow of air. Hence, the determined corner can be gripped by providing a flow of air over the corner which is to be gripped. The first gripper may for example be a coanda gripper which makes use of the lifting force of air in order to (temporarily) lift the flatwork item, such that the item can be gripped. it is for example conceivable that the first gripper makes use of compressed air.

In a preferred embodiment, the system further comprises at least one drive unit for displacing the first gripper towards the location determined by the neural network.

The first gripper, and in particular the gripping mouth, is preferably rotatable over at least 180 degrees, preferably 360 degrees, over an axis of rotation. This will result in the gripper having a relatively large degree of freedom. It also provides more flexibility in which corners can be reached and gripped by the first gripper.

The visual imaging device of the system may for example be a camera. The visual imaging device can be any conventional visual imaging means, non-limiting examples thereof are a 2D camera and a 3D camera.

The invention also relates to use of a system according to the present invention. When the word ‘gripping’ is used, also the terms grasping, picking, holding, grabbing and/or clasping can be meant. When it is referred to a corner of a flatwork item, the point where two edges of the flatwork item meet is meant. It also possible that the corner is a shoulder part of for example a shirt or (bath)robe. The corner is typically gripped by double sided engagement of the textile. The position and/or orientation of the first image can be determined in both a 2D and/or 3D configuration.

Claims (15)

Conclusions A method for detecting and engaging a corner of a flat object, comprising the steps of: a) providing a neural network that is trained to recognize at least one corner of a flat object in an image and to determine the position and / or indicate orientation of said at least one angle, b) providing at least one flatware object, C) capturing at least one first image of at least a portion of the flatware object, d) providing the first image to the neural network, e) receiving from the neural network a possible position and / or orientation of at least one corner of the flat material object, f) moving a first gripper to the determined corner of the flat material object 9) engaging the first corner by the first gripper. The method of claim 1, further comprising the steps of: h) taking multiple images of at least one flatware object, each image showing at least one flatware object, i) annotating each image with position information and / or orientation of at least one corner and preferably all corners of said flat object, and recording the annotated images in a neural network. The method of claim 1 or 2, further comprising the steps of: K) determining whether the first gripper engaging the determined angle was successful, f) annotating at least one image taken during step c) with information about the position and / or orientation of at least one corner, and m) processing the annotated image into the neural network. A method according to any one of the preceding claims, wherein the defined angle is at least partially lifted before being engaged, preferably by providing an air flow. A method according to any one of the preceding claims, wherein the determined angle is engaged by enclosing at least a part of the aforementioned corner of the flat goods object by the first gripper, in particular by a gripping mouth of the first gripper. A method according to any one of the preceding claims, wherein the flatware object is provided in a lying position during step b). A method according to any one of the preceding claims, further comprising the step of: n) moving the flatware object via the engaged angle in a substantially horizontal orientation. A method according to any preceding claim, further comprising the steps of: 0) receiving from the neural network a score for each corner of at least one flat object. p) selecting an angle to be engaged with a score that is higher than a predetermined threshold score, and / or q) rejecting the flat-good object if the score is lower than the threshold score. A method according to any preceding claim, wherein the first gripper is regulated to engage the nearest corner if multiple angles are detected. System for detecting and engaging a first corner of a flat object, preferably via a method according to any one of claims 1-9, comprising: - at least one visual imaging device adapted to capture at least one first image of at least part of a flat object, a neural network trained to recognize at least one corner of a flat object in an image and to indicate the position and / or orientation of said at least one corner, a movable first gripper adapted for gripping a corner of a flat object. System according to claim 10, wherein the first gripper comprises a gripping mouth adapted to enclose at least a part of a corner of the flat goods object. System according to claim 10 or 11, further comprising a drive unit for moving the first gripper towards the location determined by the neural network. System according to any of claims 10-12, wherein the first gripper, and in particular the gripper mouth, is rotatable through at least 180 degrees, preferably 360 degrees, about an axis of rotation. The system of any of claims 10-13, wherein the visual imaging device comprises a camera. Use of a system according to any of claims 10-14.