NL2023652B1 - Method and system for detecting and gripping a second corner of a flatwork item - Google Patents

Abstract

The invention relates to a method, system and gripper for detecting and gripping a second corner of a flatwork item. This is achieved by using images of at least one 5 flatwork item in combination with a neural network. The determined corner is subsequently gripped by a gripper.

Description

Method and system for detecting and gripping a second corner of a flatwork item The invention relates to a method for detecting and gripping a second corner of a flatwork item. The invention also relates to a system for detecting and gripping of a second corner of a flatwork item. The invention also relates to the use of such system. The invention further relates to a gripper for gripping an edge or a corner of a flatwork item.

Industrial laundry services are specialized in providing a full service laundry solution for, for example, hotels, restaurant, gyms, spas, schools, care and nursing homes or any further. This means that the dirty washing of such party is provided to the industrial laundry service and will be returned once clean. Therefore, batches of used or soiled laundry need to be sorted, washed, dried, optionally ironed or pressed and subsequently folded such that they can be returned to their place of use. The washing and drying process is almost fully automated by making use of a sequence of washing machines, dryers and/or presses and automated transport systems. Also folding of laundry can be automated by making use of a folding device. However, for some steps the laundry item needs to be provided to the following processing unit in a rather specific arrangement. For example the sorting of a bunch of laundry is done manually. Another example is feeding of laundry to a feed conveyor or a mangle, or the insertion of textile in a folding machine. For the latter, it is of importance that for example a sheet (or towel) is positioned such that a first side of the typically rectangular or square sheet is perpendicular to a transport direction and that two corners of the sheet are parallel to each other when inserted in two vertically positioned rollers or on a conveyor. This specific alignment is a typical requirement of conventional folding devices and further machinery used in this technical field. Since this specific orientation of sheets it is difficult, or even unachievable, to automate this step. Hence, said steps are nowadays still done manually. The overall process still having steps to be done manually is undesired for several reasons, this is for example both labor intensive and costly. Further, it is physically demanding work and also relatively time consuming, and thus a limiting factor in the overall process. Hence, there is a need for further automating of the entire system.

A difficulty for further automating abovementioned laundry process is that it is difficult to develop a device which can detect and grab a corner of a piece of laundry, such as a flatwork item, in a controlled and repeatable manner.

Distinguishing corners and edges appears to be difficult for automated system and this also applies to subsequent gripping of the detected edge or corner.

it is known to identify a second or third corner of a flatwork item once one or two corners are already identified. The flatwork item can for example be in a hanging position, such that the freely hanging corner can be detected and gripped.

However, gripping a free hanging corner cannot be done with a very high accuracy. Hence, it is a goal of the invention to provide an improved method to identify a second or third corner of a flatwork item in order to contribute to further automating of an industrial laundry process.

The invention provides thereto a method for detecting and gripping a predetermined part of a flatwork item, such as a second corner of a flatwork item, comprising the steps of: a) providing a neural network trained to recognize at least one corner of a flatwork item in an image and trained to indicate the position and/or orientation of said at least one corner, b) providing a flatwork item which is clamped at a first corner by a first gripper, and which is preferably held at a distal end opposing the first corner by a clamp, such that at least one second corner substantially hangs fee, c) displacing of the flatwork item over a carrying surface preferably such that the second corner is positioned upon said carrying surface, d) capturing at least one second image of at least part of the second corner of the flatwork item, e) providing the second image to the neural network, f} receiving a possible position and/or orientation of at least one second corner of the flatwork item (provided at step b) ) from the neural network; g) displacing a second gripper towards the determined second corner of said flatwork item, and h) gripping of the determined second corner by said second gripper.

The method according to the invention provides an accurate and replicable way to detect and grip a second corner of a flatwork item. Due to the provision of a neural network trained to recognize at least one corner of a flatwork item in an image and to indicate the position and/or orientation of said at least one corner which is configured to provide information of at least one second image made during step d), the location and/or orientation of the second corner of the flatwork item can be determined in a relatively simple and efficient manner. The use of the captured second image within the neural network, results in receiving a possible position and/or orientation of the second corner of the flatwork item (provided at step b) ) from the neural network.. Basically, the method according to the present invention makes use of image recognition and/or classification in a practical manner. The neural network is trained to determine the orientation and/or position of least one second corner of the flatwork item . The position and/or orientation of the second image is in particular its position and/or orientation with respect to the carrying surface. Typically, the flatwork item is at least partially in a substantially horizontal orientation. It is more accurate to determine the position and/or orientation of the second corner with respect to a carrying surface than to determine such for a freely handing second corner. Further, this may increase the detection speed, and subsequently the speed and ease of gripping. Also the success rate of detection and/or gripping will increase with respect to the conventional methods, and thus the repeatability.

Preferably the flatwork item is held at a distal end opposing the first corner by a clamp. When the first corner is clamped by a first gripper and the distal end of the flatwork item is clamped as well at least one second corner will typically hang free. Further the use of such clamp can ensure that the flatwork item at least temporarily can keep its substantially horizontal orientation. For example, during displacing the flatwork item over the carrying surface. This may contribute to achieving correct positioning of the flatwork item on the carrying surface. It is also conceivable that an air knife is used at any step of the method according to the present invention to control and/or guide the position and/or orientation of the flatwork item. The second gripper is also configured to release the second corner.

The steps of the method according to the present invention are typically successive steps. Capturing of at least one second image can for example be done by visual imaging means or a visual imaging device. This can be any conventional visual imaging means, non-limiting examples thereof are a 2D camera and a 3D camera. When it is referred to a flatwork item, this can be type of conventional flatwork item. Non-limiting examples of flatwork items according to the present invention are towels, cloths, (bed) sheets, napkins, bed linen, duvets, tabletops and/or pillowcases. Such flatwork item is typically substantially rectangular or square. Typically, the flatwork item has at least one corner, preferably at least two corners, and more preferably the flatwork item has four corners. The flatwork item is typically at least partially made of a textile or fabric, which can be either woven or nonwoven. However, it is also conceivable that the flatwork item is a garment. Hence, further non-limiting examples of the flatwork item are a top, shirt, blouse, sweater, trousers and/or a dress.

With respect to the present invention, the term corner is just exemplary, this can also be any predetermined part or area of the flatwork item. Non-limiting examples thereof are an edge, corner region, shoulder part, collar, sleeve, button and/or zipper. Hence, the invention also relates to a method for detecting and gripping a predetermined part of a flatwork item, wherein the neural network is trained to recognize a predetermined part of a flatwork item in an image and to indicate the position and/or orientation of said predetermined part and wherein a possible position and/or orientation is received of said predetermined part of the flatwork item from the neural network, such that a second gripper can be displaced towards the determined corner of the flatwork item.

A further embodiment of the method is conceivable, wherein the method comprises the step of displacing the first gripper and the second gripper with respect to each other, preferably such that the first and second corners of the flatwork item are substantially aligned or parallel to another. With this step the flatwork item can be prepared for further processing step in a relatively simple manner. Mutual displacement of the first and second gripper in a direction facing away from another enables alignment of the first and second corner of the flatwork item. Subsequently via this alignment of the first and second corner, a first edge, or first side, of the flatwork item situated between said first and second corner can be stretched.

Hence, the flatwork item is prepared for several further processing steps. The flatwork item can for example be brought in a position wherein it can be inserted in a further device, such as a conveyor, wherein the first edge is substantially perpendicular to the transport direction. The first and second gripper are typically independently controllable, which can further improve the efficiency of the entire 5 method and/or system. lt is also possible that the method further comprises the steps of taking multiple images of at least one flatwork item preferably wherein the flatwork item is positioned upon a carrying surface, wherein each image shows at least one corner of the flatwork item, annotating, and in particular manually annotating each image with information about a position and/or orientation of at least one corner of said flatwork item, and including the annotated images of flatwork items in a neural network. Typically, these steps are done prior to the provision of the neural network. Manual annotating of images generally only needs to be done once, wherefore this will not significantly limit the overall process in practice. The information about a position and/or orientation of at least one corner of the flatwork item can be 2D and/or 3D related. It is also conceivable that the annotating of the first image is computer-controlled. It is conceivable that no corners are observed during annotating of the images. This is also useful since it is conceivable that the flatwork item is not correctly positioned upon the carrying surface such that no corners can be visually observed.

The method may further comprise steps wherein it is determined if gripping of the second corner by the second gripper was successful, annotating at least one image made at step d) with information of the position and/or orientation of the second corner, and adding of the annotated second image to the neural network. Preferably annotating of an image is only done when the gripping of the second was classified as successful. Annotating of the captured first image with information about the position and/or orientation and adding this annotated image to the neural network can contribute to the ability of the method to faster and/or better recognize a corner in a newly captured second image. Including the annotated second image in the neural network may further enhance the reliability of the method. The annotating of the second image may for example be done manually. However, it is also conceivable that the annotating of the first image is computer-controlled.

Basically, a continuous method for determining a second corner of a flatwork item can be obtained.

The method may even qualify as a continuous deep learning system.

Gripping of a second corner can for example be classified as successful if the corner is gripped by the second gripper such that the second gripper can take along the flatwork item to a further processing step, preferably in co-acting with the first gripper holding the first corner.

If the second corner was not sufficiently gripped the second gripper may lose the corner.

If this is detected, for example by a further visual imaging device, the gripping cannot be classified as successful.

The entire flatwork item may then be discarded.

It is conceivable that the second image will not be annotated and/or added to the neural network if the gripping was not successful.

Hence, the annotation and addition of an image is possibly only done when the gripping can be classified as successful.

Displacing of the flatwork item over a carrying surface preferably such that the second corner is positioned upon said carrying surface, can be performed by displacing the first corner at least via the first gripper.

The first gripper is preferably displaced substantially parallel to the carrying surface, in order to achieve a substantially horizontal orientation of at least part of the flatwork item.

It is also conceivable that the displacement is achieved via displacing the first corner via the first gripper and via displacing the distal end of the flatwork item via the clamp.

With the latter method a more controlled displacement of the flatwork item may be achieved.

Further this enables a substantially horizontal orientation of the flatwork item during displacing thereof.

Preferably, at least part of the flatwork item is in substantially horizontal orientation during the detecting and/or gripping of the second corner.

The flatwork item being in a substantially horizontal orientation during detection and/or gripping of the second corner can further simplify the detection and/or gripping of said second corner.

The detection and gripping of a corner when the flatwork item is in a substantially horizontal orientation, for example because it is supported by the carrying surface, is easier than detection and gripping of a hanging corner due to a reduced degree of freedom.

It is further possible that the method comprises the step of determining the distance between the first corner and the second corner after step c), and discarding the flatwork item if the distance does not exceed a predetermined threshold distance,

and/or selecting the second corner which is at the closest distance to the first corner for gripping by the second gripper if more than one second corner is detected. It is possible that during more than one second corner is identified in the second image. This can for example be the situation when the flatwork item is substantially square and/or quadratic and/or relatively small. If two corners are detected, the corner closest to the first corner should be depicted. This means generally that the shortest side of the flatwork item is identified. If the determined distance does not exceed the predetermined distance, it may be possible that instead of a second corner the first corner is accidentally detected. lt is also possible that something went wrong during displacing of the flatwork item. A non- limiting measure for the threshold distance is for example 200 mm or lower, preferably 15 mm or lower and more preferably 100mm or lower. it is further possible, that prior to step c), the step of shaking the flatwork item prior such that at least a second corner which is at shortest distance from the first corner will hang free is applied. This step can contribute to preparation of the flatwork item before it is displaced towards or over the carrying surface. It enables better starting position of the second corner of the flatwork item before the corner is brought into position for detection and gripping. The shaking will loosen the structure of the flatwork item in case the flatwork item is for example twisted, folded or wrinkled. Under the term shaking, any vibrating or oscillating movement should be understood. Also shuddering or swinging can be meant. The purpose of the shaking movement is preparing the flatwork item for the displacement and for the identification of the second corner.

In a preferred embodiment of the method according the invention, step h) comprises the (subsequent) steps of lifting the second corner of the flatwork item via suction, and gripping the second corner via a enclosing at least part of the second corner by the second gripper, in particular by a clamping element of the second gripper. Hence, the second gripper preferably comprises a suction element in combination with a (movable) clamping element. Lifting of the second corner of the flatwork item via suction can for example be done by applying a vacuum. With suction, typically suction of air is meant. The suction is preferably temporarily and may be stopped once the second corner is gripped by the second gripper. The clamping element can for example be a gripping mouth. Enclosing of at least part of the second corner can provide a reliable and firm engagement of the corner, and thus the flatwork item, by the second gripper. The combination of lifting and gripping of the second corner enables that an item of flatwork, which is mostly a textile or the like, can be gripped in a controlled manner. Due to the flatwork item typically being made of a textile or fabric it is difficult to grip a corner which is positioned on a substantially flat surface, such as a carrying surface. It is preferred that the second corner is gripped via enclosing at least part of the corner by the second gripper, such that controlled further displacing of the flatwork item is enabled. If the first and second corners are gripped via enclosing at least part of each corner by the first or second gripper, controlled further displacing of the flatwork item is enabled. Hence, the method equals the steps which are typically done manually, and can therefore replace this manual steps.

In a beneficial embodiment of the method, the method further comprises the steps of suction of air from underneath the carrying surface such that the flatwork item is pressed against the carrying surface after step c) and preferably before step d), and stopping, or discontinuing, the suction of air after the second gripper has gripped the second corner. For example, a vacuum can be applied underneath the carrying surface. Hence, suction of air can be created, thereby drawing the flatwork item towards the carrying surface. The carrying surface is typically at least partially permeable to air. The carrying surface for example comprises at least one air opening. The vacuum, or suction of air, is typically released after the second corner has been detected and gripped. In a further possible embodiment, the method comprising the step of blowing air through the carrying surface such that the flatwork item is pushed away from the carrying surface. Hence, in addition to the suction of air, it is also possible to use an air flow for guiding the flatwork item after the second corner is gripped by the second gripper. Such air flow can for example be created via at least one fan.

Possibly the method also comprises the step of feeding the flatwork item to a transport conveyor and/or a flatwork treating device. A non-limiting example thereof is a folding machine and/or a mangle.

The invention also relates to a system for detecting and gripping a second corner of a flatwork item, preferably via a method as described above, the system comprising atleast one visual imaging device configured for capturing at least one second image of at least part of a flatwork item, a neural network trained to recognize at least one corner of a flatwork item in an image and to indicate the position and/or orientation of said at least one corner, , a displaceable first gripper configured for gripping a first corner of a flatwork item, a displaceable second gripper configured gripping a second corner of the flatwork item, and a carrying surface configured for carrying at least part of a flatwork item. The system according to the invention experiences the same benefits as described above for the corresponding method. Hence, the system enables the detection and gripping of a second corner of a flatwork item in a controlled and repeatable manner.

The second gripper of the system is preferably configured for lifting at least part of the second corner before gripping said corner. Preferably, the second gripper comprises a suction element in combination with movable clamping element. The second gripper may for example comprise at least one suction element configured for at least partially lifting an edge and/or a corner of a flatwork item, and at least one clamping element for clampingly engaging said flatwork item, wherein the clamping element is configured to enclose at least part of an edge and/or a corner of the flatwork item.

lt is beneficial if the carrying surface is at least partially permeable to air. The carrying surface can for example comprises at least one air opening and/or may be at least partially configured by a mesh or mesh structure. Hence, the carrying surface can enable that a flow of air through the carrying surface can be applied. This can be either attract (draw or pull) the flatwork item towards the carrying surface, or push the flatwork item away from it. This can be beneficial for controlling the motion or motionlessness of the flatwork item. Typically, the carrying surface is at least partially substantially flat. A substantially flat carrying surface can provide improved support of the flatwork item.

The system may further comprise at least one clamp configured to hold a distal end of the flatwork item opposing the first corner. The clamp can be any conventional clamp configured to hold or clamp part of a flatwork item. The function of the clamp is guiding the distal end of the flatwork during use of the system. The clamp can for example be a claw.

The first gripper and the second gripper are preferably independently movable and controllable. This may further improve the efficiency of the entire system. The first gripper, and in particular the gripping mouth, and/or the second gripper, and in particular the clamping element thereof, is preferably rotatable over at least 180 degrees, preferably 360 degrees, over an axis of rotation. This will result in the gripper(s) having a relatively high degree of freedom. The system may comprise at least one drive unit to drive one or more grippers and/or the clamp. The visual imaging device of the system may for example be a camera. The visual imaging device can be any conventional visual imaging means, non-limiting examples thereof are a 2D camera and a 3D camera. The invention also relates to the use of a system according to the present invention.

The invention also relates to a gripper for gripping an edge and/or corner of a flatwork item, the gripper comprising a suction element configured for at least partially lifting an edge and/or a corner of a flatwork item, and a clamping element for clampingly engaging said flatwork item, wherein the clamping element is configured to hold and/or enclose at least part of said edge and/or a corner of the flatwork item. Such gripper is configured for and suitable for use in a method and system according to the present invention. The suction element is typically configured for suction of air. Due to the suction of air, at least part of the flatwork item can be drawn towards the suction element and at least temporarily retained.

The suction element and the clamping element are preferably mutually displaceable. It is for example conceivable that the clamping element is pivotable with respect to the suction element. After the suction element has lifted at least part of the flatwork item, the clamping element can be displaced and used in order to clampingly engage the edge and/or corner of the flatwork item. The air intake of the suction element can be discontinued once a clamping engagement is established. The present invention will be explained further on the basis of the non-limiting embodiment examples presented in the following figures.

Figures 1a- shows a schematic representation of a system 100 according to the present invention. The system 100 is configured for detecting and gripping a second corner 101b of a flatwork item 101, preferably via a method according to the present invention. The system 100 comprises a neural network trained to recognize at least one corner of a flatwork item in an image and to indicate the position and/or orientation of said at least one corner. The system 100 also comprises a visual imaging device 108 configured for capturing at least one second image of at least part of the flatwork item. With respect to the method, figure 1a shows that a flatwork item 101 is provided which is clamped at a first corner 101a by a first gripper 104, and which is, in the shown embodiment held at a distal end D opposing the first corner 101a by a clamp 106. This enables that at least one second corner 101b substantially hangs fee. Subsequently the flatwork item 101 is displaced over a carrying surface 107. The arrow indicated the direction of displacement. The carrying surface 107 is at least partially permeable to air. The carrying surface 107 is substantially flat and comprises multiple air openings 107a. Figure 1b shows the flatwork item 101 being positioned upon the carrying surface

107. The second corner 101b is positioned upon said carrying surface 107. Then at least one second image of at least part of the second corner 101b of the flatwork item 101 is to be captured by the visual imaging device 108. Subsequently the neural network provides a possible position and/or orientation of at least one second corner of the flatwork item. Then the second gripper 109 can be displaced towards the determined second corner 101b which can then be gripped.

Figure 2 shows a schematic representation of a possible embodiment of a second gripper 209 according to the present invention and configured for use in a method and system according to the present invention. The gripper 209 is configured for gripping, and releasing, an edge and/or corner of a flatwork item (not shown). The gripper 209 comprising a suction element 210 configured for at least partially lifting an edge and/or a corner of the flatwork item, and a clamping element 211 for clampingly engaging said flatwork item. The clamping element 211 is configured to hold and/or enclose at least part of an edge and/or a corner of the flatwork item. The clamping element 211 and the suction element 210 are mutually displaceable. Typically, the clamping element 211 is displaceable with respect to the suction element 210. The arrows indication possible displacement directions of clamping element 211 and the suction element 210. The clamping element 211 is in the shown embodiment pivotably displaceable with respect to the suction element 210. The suction element 210 is also displaceable. The suction element 210 may for example comprise a telescopic arm. -Due to the clamping element 211 being displaceable with respect to the suction element 210, the corner or edge can be firmly gripped. Typically, the suction element 210 draws air into the device thereby attracting the flatwork item. The gripper 209, and in particular the suction element 210, is then lifted such that the flatwork item is no longer positioned upon the carrying surface. Then the clamping element 211 is displaced such that it provided a clamping engagement with the corner and/or edge of the flatwork item. In the shown embodiment, the clamping element 211 comprises a telescopic arm such that the clamping element 211 can be displaced with respect to the suction element 210 in a relatively easy manner.

It will be clear that the invention is not limited to the embodiment examples presented and described here, but that numerous variants are possible within the scope of the appended claims, which will be obvious to a person skilled in the art. It is conceivable that various inventive concepts and/or technical measures of the embodiment variants described above may be combined completely or partially without departing from the inventive concepts described in the appended claims. The verb “comprise” and conjugations thereof used in this patent specification mean not only "comprise", but also the expressions "include", "consist essentially of", "formed by", and conjugations thereof.

When the word ‘gripping’ is used, also the terms grasping, picking, holding, grabbing and/or clasping can be meant. When it is referred to a corner of a flatwork item, the point where two edges of the flatwork item meet is meant. lt also possible that the corner is a shoulder (part) of for example a shirt, sweater or a (bath)robe.

The corner is typically gripped by double sided engagement of the textile. The position and/or orientation of the first image can be determined in both a 2D and/or 3D configuration.

Claims (1)

Conclusions A method for detecting and engaging a second corner of a flat object, comprising the steps of: a) providing a neural network that is trained to recognize at least one corner of a flat object in an image and to determine the position and / or to indicate orientation of said at least one corner, b) providing a flat object that is clamped at a first corner by a first gripper, and which is preferably held at a distal end opposite the first corner by a clamp , such that at least one second corner is substantially free-hanging, C) moving the flat object over a support surface, preferably such that the second corner is positioned on the support surface, d) capturing at least one second image of at least one at least a portion of the second corner of the flat object, e) providing the second image to the neural network, f) receiving e from the neural network and possible position and / or orientation of at least one second corner of the flatware object, g moving the second gripper to the determined second corner of said flatware object, and h) engaging the second corner by said second gripper. The method of claim 1, further comprising the step of: i) moving the first gripper and the second gripper relative to each other. A method according to claim 1 or 2, further comprising the steps of: i) taking multiple images of at least one flatware object, the flatware object being preferably positioned on a support surface, each image having at least one corner of a flatware. object, k} annotating each image with information about the position and / or orientation of at least one corner of the flat object, and h incorporating the annotated images into a neural network. A method according to any preceding claim, further comprising the steps of: m) determining whether the second gripper engaging the second corner was Successful, n) annotating at least one image with information taken during step d) about the position and / or orientation of the second angle, and 0) processing the annotated second image in the neural network. A method according to any one of the preceding claims, wherein step c) is performed at least by moving the first corner via the first gripper. 6. A method according to any one of the preceding claims, wherein at least a part of the flatware object is oriented substantially horizontally during the detection and / or engagement of the second corner. A method according to any one of the preceding claims, comprising the steps of: p) determining after step c) the distance between the first corner and the second corner, and q) rejecting the flatware object if the distance is not a predetermined one. certain threshold distance, and / or r) selecting a second corner that is the closest distance from the first corner for engagement by the second gripper if more than one second corner is detected. A method according to any one of the preceding claims, comprising the step of: s) shaking the flatware object prior to step c) such that a second corner that is closest to the first corner is exposed. to hang. A method according to any one of the preceding claims, wherein step h) comprises the following steps: hi) lifting the second corner of the flat object by means of suction, and h2) engaging the second corner by enclosing at least a part of the second corner by the second gripper, in particular a clamping element of the second gripper . A method according to any one of the preceding claims, further comprising the steps of: t) after step c), and preferably before step d), drawing in air under the bearing surface, such that the flat material object is pressed against the bearing surface, and u) stopping the suction of air after the second gripper has engaged the second corner. The method of claim 10, further comprising the step of blowing air through the support surface such that the flatware object is pushed away from the support surface. A method according to any one of the preceding claims, further comprising the step of: Vv) supplying the flatware object to a conveyor belt and / or a flatware treatment device. System for detecting and engaging a second corner of a flat object, preferably by means of a method according to any one of claims 1-12, comprising: - at least one visual imaging device which is adapted to capture at least one second image of at least part of a flat object, - a neural network trained to recognize at least one corner of a flat object in an image and to indicate the position and / or orientation of said at least one angle, - a movable first gripper adapted to engage a first corner of a flat object, - a displaceable second gripper adapted to engage a second corner of a flatware object, and - a carrying surface adapted to support at least a part of a flatware object. The system of claim 13, wherein the second gripper is configured to lift at least a portion of the second corner before engaging the corner. System according to claim 13 or 14, wherein the support surface is at least partially air permeable. A system according to any of claims 13-15, comprising at least one clamp adapted to hold a distal end of the flatware object opposite the first corner. System according to any of claims 13-16, wherein the first gripper and the second gripper are independently movable and controllable. 18. System as claimed in any of the claims 13-17, wherein the second gripper comprises a suction element in combination with a movable clamping element. The system of any of claims 13-18, wherein the visual imaging device comprises a camera.