JP7353929B2 - Image processing device, image processing method, control program, recording medium, robot system, imaging device, robot system control method, article manufacturing method - Google Patents

Description

The present invention relates to an image processing device and an image processing method that are used to detect the state of a tape pasted, for example, when assembling a box by pasting tape onto a material such as cardboard.

In the automatic assembly of boxes made of cardboard or the like, a box-making machine equipped with a specialized tool for folding flaps or a robot hand is generally used. Some box making machines not only automatically fold and assemble cardboard boxes, but also automatically apply tape to close cardboard boxes. To enable the tape pasting work, a special mechanism is installed in the box making machine.

For example, in the case of a box making machine that uses a specialized tool for folding flaps, a commercially available tape applicator is placed above a conveyor that moves cardboard boxes, and is incorporated into the box making machine to apply tape. Specifically, when a cardboard box is moving on a conveyor, a tape applicator is pressed against the box, the tape is fixed, and the box is moved and pasted.

Furthermore, as described in Patent Document 1, for example, in the case of a box making machine using a robot hand, a small taping unit is mounted on the robot hand and tape is pasted. That is, the box is fixed and the tape is moved and pasted using a robot hand.

However, if the operating precision of the flap folding jig or robot hand is insufficient, or if the operating precision of the tape applicator or taping unit deteriorates over time, problems may occur in the application of the tape. be. For example, in the former case, when the flaps on both sides of the cardboard are closed, the closed portions will wobble in the height direction, causing problems in pasting the tape. In the latter case, for example, the tension of the tape is not sufficiently controlled when it is pasted onto the cardboard, resulting in problems such as the tape becoming wrinkled or running diagonally to the closed edges of the cardboard. arise.
Cardboard boxes that have problems with tape attachment need to be excluded as defective products, and measures need to be taken on the box making machine to prevent recurrence of the problem. Therefore, there is a need for an adhesion defect detection system that automatically determines the quality of the adhesion of the tape.

Patent Document 2 proposes a system for detecting wrinkles formed on the surface of tapes or films. In other words, it is equipped with an imaging section and an image processing section, and the subject is illuminated and photographed using an imaging means such as a video camera or a line sensor. This is to clarify.
With reference to FIGS. 9(a) to 9(c), a case will be described in which the wrinkle determination method described in Patent Document 2 is applied to a pasting defect detection system for a box making machine.

First, FIG. 9(a) shows a part of the assembled cardboard box, that is, the part where the tape 202 is pasted on the closed seam 203 of the two closed flaps 201, by the imaging method described in Patent Document 2. This schematically represents an image taken from above using a method. Moreover, FIG.9(b) is the figure which observed this cardboard box from the side.
The curved line visible around the closed eye 203 in FIG. 9(a) represents wrinkles generated on the surface of the tape 202. The wrinkles generated on the tape surface in the manner shown in FIGS. 9(a) and 9(b) are called tape wrinkles 904. Note that tape wrinkles are of a different type from sagging wrinkles, which will be explained later with reference to FIGS. 10(a) and 10(b). The tape wrinkles 904 run irregularly in the X and Y directions in a plan view, and have a mountain-like shape with an irregularly protruding height H in a side view. Therefore, when illumination is applied, the incident light is diffusely reflected by the irregularly shaped tape wrinkles 904, and a change in brightness (shade) occurs at the tape wrinkles 904 when viewed from above.

FIG. 9(c) is a diagram for explaining the image processing that the image processing unit provided in Patent Document 2 performs on the photographed image. Changes are quantified. While performing differential processing along a differential processing line parallel to the X axis, the differential processing line is scanned in the Y direction to process the entire image area. In the differential processing line 905a at the position where the tape wrinkle 904 is not formed, there is almost no change in shading. However, in the differential processing line 905b at the position where the tape wrinkle 904 exists, there is a change in brightness in the X direction, so a change occurs in the numerical value. If the numerical change is greater than the threshold value, the pasting defect detection system determines that tape wrinkles 904 are present.

JP 2014-823 Publication Japanese Patent Application Publication No. 4-305145

If the pasting defect detection system for a box making machine is applied using the method described in Patent Document 2, the boxes run irregularly in the X and Y directions when viewed from above, and the height H protrudes in the shape of an irregular mountain range when viewed from the side. There is a certain effect in detecting tape wrinkles having a shape.
However, the method described in Patent Document 2 has insufficient detection sensitivity for wrinkles of a type different from tape wrinkles (hereinafter referred to as sagging wrinkles).

Sagging wrinkles will be explained with reference to FIGS. 10(a) and 10(b). Sagging wrinkles are not wrinkles that run irregularly in the X and Y directions in a plan view and protrude like a mountain range with an irregular height H in a side view, like tape wrinkles, but extend over the entire width in the width direction in a plan view. When viewed from the side, it refers to wrinkles that bulge out to form a ridge of approximately constant height. Since the tape was attached to the cardboard in a slack state in the longitudinal direction, the entire width of the tape in the width direction (short direction) was lifted from the cardboard in a partial area.

FIG. 10(a) schematically represents an image taken from above using an imaging means of the part where the tape 202 is pasted on the closed eye 203 of the two closed flaps 201, and shows an image taken from above in the X direction. The figure shows a state in which sagging wrinkles 204 are formed along. FIG. 10(b) is a side view of this cardboard box. The sagging wrinkles 204 are substantially semicylindrical bulges that protrude from the flap 201 by a height H. The height H of the sag wrinkles is determined depending on the relationship between the tape width, the tape thickness, and the tension applied to the tape. If the tension is sufficient, the height will be 0 (that is, no sagging wrinkles will be formed), but if the tension is insufficient, sagging wrinkles will occur. For example, when a vinyl tape with a width of 65 mm is used, sagging wrinkles with a height of up to about 3.4 mm may occur.

When photographing from above as shown in FIG. 10(a), unlike the tape wrinkles 904, the sagging wrinkles 204 do not cause diffuse reflection even when illuminated, so even if the image is scanned along the X direction, the shading changes. is small. Therefore, even if the change in shading is read by the differential processing according to Patent Document 2, the amount of change is small, so it is difficult to detect sagging wrinkles.

Furthermore, even if the method described in Patent Document 2 is applied to a pasting defect detection system for a box making machine, it will not be possible to detect the pasting position shift that occurs when the tape is pasted obliquely to the closed seam of the cardboard. I couldn't.

Therefore, there is a need for an image processing device and an image processing method that can detect various types of wrinkles and misalignment of the tape attached to a cardboard box or the like based on a photographed image.

A first aspect of the present invention is to acquire an image taken to include the tape attached to the closed area of a first member and a second member, calculate the contrast ratio of each part of the image, and calculate the contrast ratio of each part of the image, and This image processing method is characterized in that the state of the attached tape is detected using the contrast ratio of the tape attached.

A second aspect of the present invention includes an image acquisition unit that acquires an image captured to include the tape attached to the closed portion of the first member and the second member, and each part of the image acquired by the image acquisition unit. a contrast ratio detection unit that calculates a contrast ratio of the tape, and a pasting detection unit that detects a state of the tape pasted using the contrast ratio of the tape pasted to the closed eye calculated by the contrast ratio detection unit. An image processing device comprising:

A third aspect of the present invention includes a robot that affixes a tape to the closed portion of a first member and a second member, and a tape affixed to the closed portion of the first member and the second member. an image acquisition unit that acquires an image captured by the image acquisition unit; a contrast ratio detection unit that calculates a contrast ratio of each part of the image acquired by the image acquisition unit; and a contrast ratio detection unit that calculates the contrast ratio of each part of the image acquired by the image acquisition unit; The robot system is characterized in that it includes a pasting detection section that detects the state of the pasted tape using the contrast ratio of the tape .

A fourth aspect of the present invention includes: an imaging unit; an image acquisition unit configured to acquire an image from the imaging unit including the tape attached to the closed portion of the first member and the second member; A contrast ratio detection unit calculates the contrast ratio of each part of the image acquired by the acquisition unit, and a contrast ratio of the tape attached to the closed eye is calculated using the contrast ratio of the tape attached to the closed eye. An imaging device characterized by comprising a pasting detection section that detects a state.

The present invention provides an image processing device and an image processing method that can detect various types of wrinkles and misalignment of the tape attached to a cardboard box or the like based on a photographed image.

(a) A functional block diagram showing the configuration of functional blocks included in the image processing apparatus according to the embodiment. (b) Side view showing the installation position of the vision. FIG. 2 is a schematic diagram showing an image taken from above of a tape application area where sagging wrinkles have occurred. (a) A diagram for explaining a method of calculating a contrast ratio. (b) A diagram showing an example of a change in contrast ratio when there are sagging wrinkles. 2 is a flowchart showing a procedure for determining the presence or absence of sagging wrinkles. 2 is a flowchart showing another procedure for determining the presence or absence of sagging wrinkles. A flowchart showing a procedure for estimating a normal contrast ratio from an acquired image. FIG. 3 is a diagram schematically showing an image taken from above of a state where misalignment has occurred. 10 is a flowchart of a process for determining whether there is misaligned pasting. (a) A diagram showing an image taken from above of the tape pasting part of the assembled cardboard box. (b) Diagram of a cardboard box observed from the side. (c) A diagram for explaining conventional image processing. (a) A schematic diagram showing an image taken from above of a tape pasting area where sagging wrinkles have occurred. (b) A side view of the tape application area where sagging wrinkles have occurred. FIG. 1 is a schematic diagram of a hardware configuration of an image processing device according to an embodiment. FIG. 1 is a schematic diagram showing the configuration of a robot system that combines an image processing device according to an embodiment and a robot equipped with a robot hand.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An image processing apparatus and an image processing method according to embodiments of the present invention will be described with reference to the drawings. The image processing apparatus and image processing method of the embodiment detect whether the tape is adhered to a box made of cardboard. However, this is just an example, and the material of the box is not limited to cardboard. Furthermore, the object for detecting the quality of the tape attachment is not limited to boxes, but can be widely applied to inspecting the tape attachment portion of objects that have been assembled or sealed using tape.
In the drawings referred to in the following description, unless otherwise specified, elements designated with the same reference numerals have the same functions.

[Functional block configuration]
FIG. 1A is a functional block diagram showing the configuration of functional blocks included in the image processing apparatus according to the embodiment. The image processing device of the embodiment detects the presence or absence of wrinkles in the tape and the presence or absence of deviation in the application position in various ways based on image data taken of the tape application part of the cardboard box that is the inspection object. .

The image processing device 102 includes an external IF 104 that is an interface part with external equipment, and is connected to the vision 101 and the lighting device 110 via the external IF 104.
Specifically, the vision 101 is a digital camera equipped with a CMOS sensor or CCD as an image sensor, and as shown in FIG. It is installed in a position where it can image the surrounding area. Although not shown in FIG. 1B, the illumination device 110 is used to detect the closing of the flap, wrinkles in the tape, and the boundary between the cardboard and the tape when photographing the vicinity of the tape 202 pasting position using the vision 101. to illuminate the shooting position so that the contrast of the image stands out. The captured image may be either color or monochrome as long as it is a gradation image.

The image processing device 102 sends control signals to the vision 101 and the lighting device 110 via the external IF 104 to control their operations, and also sends image data (multi-gradation digital signal) captured by the vision 101 via the external IF 104. can be received.
The image processing device 102 includes an image acquisition unit 105 that is a memory for storing received image data. image processing is performed.

The pasting defect detection section 103 includes a brightness change detection section 107 , a flap closure contrast ratio detection section 108 , and a tape edge contrast ratio detection section 109 . The brightness change detection unit 107 is used to detect the presence or absence of tape wrinkles. The flap closed eye contrast ratio detection unit 108 is used to detect the presence or absence of sagging wrinkles. Further, the tape end contrast ratio detection unit 109 is used to detect the presence or absence of a deviation in the pasting position. The brightness change detection unit 107, the flap closed eye contrast ratio detection unit 108, and the tape edge contrast ratio detection unit 109 compare the image processing result with the threshold value stored in the threshold value management unit 106, and determine the degree of tape attachment. Detects the quality of the product. The method of detecting pass/fail will be described later.

A threshold management unit 106 manages thresholds for brightness values and contrast ratios calculated by a brightness change detection unit 107, a flap closure contrast ratio detection unit 108, and a tape end contrast ratio detection unit 109, which will be described later. It also manages a threshold value for the distance between the flap closing eye and the tape end calculated by the tape end contrast ratio detection unit 109. The threshold value can be set arbitrarily by the user, and can be changed as necessary.

Each functional block has been described above, and these functional blocks are configured by the CPU of the image processing device reading and executing a control program stored in, for example, a storage device. Alternatively, part or all of the functional blocks may be configured by hardware such as an ASIC included in the image processing device.

[Hardware configuration]
FIG. 11 schematically shows an example of the hardware configuration of the image processing apparatus according to the embodiment. As shown in FIG. 11, the image processing apparatus can include PC hardware including a CPU 1601 as a main control means, a ROM 1602 as a storage device, and a RAM 1603. The ROM 1602 can store information such as processing programs and inference algorithms for implementing a pasting defect detection method to be described later. Further, the RAM 1603 is used as a work area for the CPU 1601 when executing the control procedure. Further, an external storage device 1606 is connected to the control system. The external storage device 1606 includes an HDD, an SSD, an external storage device of another system mounted on a network, and the like.

The processing program of the CPU 1601 for implementing the image processing method of this embodiment, which will be described later, can be stored in an external storage device 1606 such as an HDD or SSD, or a storage unit such as a ROM 1602 (for example, an EEPROM area). can. In that case, the processing program of the CPU 1601 for implementing the image processing method can be supplied to each of the above storage units via the network interface (NIF) 1607, and can be updated to a new (different) program. Alternatively, the processing program of the CPU 1601 for realizing the image processing method may be supplied to each of the above-mentioned storage units via storage means such as various magnetic disks, optical disks, flash memories, etc., and a drive device for that purpose. Its contents can be updated. Various storage means, storage units, or storage devices storing the processing program of the CPU 1601 for realizing the image processing method constitute a computer-readable recording medium storing the image processing procedure of the present invention.

The vision 101 and lighting device 110 shown in FIG. 1(a) are connected to the CPU 1601. In FIG. 11, in order to simplify the illustration, the vision 101 is shown as being directly connected to the CPU 1601, but it may be connected via, for example, IEEE488 (so-called GPIB). Further, the vision 101 may be configured to be connected via a network interface 1607 and a network 1608.

The network interface 1607 can be configured using a communication standard such as wired communication such as IEEE 802.3 or wireless communication such as IEEE 802.11 or 802.15. CPU 1601 can communicate with other devices 1104 and 1121 via network interface 1607. The devices 1104 and 1121 may be, for example, a box-making machine or robot that assembles cardboard boxes, a general control device such as a PLC or sequencer arranged for production control and management, a management server, or the like.

In the example shown in FIG. 11, an operation unit 1604 and a display device 1605 related to the image processing apparatus 102 shown in FIG. 1 are connected as a UI device (user interface device). The operation unit 1604 can be configured by a terminal such as a handy terminal, or a device such as a keyboard, jog dial, mouse, or pointing device (or a control terminal including these). The display device 1605 may be any device as long as it can display information related to photographed images, evaluation of the state of tape attachment, etc. on a display screen, and for example, a liquid crystal display device can be used.

[Image processing method]
Next, an image processing method according to the present embodiment, that is, a process for inspecting and determining various pasting defects will be described.

(Inspection of presence or absence of tape wrinkles)
The procedure for inspecting the presence or absence of tape wrinkles will be explained. The brightness change detection unit 107 of the image processing device 102 extracts the tape portion from the image acquired by the image acquisition unit 105, and detects the presence or absence of a portion where the brightness changes significantly relative to the surrounding area (portion with a large contrast ratio). To detect. To detect a portion where the contrast ratio is large, for example, as shown in FIG. 9(b), differential processing is performed on the luminance data along a one-dimensional direction, and the change in the calculated luminance value is calculated using the data. List as a value. If there are no tape wrinkles that run irregularly in the X and Y directions when viewed from the top and have irregularly protruding heights when viewed from the side, there will be almost no change in the brightness value of shading, but tape wrinkles will occur. , the brightness value of the gradation changes. The brightness change detection unit 107 accesses the threshold value management unit 106, obtains a threshold value corresponding to a contrast ratio for detecting tape wrinkles, and compares whether the calculated change in brightness value is within the threshold value. In this way, it is determined whether or not a sticking defect due to tape wrinkles has occurred.

Information related to the above detection results and judgment results, such as the presence or absence of tape wrinkles, the position and number of tape wrinkles, etc., is notified to the user via the user interface, and is also stored in the storage device. , and send it to the box making machine through an external interface. To notify the user, processing such as displaying on a display device connected to the image processing device or issuing a voice message may be performed.

(Inspection for presence of sagging wrinkles)
The procedure for inspecting the presence or absence of sagging wrinkles will be explained. The sagging wrinkles are wrinkles in the form of the tape raised so as to form a ridge having a substantially constant height over the entire width of the tape in the width direction.
FIG. 2 schematically shows an image taken from above using an imaging means of an object in which the tape 202 is pasted on the closed eye 203 of the two closed flaps 201, and the image is taken from above using an imaging means. A state in which sagging wrinkles 204 are formed is shown. One of the two flaps 201 may be called a first member and the other may be called a second member, and the first member and the second member may be integral or separate.

At the sagging wrinkles 204, the tape 202 is raised over the entire width from the upper surface of the flap 201, but at the portion 207 where it is pasted without any sagging wrinkles, the tape 202 is in close contact with the flap 201. Therefore, in an image taken from the top, the appearance of the closed flap 203 differs between the part with the sagging wrinkles 204 and the part 207 pasted without the sagging wrinkles.
That is, since there is a distance between the tape and the closed part in the part of the slack wrinkle 204, the closed part 205 visible through the tape is the closed part 206 visible through the tape in the part 207 pasted without slack wrinkles. The brightness value becomes higher than In FIG. 2, in order to schematically represent this, a closed eye portion 206 that has no sagging wrinkles and can be seen through the tape is shown with a thick broken line, and a closed eye portion 205 that is visible through the sagging wrinkles is shown with a thin broken line. In light of the contrast ratio calculation formula described later, the contrast ratio calculated in the sagging and wrinkled portions is low. On the contrary, in the part 207 pasted without any slack or wrinkles, the tape 202 and the closed part 206 are in close contact and there is no distance, so the brightness value of the closed part 206 visible through the tape becomes low, that is, it becomes dark. . Therefore, the calculated contrast ratio becomes high, contrary to the case of sagging wrinkles. In this way, by detecting the contrast ratio of the closed flap 203, it is possible to inspect and determine the presence or absence of sagging wrinkles.

The flap closed eye contrast ratio detection unit 108 performs a process of extracting the flap closed eye portion from the image acquired by the image acquisition unit 105 and calculating the contrast ratio. The flap closed eye contrast ratio detection unit 108 accesses the threshold value management unit 106 to obtain a threshold value for the amount of change in contrast ratio, compares whether the calculated change in contrast ratio is within the threshold value, and determines the presence or absence of sagging wrinkles. judge. Furthermore, the flap closing position in the acquired image is estimated from the position where the calculated contrast ratio changes.

FIG. 3(a) is a diagram for explaining a contrast ratio calculation method, and FIG. 3(b) is a diagram showing an example of a change in contrast ratio when there are sagging wrinkles.
FIG. 3(a) is an enlarged schematic plan view of the tape 202 and the closure 203 of the flap. When the XY directions are set as shown in the figure, the contrast ratio of the closed eye 203 is calculated in the order of Y(n), Y(n+1), Y(n+2), . . . . The contrast ratio of Y(n) is determined by the luminance value of the pixel X(n) of the closed eye 203 and the luminance value of X(n+k) around the closed eye 203. That is, since the luminance value of pixel X(n) is a dark area and the luminance value of pixel X(n+k) is a bright area, the contrast ratio is calculated by the following equation (1).

Contrast ratio = (bright area) / (dark area)
= Brightness value of pixel X(n+k)/luminance value of pixel X(n+k)...(1)

In FIG. 3(a), for convenience of illustration, Y(n), Y(n+1), and Y(n+2) are shown separated, but basically Y(n) is scanned every pixel. conduct. In this example, it is assumed that the start and end of n are input in advance by the user via the user interface. It is also assumed that X(n) and X(n+k), which are the positions at which the bright and dark areas are measured, are also input and specified by the user via the user interface. However, since the bright and dark areas spread depending on the image, instead of using only one pixel, for example, X(n) and X(n+k) may both be average values of a plurality of pixels.

FIG. 3(b) is a schematic graph showing the change in contrast ratio of the closed eye 203 when there are sagging wrinkles in FIG. 2. Since sagging wrinkles are generated from Y(15), a change in contrast ratio also occurs at this position. Then, the decrease in contrast ratio reaches a peak around Y(21), Y(22), and Y(23), where the height of the sagging wrinkles is the highest. In this way, sagging wrinkles can be detected based on changes in contrast ratio.

FIG. 4 is a flowchart showing a procedure for determining the presence or absence of sagging wrinkles. In this embodiment, a process is performed in which the amount of change due to the difference between the contrast ratios before and after in the Y direction is determined, and when the difference is larger than a threshold value, it is determined that sagging wrinkles are present.

First, in S401, a contrast ratio threshold is set. The threshold value is a reference value indicating to what extent the contrast ratio may change when there are no sagging wrinkles. The threshold value is set by the user.
Next, in S402, pixel coordinates from the start to the end of scanning in the Y direction are set. In addition, pixel coordinates for calculating brightness values in the X direction are set.
Next, in S403, calculation of contrast ratio is started for the acquired image. That is, the contrast ratio of Y(n) is calculated from the start pixel coordinates set in S402. Then, in S404, the first pixel coordinates Y(n) and Y(n+1) are calculated.
In S405, the amount of change P is calculated by calculating Y(n+1)−Y(n).

Then, in S406, a comparison is made to see if the amount of change P is larger than a threshold value. Here, if the amount of change P is larger than the threshold value (S406: Yes), it is determined in S407 that there are sagging wrinkles. If the amount of change P is smaller than the threshold (S406: No), it is determined that there is no sagging wrinkle between Y(n+1)-Y(n), and it is determined in S408 whether scanning of the entire range has been completed. judge. If the scanning of the entire range has not been completed (S408: No), the scanning proceeds to the next n, that is, n+1, in S409, and the process returns to S404. Unless it is determined in S406 that sagging wrinkles are present, the scan line is updated until the entire range is completed, and the determination of the presence or absence of sagging wrinkles is repeated. If the entire range is completed without determining that there are sagging wrinkles (S408: Yes), it is determined in S410 that there are no sagging wrinkles.
In the manner described above, sagging wrinkles can be detected and their presence or absence can be determined.

Note that the flowchart for detecting sagging wrinkles is not limited to the method shown above. For example, a method can be considered in which the contrast ratio without sagging wrinkles is input and stored in advance, and if the difference between the contrast ratio and the contrast ratio of each Y(n) is larger than a threshold value, it is determined that sagging wrinkles are present. .

Another processing procedure for determining the presence or absence of sagging wrinkles will be described with reference to the flowchart of FIG. In the following description, the contrast ratio without sagging and wrinkles will be referred to as a normal contrast ratio.
First, in S501, a normal contrast ratio is set. The normal contrast ratio is obtained by molding various cardboard boxes and determining the value obtained through experimental verification.

Next, in S502, a contrast ratio threshold is set. The threshold value is a standard indicating how far the contrast ratio can change in the absence of sagging wrinkles. The threshold value is set by the user.
Next, in S503, pixel coordinates from the start to the end of scanning are set in the Y direction. Furthermore, in the X direction, pixel coordinates for calculating brightness values are set.
In S504, a scan is started to calculate the contrast ratio of the acquired image. Then, in S505, the contrast ratio is calculated from Y(n), that is, the start pixel coordinates set in S504.

Next, in S506, the difference between the contrast ratio Y(n) and the normal contrast ratio -Y(n) is determined, and the amount of change P thereof is calculated. Then, in S507, a comparison is made to see if the amount of change P is larger than a threshold value. Here, if the amount of change P is larger than the threshold value (S507: Yes), it is determined in S508 that there are sagging wrinkles. If the amount of change P is smaller than the threshold (S507: No), it is determined that there is no sagging wrinkle in Y(n), and it is determined in S509 whether scanning of the entire range has been completed. If the scanning of the entire range has not been completed (S509: No), the scanning proceeds to the next n, that is, n+1, in S510, and the process returns to S505. Unless it is determined in S507 that sagging wrinkles are present, the scan line is updated until the entire range is completed, and the determination of the presence or absence of sagging wrinkles is repeated. If the entire range is completed without determining that there are sagging wrinkles (S509: Yes), it is determined in S511 that there are no sagging wrinkles.
In the manner described above, sagging wrinkles can be detected and their presence or absence can be determined.

Furthermore, although the normal contrast ratio is input in advance in the above example, a method of estimating the normal contrast ratio from the acquired image may also be considered. This will be explained with reference to the flowchart in FIG.
First, in S601, a contrast ratio threshold is set. The threshold value is a standard indicating how far the contrast ratio can change in the absence of sagging wrinkles. The threshold value is set by the user.

Next, in S602, pixel coordinates from the start to the end of scanning are set in the Y direction. Furthermore, in the X direction, pixel coordinates for calculating brightness values are set.
In S603, a scan is started to calculate the contrast ratio of the acquired image. Then, in S604, the contrast ratio is calculated from Y(n), that is, the start pixel coordinates set in S602.
Next, in S605, it is determined whether n is the end. If n is not the end (S605: No), the process advances to S606 and the scan advances to the next n, that is, n+1. Contrast ratio calculations are repeated until n reaches the end.

If n has reached the end (S605: Yes), the process advances to S607, and a normal contrast ratio is estimated from the calculated contrast ratio matrix. As a means of estimating, the standard deviation is determined from the matrix of contrast ratios and is taken as the normal contrast ratio. Alternatively, as another means, there is a method of extracting only a group of contrast ratios with a small amount of change before and after from a matrix of contrast ratios using the threshold value in S601, and using the average value thereof. The processing procedure for determining sagging wrinkles using the estimated normal contrast ratio in S608 and subsequent steps is the same as the example in FIG. 5, so a description thereof will be omitted here.

As described above, according to the present embodiment, sagging wrinkles can be detected by arranging a camera on the upper side so as to photograph the attached tape and observing changes in the contrast ratio of the closed flap. The presence or absence of sagging wrinkles can be determined.

Information related to the above detection results and determination results, such as the presence or absence of sagging wrinkles and the location of sagging wrinkles, will be notified to the user via the user interface, and will also be stored in a storage device or stored externally. Send it to the box making machine through the interface. To notify the user, processing such as displaying on a display device connected to the image processing device or issuing a voice message may be performed.

(Inspection for presence or absence of pasting position shift)
Next, the procedure for detecting the presence or absence of positional deviation in the position where the tape is applied will be explained.
The tape edge contrast ratio detection unit 109 of the image processing device 102 extracts the tape edge portion from the image acquired by the image acquisition unit 105 and performs a contrast ratio calculation process. The tape end contrast ratio detection unit 109 accesses the threshold management unit 106, obtains an allowable threshold value for the distance between the flap closing eye and the tape end, and compares whether the calculated amount of change in the distance value is within the allowable threshold value. Then, it is determined whether there is any misalignment of the pasting position. Furthermore, the tape end position in the acquired image is estimated from the calculated contrast ratio. The internal processing of the flap closed eye contrast ratio detection section 108 and the tape edge contrast ratio detection section 109 will be described later.

Below, an outline of an algorithm for detecting the presence or absence of misalignment of the tape pasting position will be explained. For convenience of explanation, a case where the position where the tape is applied deviates from the original setting position by more than an allowable range is referred to as misaligned application.

FIG. 7 is a diagram schematically showing a photographed image taken from above of a state in which the tape 202 is pasted on the closed eye 203 of the two closed flaps 201 with a misaligned pasting. One of the two flaps 201 may be called a first member and the other may be called a second member, and the first member and the second member may be integral or separate. If no misalignment occurs, the distance between the flap closure 203 and the tape end 701, which is the end of the tape 202 in the width direction, will always be within the allowable range. However, when the tape is applied obliquely to the closed eye as shown in FIG. 7, the distance between the closed eye and the end of the tape changes depending on the location. The difference (Wa-Wb) between Wa at the position 702 and Wb at the position 703 corresponds to the amount of deviation.

In this embodiment, the positions of the closed eye 203 and the tape end 701 are calculated using the contrast ratio, and the difference between Wa and Wb (Wa-Wb), which is the distance between the closed eye and the tape, is determined. The contrast ratio can be calculated according to the procedure already described, and the positional information X(n) of the closed eye 203 and the positional information X(n+u) of the tape end at Y(n) can be obtained. Then, calculate Wa, which is the distance between the closed stitch and the end of the tape, using X(n+u)-X(n), and use the same method to calculate the distance between the closed stitch and the tape edge in the X direction for all Y(n) from the start to the end. By calculating , it is possible to detect misaligned pasting.

FIG. 8 shows a flowchart of a process for determining the presence or absence of misaligned pasting. That is, the process calculates the Max and Min distances between the closed eye and the end of the tape in the Y direction, and if the difference between Max and Min is larger than a predetermined threshold, it is determined that the tape is misaligned.

First, in S801, a threshold value for the difference between the Max and Min distances between the closed eye and the tape end is set. The threshold value is a standard representing the permissible range of skewing of the tape with respect to the closed flap. The threshold value is set in advance by the user.
Next, in S802, pixel coordinates from the start to the end of scanning in the Y direction are set. In addition, pixel coordinates for calculating brightness values in the X direction are set.
Next, in S803, calculation of contrast ratio is started for the acquired image. In S804, the contrast ratio of Y(n) is calculated from the start pixel coordinates set in S802.

Next, in S805, positional information X(n) of the closed eye 203 and tape end positional information X(n+u) in Y(n) are determined from the calculation result of the contrast ratio in Y(n).
Then, in S806, the difference between the position information X(n) of the closed eye 203 and the position information X(n+u) of the tape end is calculated to determine the distance between the closed eye and the tape end.
In S807, it is determined whether n is the end, that is, whether scanning in the Y direction has been completed.
If n is not the end (S807: No), the process advances to S808, and the scan advances to the next n, that is, n+1. The difference between the position information X(n) of the closed eye 203 and the position information X(n+u) of the tape end is repeatedly calculated until n reaches the end. When the scanning in the Y direction is completed (S807: Yes), the process advances to S809, and the Max (maximum value) and Min (minimum value) of the distance between the closed eye and the tape end are extracted. Then, in S810, the difference between Max and Min distances between the closed eye and the tape end is calculated.

In S811, the difference between the Max and Min distances between the closed eye and the tape end is compared with an allowable threshold. If it is larger than the threshold (S811: Yes), the shift is large, so the process proceeds to S812, and it is determined that there is a shift. If it is smaller than the threshold (S811: No), the deviation is within the allowable range, so it is determined in S813 that there is no deviation pasting.

As described above, according to the present embodiment, by observing changes in the contrast ratio between the flap closure and the tape edge using the camera placed on the upper side, and determining the respective positions, it is possible to detect whether or not there is misaligned pasting. can be determined.

Note that when detecting the above-mentioned misalignment, the contrast ratio was calculated to detect the boundary (tape end) between the flap and the tape, but this detection result can also be applied to detecting sagging wrinkles. In other words, the upper surface of the tape is located at a higher position than the flap at the sag wrinkled portion, so that it is close to the camera placed above the flap. Therefore, as shown in FIG. 2, the width of the sagging wrinkle 204 is photographed with the tape wider than the normally pasted part. That is, the images are taken so that the boundary positions between the tape and the flap are separated on both the left and right sides. Utilizing this, a threshold value that takes into account the width increase due to sagging wrinkles is set, and by comparing it with the distance between the left and right ends of the tape, sagging wrinkles can be detected.

Information related to the above detection results and judgment results, that is, the presence or absence of misalignment or the presence or absence of sagging wrinkles, is notified to the user via the user interface, and is also stored in a storage device or displayed on an external interface. to the box making machine. To notify the user, processing such as displaying on a display device connected to the image processing device or issuing a voice message may be performed.

[Robot system]
FIG. 12 is a schematic diagram showing an example of a robot system that combines the image processing device of the embodiment described above and an assembly robot.
The assembly robot includes a first robot arm 310, a second robot arm 320, an inspection table 300, and a control computer (not shown). A cardboard holding section 311 for holding a cardboard material W, which is a material for a cardboard box, is mounted on the tip of the first robot arm 310. A robot hand 321 is attached to the tip of the second robot arm 320 . The robot hand 321 includes fingers 323 and has a taping unit 322 mounted thereon. The present robot system can also be said to be a manufacturing device or a manufacturing system that manufactures cardboard boxes as articles.

The fingers 323 of the second robot arm 320 are used to fold and assemble the cardboard material W held by the first robot arm 310. The taping unit 322 is used, for example, to apply tape along the closing seam of the flap. The first robot arm 310 cooperates with the second robot arm 320 to assemble the cardboard box, and then places the assembled cardboard box 330 on the inspection table 300.

A lighting device 110 and a vision 101 are installed in the robot system so that the taped portion of the cardboard box 330 placed on the inspection table 300 can be appropriately illuminated and photographed. The vision 101, the lighting device 110, and the image processing device 102 are communicably connected by wire or wirelessly so that control signals and image signals can be exchanged.

The image processing device 102 performs the processing already described, and inspects and determines the presence or absence of pasting defects such as tape wrinkles, sagging wrinkles, and misaligned pasting positions. Information related to the determination result is notified to the user via the user interface, and is also stored in a storage device or transmitted to the robot control unit via an external interface. To notify the user, processing such as displaying on a display device connected to the image processing device or issuing a voice message may be performed.

In the above robot system, the inspection for attachment defects was performed by placing the assembled cardboard box 330 on the inspection table 300, but the inspection was carried out while the cardboard box 330 was held by the first robot arm 310. You may go. In that case, instead of the vision 101, a robot camera for assembly work may be used to photograph the tape pasting section, and the image data may be transmitted from the robot control section to the image processing device 102 to perform image processing. The robot camera for assembly work may be installed above the assembly work space, or may be mounted on the second robot arm 320 in some cases.

[Other embodiments]
Note that the present invention is not limited to the embodiments described above, and many modifications can be made within the technical idea of the present invention.
The present invention is suitably implemented for use in detecting the quality of tape attachment for boxes made of cardboard, but the material of the box is not limited to cardboard. The present invention can be implemented, for example, in combination with a box-making machine equipped with a dedicated flap folding jig or a robot hand, and the mechanism of the tape pasting machine is not limited either. Furthermore, the object for detecting the quality of the tape attachment is not limited to boxes, but can be widely applied to inspecting the tape attachment portion of objects that have been assembled or sealed using tape.
The device to be combined with the image processing device of the embodiment is not limited to a dual-arm robot, but may be a robot of another type, a device exclusively for folding flaps, or the like.

The present invention provides a process in which a program that implements one or more functions of the embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. But it is possible. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101...Vision/102...Image processing device/103...Paste failure detection unit/104...External IF/105...Image acquisition unit/106...Threshold value management unit/107...・Brightness change detection section / 108 ... Flap closed eye contrast ratio detection section / 109 ... Tape edge contrast ratio detection section / 110 ... Illumination device / 201 ... Flap / 202 ... Tape / 203. ...Closed eye/204...Sagging wrinkles/205...Closed eye part visible through the tape/206...Closed eye part/207...Part pasted without sagging wrinkles

Claims (14)

Obtaining an image that includes the tape attached to the closed portion of the first member and the second member,
Calculating the contrast ratio of each part of the image,
detecting the state of the tape attached using the contrast ratio of the tape attached to the closed eye;
An image processing method characterized by: The brightness value at the raised sagging wrinkles of the tape applied to the closed eye is higher than the brightness value at the part where the tape is applied without the sagging wrinkles .
The image processing method according to claim 1, characterized in that: Using the contrast ratio of each part of the image including the contrast ratio of the closed eye part, the position information of the closed eye, the position information of the boundary between the first member and the tape, and the second member and the tape are determined. Find the position information of the boundary of
detecting a positional deviation of the tape from a predetermined pasting position based on the positional information;
The image processing method according to claim 1, characterized in that: Furthermore, differential processing is performed on the luminance data of the image along a one-dimensional direction ,
Detects irregularly protruding mountain range tape wrinkles in plan view and side view.
The image processing method according to claim 2 or 3, characterized in that: A control program for a computer to execute the image processing method according to claim 1. A computer readable recording medium having recorded thereon the control program according to claim 5. an image acquisition unit that acquires an image that includes the tape attached to the closed portion of the first member and the second member;
a contrast ratio detection unit that calculates a contrast ratio of each part of the image acquired by the image acquisition unit;
an attachment detection unit that detects a state of the attached tape using the contrast ratio of the tape attached to the closed eye calculated by the contrast ratio detection unit;
An image processing device comprising: Furthermore, it includes a threshold value management unit that stores a predetermined threshold value,
The pasting detection section compares the contrast ratio of the closed eye portion calculated by the contrast ratio detection section with a predetermined threshold stored in the threshold management section, and calculates the height over the entire width of the tape in the width direction. detecting sagging wrinkles in which the tape is raised so that the tape forms a substantially constant ridge;
The image processing apparatus according to claim 7, characterized in that: The pasting detection unit uses the contrast ratio of each part of the image, including the contrast ratio of the closed eye part calculated by the contrast ratio detection unit, to determine the position information of the closed eye, the first member, and the first member. obtaining positional information of a boundary between the tapes and positional information of a boundary between the second member and the tape;
8. The image processing apparatus according to claim 7, wherein a positional deviation of the tape from a predetermined pasting position is detected based on the positional information. Furthermore, the pasting detection unit performs differential processing on the luminance data of the image along a one-dimensional direction ,
Detects irregularly protruding mountain range tape wrinkles in plan view and side view.
The image processing apparatus according to claim 8 or 9. A robot that attaches tape to the closed portion of the first member and the second member;
an image acquisition unit that acquires an image captured to include the tape attached to the closed portion of the first member and the second member;
a contrast ratio detection unit that calculates a contrast ratio of each part of the image acquired by the image acquisition unit;
an attachment detection unit that detects a state of the attached tape using the contrast ratio of the tape attached to the closed eye calculated by the contrast ratio detection unit;
A robot system comprising: an imaging unit;
an image acquisition unit that acquires an image captured from the imaging unit including the tape attached to the closed portion of the first member and the second member;
a contrast ratio detection unit that calculates a contrast ratio of each part of the image acquired by the image acquisition unit;
an attachment detection unit that detects a state of the attached tape using the contrast ratio of the tape attached to the closed eye calculated by the contrast ratio detection unit;
An imaging device comprising: A robot that attaches tape to the closed portion of the first member and the second member;
an image acquisition unit that acquires an image captured to include the tape attached to the closed portion of the first member and the second member;
a contrast ratio detection unit that calculates a contrast ratio of each part of the image acquired by the image acquisition unit;
A method for controlling a robot system, comprising: a pasting detection unit that detects a state of the tape pasted using the contrast ratio of the tape pasted to the closed eye calculated by the contrast ratio detection unit, ,
a step of applying tape to the closed portion of the first member and the second member by the robot;
an image acquisition step of acquiring an image captured by the image acquisition unit so as to include the tape attached to the closed portion of the first member and the second member;
a contrast ratio detection step in which the contrast ratio detection unit calculates a contrast ratio of each part of the image acquired by the image acquisition unit;
A pasting detection step in which the pasting detection unit detects a state of the pasted tape using the contrast ratio of the closed eye portion calculated by the contrast ratio detection unit;
A method for controlling a robot system, comprising: A robot that attaches tape to the closed portion of the first member and the second member;
an image acquisition unit that acquires an image captured to include the tape attached to the closed portion of the first member and the second member;
a contrast ratio detection unit that calculates a contrast ratio of each part of the image acquired by the image acquisition unit;
Manufacturing an article using a robot system comprising: a pasting detection unit that detects a state of the tape pasted using the contrast ratio of the tape pasted to the closed eye calculated by the contrast ratio detection unit A method,
a step of applying tape to the closed portion of the first member and the second member by the robot;
an image acquisition step of acquiring an image captured by the image acquisition unit so as to include the tape attached to the closed portion of the first member and the second member;
a contrast ratio detection step in which the contrast ratio detection unit calculates a contrast ratio of each part of the image acquired by the image acquisition unit;
A pasting detection step in which the pasting detection unit detects a state of the pasted tape using the contrast ratio of the closed eye portion calculated by the contrast ratio detection unit;
A method for manufacturing an article characterized by having the following.

Images