JP7472737B2 - Chain inspection device, chain inspection method, and program - Google Patents

Description

The present disclosure relates to a chain inspection device, a chain inspection method , and a program.

Previously, when inspecting chains in factories, etc., the equipment had to be stopped and the chain's elongation was measured manually, and a visual check was made to see if there was any damage or rust, uneven wear on the rollers, or if the pins were rotating.

In response to this, in order to avoid long-term shutdowns of equipment and the burden of removing the chain, a method has been proposed for inspecting the chain by photographing the chain with a camera and analyzing the captured image (Patent Document 1). The method proposed in Patent Document 1 detects chain elongation by estimating the distance between adjacent pins of the chain using image analysis.

JP 2019-132668 A

However, conventional technology is unable to detect pin rotation caused by excessive load on the chain, and there is a need to detect pin rotation in order to detect damage or deterioration of the chain or pins at an early stage.

In addition, when performing image analysis on images captured in a factory or other location, it is desirable to capture the images under appropriate lighting.

In view of the above problems, the objective of this disclosure is to provide a technology for detecting the rotation state of a chain pin using image analysis.

In order to solve the above problems, one aspect of the present disclosure is characterized by having an image acquisition unit that acquires an image of a chain in which plates are connected by a rivet pin having a convex area with a straight edge formed at its head, a pin state detection unit that detects a state regarding the rotation of the rivet pin by detecting the inclination angle of the straight edge with respect to the longitudinal direction of the plates based on the image acquired by the image acquisition unit, and a lighting control unit that controls lighting that illuminates the rivet pin so that the brightness difference or brightness ratio between an area of reduced brightness adjacent to the straight edge and the convex area in the image is equal to or greater than a predetermined threshold .

According to this disclosure, the rotation state of the chain pins can be detected using image analysis.

FIG. 1 is a schematic diagram illustrating a chain inspection device according to one embodiment of the present disclosure. FIG. 13 is a diagram showing a chain capture image according to an embodiment of the present disclosure. 1 is a block diagram showing a hardware configuration of a chain inspection device according to an embodiment of the present disclosure. 1 is a block diagram showing a functional configuration of a chain inspection device according to an embodiment of the present disclosure. FIG. 13 illustrates a rotation angle according to one embodiment of the present disclosure. FIG. 2 illustrates an arcuate section according to one embodiment of the present disclosure. FIG. 1 illustrates a pin template according to one embodiment of the present disclosure. FIG. 13 is a diagram illustrating a binarization process according to an embodiment of the present disclosure. FIG. 1 illustrates a masking process according to an embodiment of the present disclosure. FIG. 13 illustrates a boundary extraction process according to an embodiment of the present disclosure. FIG. 13 illustrates a boundary extraction process according to an embodiment of the present disclosure. 11 is a flowchart illustrating a chain inspection process according to an embodiment of the present disclosure.

In the following embodiment, a chain inspection device is disclosed that uses image analysis to detect the rotation state of pins in a chain.
[Summary of the Disclosure]
1, a chain inspection device 100 controls the illumination of a lighting fixture 50 and captures an image of a pin of a chain 20 under the illumination from the lighting fixture 50. The chain inspection device 100 performs image analysis on the image of the pin and estimates a state related to rotation, such as a rotation angle, of the pin.

When capturing an image of the chain 20, depending on the lighting conditions at the location where the chain 20 is placed, it may not be possible to recognize the boundary between the square convex area of the head of the four-sided rivet pin and its adjacent area or the outline of the pin from the captured image, as shown in FIG. 2(a). Typically, the head of a four-sided rivet pin is circular, and a square convex area is formed on part of its surface. When light hits the head of the four-sided rivet pin, a shadow, i.e., an area of reduced brightness, is created on the head other than the convex area. According to the present disclosure, the chain inspection device 100 controls the illumination of the illumination device 50 so that the illumination from the illumination device 50 causes the adjacent area of the convex area to cast a shadow of a predetermined shape (e.g., an arch shape), and an image can be obtained in which the positional relationship between the convex area and the adjacent area is clear, as shown in FIG. 2(b).

In this way, it is possible to estimate the rotation angle of the pin with high accuracy by image analysis using an image that clearly shows the convex area of the head of the four-sided rivet pin.
[Hardware configuration]
Here, the chain inspection device 100 may have a hardware configuration as shown in Fig. 2. That is, the chain inspection device 100 has a CPU (Central Processing Unit), a memory device 102, an auxiliary storage device 103, an imaging device 104, a communication device 105, and an operation means 106, which are interconnected via a bus B.

Various computer programs including programs or instructions for implementing various functions and processes in the chain inspection device 100 described below are installed in the auxiliary storage device 103, and the memory device 102 reads and stores the programs and data from the auxiliary storage device 103 when an instruction to start the chain inspection device 100 is received. The auxiliary storage device 103 and the memory device 102 are realized as non-transient computer-readable storage media that store programs or instructions. The CPU 101, which functions as a processor, executes various functions and processes of the chain inspection device 100 based on images or videos acquired from the imaging device 104 in accordance with various data such as programs stored in the memory device 102 and parameters required to execute the programs. The imaging device 104 may be a digital video camera or the like, and acquires images or videos of the imaging target. The communication device 105 transmits and receives various signals to and from external devices such as the lighting fixture 50. The operation means 106 may be an operation panel or operation buttons on the chain inspection device 100, and may be composed of a switch for turning the power of the chain inspection device 100 on and off, a lamp indicating the power on/off state, a setting button, a display panel, etc.

However, the chain inspection device 100 is not limited to the hardware configuration described above, and may be realized by any other suitable hardware configuration, such as, for example, one or more circuits that realize one or more of the functions and processes performed by the chain inspection device 100.

In addition, in the following examples, the lighting fixture 50 is described as a device independent of the chain inspection device 100, but the present disclosure is not limited to this, and the lighting fixture 50 may be mounted on the chain inspection device 100.

In addition, in the illustrated embodiment, there is one lighting fixture 50, but the present disclosure is not limited to this, and multiple lighting fixtures 50 arranged at different positions may be used, making it possible to illuminate the chain 20 from various positions and at various angles.
[Chain inspection device]
Next, a chain inspection device 100 according to an embodiment of the present disclosure will be described with reference to Figures 4 to 11. Figure 4 is a block diagram showing the functional configuration of the chain inspection device 100 according to an embodiment of the present disclosure. As shown in Figure 4, the chain inspection device 100 has an illumination control unit 110, an imaging unit 120, and a pin state detection unit 130.

The lighting control unit 110 controls the lighting that is irradiated onto the chain 20. Specifically, the lighting control unit 110 controls the lighting state irradiated from the lighting device 50. For example, if the pin attached to the chain 20 to be inspected is a four-sided rivet pin as shown in FIG. 5, the lighting control unit 110 controls the illumination of the lighting device 50 so that the four arcuate portions surrounding the square convex area of the head of the four-sided rivet pin as shown in FIG. 5(b) can be clearly imaged with their boundary lines as shown in FIG. 5(a).

Here, the rotation angle of the pin may be defined as the angle between the longitudinal direction of the chain and the straight edge corresponding to the outer edge of the convex region of the pin, as shown in FIG. 5(b). As shown, if the angle between the longitudinal direction of the chain and the straight edge of the convex region of the pin is a design angle at the time of manufacturing the chain, such as 90 degrees, the rotation angle of the pin may be determined to be normal, and if the angle deviates from the design angle by more than a predetermined threshold value, the rotation angle of the pin may be determined to be abnormal.

Specifically, the lighting control unit 110 controls the illumination of the lighting device 50 so that the brightness of the arcuate portion in the image of the four-sided rivet pin captured by the imaging unit 120 deviates from the brightness of the area surrounding the arcuate portion by a threshold value or more.

For example, the lighting control unit 110 may control the lighting that illuminates the four-sided rivet pin so that a region of reduced brightness appears in the captured image. For example, if the lighting is too bright and there is no difference between the brightness of the arched portion and the brightness of the adjacent region of the arched portion that is greater than or equal to the threshold, the lighting control unit 110 may instruct the lighting device 50 to reduce the amount of light. On the other hand, if the lighting is too dim and there is no difference between the brightness of the arched portion and the brightness of the adjacent region of the arched portion that is greater than or equal to the threshold, the lighting control unit 110 may instruct the lighting device 50 to increase the amount of light. Alternatively, if there is no difference between the brightness of a portion of the four arched portions and the brightness of the adjacent region that is greater than or equal to the threshold, the lighting control unit 110 may increase or decrease the amount of light of the lighting device 50 that has an illumination direction toward the arched portion.

The pin according to the present disclosure is not limited to a four-sided rivet pin having a head as shown in FIG. 6(a) above, but may be a two-sided rivet pin as shown in FIG. 6(b). In the two-sided rivet pin, two arcuate portions are formed as shown. The chain 20 shown in FIG. 6(a) has a plate 21, a four-sided rivet pin 22, a circular head 23, a square convex region 24 in the head 23, a straight edge 25 which is the outer edge of the convex region 24, and an arcuate portion 26 formed between the plate 21 and the convex region 24. Similarly, the chain 20' shown in FIG. 6(b) has a plate 21', a two-sided rivet pin 22', a circular head 23', a convex region 24' in the head 23', a straight edge 25' which is the outer edge of the convex region 24', and an arcuate portion 26' formed between the plate 21' and the convex region 24'. The shaded portion according to the present disclosure is not limited to an arcuate shape, and other shapes may be used depending on the type of pin.

When infrared (IR) lighting is used, the brightness of the arched portion and the adjacent area may not differ significantly. Even in this case, if the outline between the arched portion and the adjacent area is clear, the arched portion may be treated as a pseudo shadow. Visible light lighting and infrared lighting may be used depending on the shooting environment of the chain 20 and the degree of oil stains. In general, visible light lighting may be reflected by oil. The lighting fixture 50 does not need to be one equipped for imaging purposes, and may be lighting in a factory.

The imaging unit 120 acquires an image including a rivet pin having a convex area with a straight edge formed at the head. That is, the imaging unit 120 images the rivet pin of the chain 20 under the illumination from the lighting fixture 50 controlled by the lighting control unit 110, and generates an image or video of the rivet pin. The imaging unit 120 is realized by an imaging device such as a digital camera or a video camera. For example, the imaging unit 120 images the chain 20 in operation by fixed point shooting, and outputs the imaging result as a video. When inspecting a specific pin, the pin moves in the moving direction of the chain 20 in the captured image frame. Therefore, in order to track the pin, the imaging unit 120 may control the imaging conditions according to the speed of the chain 20. For example, the imaging unit 120 may change camera parameters such as the frame rate and shutter speed of the video according to the speed of the chain 20 detected by the pin state detection unit 130. Specifically, when the speed of the chain 20 is fast, the imaging unit 120 may increase the frame rate and shutter speed. This makes it possible to capture images of the chain 20 and rivet pins without blurring them.

The pin state detection unit 130 analyzes the area of reduced brightness adjacent to the straight edge in the captured image acquired by the image acquisition unit 120 to detect the state of the rotation of the rivet pin. For example, the pin state detection unit 130 may perform template matching between a pin template imitating the shape of the rivet pin to be inspected and an image of the rivet pin based on the arcuate part of reduced brightness adjacent to the straight edge of the convex region of the rivet pin in the captured image, and determine the state of the rotation of the rivet pin. Specifically, the pin state detection unit 130 may hold a pin template as shown in FIG. 7(a) corresponding to a four-sided rivet pin. When an image of a pin as shown in FIG. 7(b) is acquired from the imaging unit 120, the pin state detection unit 130 can search for a pin template that matches or is close to the shape of the captured pin and determine that the captured pin is a four-sided rivet pin. When it is determined that the captured pin is a four-sided rivet pin, the pin state detection unit 130 estimates the rotation angle from the image of the four-sided rivet pin based on the shape of the convex area shown in the pin template, and is able to determine whether the estimated rotation angle deviates from the design angle by a predetermined threshold or more. In this way, the pin state detection unit 130 may estimate the rotation angle based on type information such as a pin template corresponding to the type of pin.

The pin state detection unit 130 may estimate the state related to rotation, such as the rotation angle, by a method other than template matching. For example, the pin state detection unit 130 may perform edge detection to detect a circumscribing rectangle or a contour line, or may estimate the positional relationship between the convex area of the pin and the plate of the chain 20 by a machine learning model, a deep learning model, pattern matching, or the like, and determine the state related to rotation, such as the rotation angle. For example, a neural network (e.g., a convolutional neural network, etc.) may be used that determines whether the rotation angle of the pin deviates from the design angle by a threshold value or more from the image of the pin. Such a neural network may be trained using training data of the image of the pin and a value indicating whether the rotation angle of the pin deviates from the design angle by a threshold value or more. For example, by including various types of pins, dirty pins, pins with various rotation angles, etc. in the training data, it becomes possible to estimate the rotation angle of the pin in a versatile and flexible manner. In addition, transfer learning or the like may be performed on the trained neural network and used for identification. The image of the pin for training may be either an image actually taken or an image artificially generated.

When a rotation state is detected, the pin state detection unit 130 may notify or record the detected rotation state together with an image of the pin to an operator of the chain inspection device 100, etc. Furthermore, if an identifier is assigned to the pin, the identifier of the pin indicating an abnormality may be notified. This allows the operator to quickly identify which pin of the chain 20 is abnormal.

In one embodiment, the pin state detection unit 130 may extract features from the image of the pin and estimate the rotation angle based on the features. For example, the pin state detection unit 130 may convert the image of the chain 20 into any of the known features such as ORB, SIFT, AKAZE, HOG, LBP, and SURF, and detect the rotation state such as the rotation angle of the pin based on the features. According to this embodiment, it is possible to detect the rotation angle of the pin without being affected by differences in the color (brightness) of the chain surface or differences in the size of the captured chain. In addition, the captured image may be expressed by features different from the brightness of the image such as ORB, SIFT, AKAZE, HOG, LBP, and SURF, and the rotation angle of the pin may be estimated by matching the features or by judgment using a classifier. In particular, a feature that can well express the features of the shape of the pin may be selected. Simple edge detection or chamfer matching may also be used.

In one embodiment, the pin state detection unit 130 may perform a filter process or a binarization process on the image of the pin to emphasize the shadow formed between the chain and the pin. For example, the pin state detection unit 130 may perform a spatial filtering process such as smoothing on the image of the pin to narrow the range of brightness changes in the image and emphasize the shadow on the pin surface. Furthermore, the pin state detection unit 130 may perform a spatial binarization process (e.g., binarization of pixel values, expansion process, contraction process, etc.) as shown in FIG. 8 to further emphasize the shadow on the pin surface and detect the rotation angle of the pin with high accuracy.

In one embodiment, the pin state detection unit 130 may apply a mask image to the image of the pin to extract the shadow formed between the chain and the pin. For example, mask images such as those shown in FIGS. 9(a) to (c) may be used. The pin state detection unit 130 may apply a mask image to the image of the pin to remove image areas that are unnecessary for detecting the rotation state of the pin or that may have a negative effect. According to this embodiment, by superimposing the mask image on the image of the pin, it becomes possible to effectively extract the shadow of the arched portion of the pin.

In one embodiment, the pin state detection unit 130 may perform contour extraction on the image of the pin to detect the boundary between the chain 20 and the pin. Specifically, the pin state detection unit 130 may apply a known detection method such as Hough transform, regression analysis, or chain code to perform straight line extraction of each edge of the top of the pin, as shown in FIG. 10. The pin state detection unit 130 can detect the rotation angle of the pin by detecting the angle between the extracted straight line and the longitudinal direction of the chain 20. Here, the pin state detection unit 130 may discard the detected straight line edges with low reliability, as shown in FIG. 11, and detect the rotation angle of the pin from the straight line edges with reliability equal to or higher than a threshold. For example, the reliability may be determined according to the degree of deviation between the detected straight line edges and the pin template. For example, in the illustrated example, the straight line edge on the left side may be determined to have low reliability and discarded.

In one embodiment, the pin state detection unit 130 may estimate the rotational state of the pin, such as the rotation angle, based on multiple image frames of the image of the pin. When the chain 20 is captured as an image, the pin moves as the chain 20 operates. At this time, the same pin is captured across multiple image frames. For this reason, the average rotation angle may be calculated from multiple detection results. A time series filter such as a Kalman filter may also be applied. The difference between two frames or optical flow may also be calculated to detect the rotation angle. This makes it possible to estimate the rotation angle from more information.

In one embodiment, the pin state detection unit 130 may estimate the size of the chain 20 from the image of the pin, and the imaging unit 120 may control the optical system based on the estimated size of the chain 20. For example, if the captured chain size is small, the imaging unit 120 may perform optical zoom or digital zoom to enlarge the image until the chain 20 is captured at a size suitable for image analysis. On the other hand, if the captured chain size is large, the imaging unit 120 may perform optical zoom or digital zoom to reduce the image until the chain 20 is captured at a size suitable for image analysis. For example, when the chain inspection device 100 is installed in a factory, the space in the factory is limited, so it is not always possible to install the chain inspection device 100 in a location where it is possible to capture an image at a size suitable for image analysis. For this reason, it is preferable to have the above-mentioned automatic zoom function.

In one embodiment, the pin state detection unit 130 may estimate the size of the chain 20 from the image of the pin and resize the template image used for image analysis. In general, the image portion of the pin in the pin template may be different in size from the image portion of the captured pin. To improve the accuracy of template matching, the pin state detection unit 130 may resize the pin template to match the size of the captured chain.

In one embodiment, the pin state detection unit 130 may perform image recognition on the image of the pin online or offline. For example, if the pin state detection unit 130 has enough processing power to process the image of the pin in real time, the pin state detection unit 130 may perform the rotation angle of the pin online. On the other hand, if the pin state detection unit 130 does not have enough processing power to process the image of the pin in real time, or if real-time processing is not necessarily required, the pin state detection unit 130 may temporarily store the captured image in a buffer and detect the rotation angle of the pin from the image stored in the buffer at an appropriate time.

In one embodiment, the pin state detection unit 130 may also perform thinning processing on the image frames of the image of the pin and estimate the rotation angle of the pin from the extracted image frames. Detecting the rotation angle for all image frames of the image may be inefficient and may not be necessary. For this reason, the rotation angle may be detected for only some of the image frames.

In one embodiment, the pin state detection unit 130 may estimate the position of the pin based on the design data of the chain 20, and estimate the rotation angle of the pin based on the image portion around the estimated position. For example, if the type of pin has been determined, the pin state detection unit 130 may use the design data of the pin to estimate the approximate position of the pin, and estimate whether the rotation angle of the pin around the estimated position deviates from the design angle. This can improve the efficiency of the detection process and the detection accuracy.

In one embodiment, the pin state detection unit 130 may perform image analysis of the changing portion in the video using a relatively high resolution image. For example, the captured image may include image parts that are not related to the inspection, such as the background. When inspecting the chain 20 and the pins, the image parts of the chain 20 surrounding the moving pin are the target, and the image parts may be treated as high resolution images, while the other image parts may be treated as low resolution images.
[Chain inspection process]
Next, a chain inspection process according to an embodiment of the present disclosure will be described with reference to Fig. 12. The chain inspection process is realized by the above-mentioned chain inspection device 100, and may be realized, for example, by a processor of the chain inspection device 100 executing a program or instructions. Fig. 12 is a flowchart showing the chain inspection process according to an embodiment of the present disclosure.

As shown in FIG. 12, in step S101, the chain inspection device 100 controls the illumination state. Specifically, the chain inspection device 100 controls the illumination of the lighting fixture 50 that illuminates the rivet pin so that the area of reduced brightness appears as a predetermined shape. For example, the illumination of the lighting fixture 50 may be controlled so that the convex area of the head of the rivet pin and the adjacent arched portion via the straight edge have a brightness that deviates from the convex area and the plate of the chain 20 by a predetermined threshold or more.

In step S102, the chain inspection device 100 captures an image of the illuminated chain 20. For example, the chain inspection device 100 may capture the chain 20 as an image or video.

In step S103, the chain inspection device 100 estimates the rotational state, such as the rotation angle of the pin, from the captured image. For example, the chain inspection device 100 may detect the angle between the straight edge of the convex area of the pin and the longitudinal direction of the chain plate from the image of the pin, and determine the detected angle as the rotation angle. Alternatively, the chain inspection device 100 may use a trained machine learning model that estimates from the image of the pin whether the rotation angle deviates by more than a predetermined threshold, input the image of the pin to the machine learning model, and obtain a judgment result.

In step S104, the chain inspection device 100 determines whether an abnormal state of the pin has been detected. If the rotation angle of the pin deviates by more than a predetermined threshold, the pin is determined to be in an abnormal state (S104: YES), and the chain inspection device 100 notifies an operator or the like of the abnormal state. At this time, an image of the abnormal pin may also be sent. On the other hand, if there is no abnormality in the pin (S104: NO), the chain inspection device 100 ends the chain inspection process.

Although the examples of the present invention have been described in detail above, the present invention is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims.

The invention as originally claimed in the present application is set forth below.
[Additional Notes]
In one aspect of the present disclosure,
an image acquisition unit for acquiring an image including a rivet pin having a convex region with a straight edge formed on a head portion thereof;
a pin state detection unit that detects a state related to the rotation of the rivet pin by analyzing a region of reduced brightness adjacent to the straight edge in the captured image acquired by the image acquisition unit;
A chain inspection device is provided having the following features:

In one embodiment, the pin state detection unit may analyze the boundary of the area of reduced brightness to detect the state of the rotation of the rivet pin.

In one embodiment, the device may have a lighting control unit that controls lighting that illuminates the rivet pin so that the area of reduced brightness appears.

In one embodiment, the lighting control unit may control the lighting so that the area of reduced brightness has a predetermined shape.

In one embodiment, the rivet pin may be a four-sided rivet pin or a two-sided rivet pin.

In one embodiment, the pin state detection unit may use a trained machine learning model to estimate the state of the rivet pin's rotation.

In one embodiment, the pin state detection unit may estimate the state of the rotation of the rivet pin based on type information corresponding to the type of the pin.

In another aspect of the present disclosure,
Computer,
an image acquisition unit for acquiring an image including a rivet pin having a convex region with a straight edge formed on a head portion thereof;
a pin state detection unit that detects a state related to the rotation of the rivet pin by analyzing a region of reduced brightness adjacent to the straight edge in the captured image acquired by the image acquisition unit;
A program is provided to function as such.

20 Chain 50 Lighting fixture 100 Chain inspection device 110 Lighting control unit 120 Imaging unit 130 Pin state detection unit

Claims (5)

an image acquisition unit for acquiring an image of a chain in which plates are connected by rivet pins having a convex region with a straight edge formed at a head portion thereof;
a pin state detection unit that detects a state related to rotation of the rivet pin by detecting an inclination angle of the straight edge portion with respect to the longitudinal direction of the plate based on the captured image acquired by the image acquisition unit;
an illumination control unit that controls illumination for illuminating the rivet pin so that a luminance difference or luminance ratio between a luminance-reduced region adjacent to the straight edge portion and the convex region in the captured image is equal to or greater than a predetermined threshold;
A chain inspection device having a The illumination control unit controls the illumination so that the area of reduced luminance has a predetermined shape.
2. The chain inspection device according to claim 1. The rivet pin is a four-sided rivet pin or a two-sided rivet pin.
3. The chain inspection device according to claim 1 or 2. A chain inspection method executed by a chain inspection device, comprising:
an image acquisition process for acquiring an image of a chain in which plates are connected by rivet pins having a convex region with a straight edge formed at a head portion thereof;
a pin state detection process for detecting a state related to the rotation of the rivet pin by detecting an inclination angle of the straight edge portion with respect to the longitudinal direction of the plate based on the captured image acquired in the image acquisition process;
an illumination control process for controlling illumination for illuminating the rivet pin so that a luminance difference or luminance ratio between a luminance reduction region adjacent to the straight edge portion and the convex region in the captured image is equal to or greater than a predetermined threshold value;
A chain inspection method comprising: Computer,
an image capturing means for capturing an image of a chain in which plates are connected by rivet pins having convex areas with straight edges formed at their heads;
a pin state detection means for detecting a state related to rotation of the rivet pin by detecting an inclination angle of the straight edge portion with respect to a longitudinal direction of the plate based on the captured image acquired by the image acquisition means;
an illumination control means for controlling illumination for illuminating the rivet pin so that a luminance difference or luminance ratio between a luminance-reduced region adjacent to the straight edge portion in the captured image and the convex region becomes equal to or greater than a predetermined threshold value;
A program to function as a

Images