JP6032812B2 - Inspection assistance device - Google Patents

Description

  The present invention is particularly suitable for an inspection of a building by an image (for example, determining a defect from an appearance image of a communication base station tower or a bridge taken with a camera) of a specific part (for example, a bolt). The present invention relates to an inspection auxiliary device that calculates a degree of abnormality numerically.

  The steel tower is a long and narrow structure composed of a metal frame structure. It is used for base stations such as mobile phones, transmission of broadcast waves, support of power transmission lines and antennas, and gazebos for fire fighting. Although the height ranges from several meters to several tens of meters, it is often set at a high place with a good view from the purpose of use.

  As soon as the new tower is built, it is checked to see if it was built correctly as designed. In addition, after that, at regular intervals, a regular inspection is performed to check whether the current steel tower has a structural strength that can be used.

  In the inspection, abnormalities caused by aging such as warpage, bending, cracks, rust deterioration such as rust and bolt loosening are found. In addition, defects such as reverse insertion of bolts, insufficient tightening torque, and the use of bolts of an appropriate length, such as defects that are not being constructed as designed at the time of construction, and deposits such as bird's nest and vinyl are also a kind of abnormality. Is considered.

  For example, Patent Documents 1 and 2 disclose conventional techniques related to an inspection and inspection method using a bolt as a clue.

  FIG. 11 is a diagram for explaining the method described in Patent Document 1. Here, the bolt looseness inspection is realized by the following procedure. Mark for looseness detection on both sides of the bolt fixing place in advance when there is no looseness. For example, as shown in (1), marking is performed with a cross. When the bolts loosen, as shown in (2), the continuity of the crosses between the parts on the frame of the bolts, nuts, washers and towers is impaired, so by detecting the marking discontinuity from the image, Check for loose bolts.

  FIG. 12 is a diagram for explaining the screw part inspection method described in Patent Document 2. Here, the presence or absence of a nut is detected from an image on the premise that the camera C can be photographed at a fixed angle and position such as a factory production line. To do. If the nut N is properly fixed to the object M as shown in (1), a wavy uneven groove is observed from the camera C, but it is not observed unless it is properly fixed as shown in (2). Is used.

  On the other hand, in order to avoid the risk of inspection work at high places such as falling objects and fall accidents, the safety of the surroundings without climbing from the conventional inspection method that inspectors actually climb the steel tower on site In recent years, a method of taking a video of a steel tower from a certain place and visually inspecting the steel tower image at a safe place has begun in recent years.

  As a typical example of shooting, a video camera is installed on the ground, and the steel frame in the steel tower is shot everywhere from the bottom to the top of the steel pillar. At this time, in order to record all sides of the tower without omission, the video camera is installed at several places around the tower. In addition, when photographing the joints between the frames in the steel tower, the zooming is performed until the looseness can be determined by focusing on the bolt. As a result of shooting in this way, the video recording time is often relatively long per tower.

  In the method of visually inspecting a steel tower image taken by video, the time for visual confirmation is basically longer than the recording time. This is because the suspicious part needs to be confirmed many times by pausing or rewinding. The inspector in charge of visual confirmation is forced to make a great effort.

  Therefore, focusing on buildings such as steel towers, focusing on the fact that steel tower failures often occur around bolted joints, improving the efficiency of visual inspection of video images of steel towers taken as a clue to bolted joints. An apparatus that performs this process has been described in Patent Document 3 so far.

  FIG. 13 is a diagram for explaining the method described in Patent Document 3, and is an inspection auxiliary device that assists the inspection of the inspection target parts arranged in the building from the inspection image conceptually shown in (1), Learn the characteristics of the part to be inspected from multiple sample images taken from the part to be inspected as represented by (2), and the partial area where the part to be inspected exists from the inspection image from which the building was taken. (3) is detected as shown in the example, and based on the detected partial area, the importance level indicating the necessity of inspecting the location in the inspection image in the building is calculated, and (4) (5 The efficiency of visual inspection is improved by presenting the degree of importance to the user as shown in FIG.

  In the present invention, the time required for visual confirmation is shortened and the labor of the inspector is reduced by setting the importance so as to preferentially present the bolt joint as a partial area considered to be particularly high in need of inspection. It was done. Specifically, as shown in the example of the inspection screen in (6), the inspection video is played back while displaying the playback position on the time series graph of importance. Intuitive and easy to understand.

Japanese Patent Laid-Open No. 2005-3658 “Bolt Looseness Checking Method” Japanese Patent Laid-Open No. 2007-10620 “Screw Part Inspection Device and Thread Part Inspection Method” Japanese Patent Application No. 2011-285612 “Auxiliary Inspection Device and Method” (prior application by the present inventor)

  However, as described below, since there are various problems in the related art related to the inspection or inspection, it is desired to realize each item [1] to [6].

  The bolt looseness inspection method described in Patent Document 1 has a problem in that it requires costly operations such as marking. Since it is necessary to manually mark at high places, [1] costly work like this marking work can be avoided, and [2] a safe and efficient inspection method is desired.

  In addition, the bolts that appear in the image of the steel tower can be seen in various directions. On the other hand, the thread inspection method described in Patent Document 2 is based on the premise that photography can be performed at a fixed angle and arrangement such as a production line in a factory, and thus cannot be applied to various viewing directions. There is a demand for an inspection method that can cope with a variety of [3] orientations in which the bolts photographed in the image can be seen without being restricted by the restriction of requiring photographing with a fixedly arranged camera.

  Furthermore, the inspection assisting device and method described in Patent Document 3 can preferentially present bolt joints that are considered to have a particularly high need for inspection, and reduce the time required for visual inspection to reduce the labor of the inspector. However, the final normal / abnormal judgment of the bolted joints presented was still required to be visually judged by the inspector. In general, a large number of bolts are used for joining steel towers, but the burden on the inspector is somewhat reduced, but still large. [4] An inspection method that can further reduce labor is desired.

  On the other hand, since it was a visual judgment regarding the final normal / abnormal judgment, there is a possibility that the inspection standards may vary among inspectors, and there is a possibility that the inspection quality may vary due to fatigue of the inspection work. In order to avoid fluctuations and to make the quality of inspection always constant, there is a need for an inspection method that can determine normal and abnormal bolt joints based on [5] objectively unified standards.

  Similarly, the final normal / abnormal determination of the bolted joints must be performed by experienced inspectors. Ultimately, [6] automatic inspection methods that can be carried out even in the absence of skilled inspectors are desired.

  In view of the above points, the present invention has the following objects.

  With respect to the above items [1] to [4], the present invention is intended to partially improve the exterior of the building to include parts to be inspected that are typically placed on the exterior of the building as represented by bolts in steel towers. Inspection target parts that are considered to be particularly highly necessary for visual inspection by automatically calculating the degree of abnormality in the installation state of the inspection target parts for users who perform visual inspections based on multiple inspection images taken The first object is to provide an inspection assisting device that reduces the labor required for visual inspection by preferentially presenting.

  In addition, regarding the above items [5] and [6], the present invention reduces the labor required for visual inspection by automatically determining the normality of the installation state of the inspection target component, and at the same time the installation state of the inspection target component. The second purpose is to provide an inspection assisting device that can judge normality and abnormalities of an object by objectively unified standards.

  Furthermore, a third object of the present invention is to make the inspection auxiliary device applicable even when the inspection target part is an assembly bolt.

  In order to achieve the above object, the present invention is an inspection auxiliary device for assisting inspection of an inspection target component arranged in a building, and extracts an outline of the inspection target component from an inspection image obtained by photographing the building. A first feature is that the apparatus includes a contour extracting unit and a component abnormality degree calculating unit that calculates an abnormality degree related to an installation state of the inspection target part based on the extracted outline of the inspection target part.

  Further, according to the present invention, the contour extracting unit detects an edge from the inspection image, and a geometric shape fitting unit that applies a predetermined geometric model to the inspection target part to the detected edge; The contour is extracted as the fitted geometric model, and the part to be inspected is an assembly bolt including a washer, a nut, and a bolt as elements, and the geometric fitting part is provided on the washer and the nut. An ellipse is fitted, and the component abnormality degree calculation unit is based on a deviation between a center axis of the assembly bolt obtained from the ellipse fitted to the nut and a center of the ellipse fitted to the washer. The second feature is calculating the degree of abnormality.

  According to the first feature, the degree of abnormality is automatically calculated based on the extracted contour, and the first object is achieved. The degree of abnormality itself is obtained as a numerical value and can be compared with a predetermined determination threshold if normality / abnormality determination is necessary, so the second object is also achieved.

  According to the second feature, an ellipse is applied to each element of the assembly bolt, and the degree of abnormality is calculated based on a deviation between predetermined ellipses corresponding to a deviation from the central axis. The objective is achieved.

It is a figure for demonstrating an example of the imaging | photography procedure of the inspection video of a steel tower. It is a functional block diagram of the inspection auxiliary device concerning one embodiment of the present invention. It is a figure for demonstrating an inspection image | video. It is a figure which shows the example of component detection. It is a figure explaining the distinction of normal and abnormal of bolted joint by the existence of deviation. It is a figure for demonstrating the principle of abnormality degree calculation. It is a figure for demonstrating the detail of the process of the edge detection part 31 and a geometric shape fitting part. It is a flowchart of the process of the geometric shape fitting part which concerns on one Embodiment. It is a figure for demonstrating the process of the geometric shape fitting part which concerns on one Embodiment. It is a figure explaining the usage example of a memo provision part. It is a figure for demonstrating a prior art. It is a figure for demonstrating a prior art. It is a figure for demonstrating a prior art.

  FIG. 1 is a diagram for explaining an example of a procedure for photographing an inspection video of a tower to be analyzed in the present invention. (A) is an example of a part of the steel tower T1 as viewed from the front, and there are bolts (not shown) for fixing members such as pillars (framework etc.) constituting the steel tower T1 constructed according to various structures. , Become the main inspection object. In addition, various problems that may exist in an arbitrary location are also inspected.

  An example of the photographing procedure will be described with reference to (B). The worker W1 continuously scans the front P1 from the bottom to the top as indicated by the arrow (1) from the position L1 facing the front P1 of the steel tower T1 (here, the member structure is not shown) using the camera C10. Then shoot. The scanning photographing may be performed so that the entire width portion of the front surface P1 is within the image, or may be performed for each pillar of the steel tower existing on the front surface P1.

  Next, the worker W1 moves to a position L2 facing the right side surface P2, and scans the right side surface P2 similarly as indicated by an arrow (2) to continuously take pictures. Next, the worker W1 moves to the position L3 facing the back surface P3, and continuously shoots the back surface P3 (not shown), and further moves to the position L4 facing the left side surface P4, and also continuously shoots the left side surface P4 ( (Not shown). Note that (B) is an example. The moving order of L1 to L4 may be arbitrary, and the image may be taken from any position other than L1 to L4. Each of the surfaces P1 to P4 may not be photographed individually. For example, both the front surface P1 and the right surface P2 may be photographed at a time at a location L5 near the middle of L2 of the location L1. The steel tower T1 may be circular instead of square. The continuous scanning imaging may not be linear as shown in (1) or (2), but may be zigzag while following the member structure as shown in (A). You may shoot in any direction or order.

  When shifting from the shooting of the front P1 to the shooting of the right side P2, the shooting may be temporarily stopped, or an arbitrary place including a place other than the tower T1 may be taken without stopping. The camera C10 may be held by the operator W1 by hand, or may be fixed to a tripod or the like.

  As described above, there is no particular restriction on the procedure for photographing the inspection image of the steel tower, and the appearance of the steel tower may be scanned substantially continuously or intermittently while appropriately changing the photographing position. However, it is preferable that the worker W1 reduces the scanning speed or pauses when photographing a point requiring inspection (particularly, a bolted portion in a steel tower) and enlarges the image centering on the point requiring inspection. To do. Further, similar images may be taken with a remotely operated camera or the like.

  Since the photography of the steel tower in the present invention has such a high degree of freedom, it was photographed instead of having a relatively small burden of photography and no need to perform any additional work (marking etc.) on the steel tower itself. The inspection video includes a bolt to be inspected and a video section other than the bolt at random and in a considerable proportion. This is because, in general, when photographing the inside of a steel tower, images are taken to every corner while overlapping slightly slightly so as not to leak. If the video is used for visual inspection by the inspector as it is, an excessive burden on the inspector is generated. However, as will be described below, the burden is remarkably reduced by the present invention.

  In FIG. 1, the inspection video has been described by taking a bolt connection portion as an example of a steel tower and an inspection point thereof, but the present invention is not limited to other general buildings and inspection points (predetermined types arranged in the building). This is applicable to inspection images taken in the same manner with respect to the inspection target parts.

  FIG. 2 is a functional block diagram of an inspection assistant device according to an embodiment of the present invention, and FIG. 3 is a diagram for explaining inspection images processed by the inspection assistant device.

  The inspection auxiliary device 1 includes, as main components, a sampling unit 10, a component detection unit 21, a feature learning unit 22, a contour extraction unit 30, a component abnormality degree calculation unit 40, an abnormality determination unit 50, an important section determination unit 60, and synchronized playback. Unit 70, control unit 80, encoding unit 90, and frame capture unit 100. The contour extracting unit 30 includes an edge detecting unit 31 and a geometric shape fitting unit 32, and the control unit 80 is a cueing unit 81, an important section extracting unit 82, a speed changing unit 83, a memo assigning unit 84, a correcting unit 85, and a capture specifying unit. Including 86. An inspection image prepared by the method described with reference to FIG. 1 and conceptually illustrated in (1) of FIG. 3 is input to the sampling unit 10.

  The sampling unit 10 captures the inspection video in a frame memory at a sampling rate set in advance as a user setting or the like, sequentially extracts it as a still image frame, and passes it to the component detection unit 21, the synchronous reproduction unit 70, and the encoding unit 90. For example, the sampling rate is set to 1 fps (1 frame per second). Each of the still image frames as shown in (2) of FIG. 3 is referred to as an inspection image with the meaning of an image extracted from the inspection video. The sampling unit 10 can also grasp the order relationship between the inspection images by associating information indicating what number frame the inspection image is on the time series of the original inspection video.

  The component detection unit 21 uses a learning result obtained by the feature learning unit 22 described later, so that the sampling unit 10 continuously scans the appearance of the building at a predetermined sampling rate. In each inspection image sequentially read, a partial region including a bolt as an inspection target part is detected, and the result is passed to the contour extraction unit 30 (the edge detection unit 31 included).

  FIG. 4 shows an example of the component detection. That is, a partial region including a bolt is detected from the inspection image shown in (1) as shown by being surrounded by a rectangle in (2). In this example, four partial areas are detected from the inspection image. However, the application of the component detection unit 21 to each inspection video is applied to determine whether or not the partial areas are detected. It depends on the results.

  The present invention is not limited to bolts, but can handle other various parts to be inspected. Hereinafter, a bolt is used as a representative example.

  When the component detection unit 21 performs the detection, the shape of the partial area is detected as a rectangle or a circle. In the case of a rectangular area, it is uniquely determined by 4 parameters of x coordinate, y coordinate, width and height of the center position, and in the case of a circular area, it is uniquely determined by 3 parameters of x, y coordinate and radius r of the center position. The detection result of the partial area can be handled as very simple information. The example of (2) in FIG. 4 is an example of detection as a rectangular area, and will be described below as handling a rectangular area. However, a circular area or other forms can be similarly implemented.

  The feature learning unit 22 learns a predetermined type of classifier in advance using a plurality of sample images so that the bolt can be identified. The component detection unit 21 detects the above-described partial region corresponding to the bolt by using the learned discriminator. An existing face detection technique can be used as the discriminator for the bolt detection. For example, a case where a face detection device represented by Non-Patent Document 1 is used will be described below.

  [Non-Patent Document 1] Paul Viola, Michael Jones, "Robust Real-time Object Detection", Second International Workshop on Statistical and Computational Theories of Vision-Modeling, Learning, Computing, and Sampling, 2001.

  In Non-Patent Document 1, an image including a large amount of faces is prepared in advance (a predetermined number is prepared), the luminance signal in the area including the face is converted into a Haar-like feature amount, and the black eye is the surrounding white eye portion. The luminance signal common to face images, such as the luminance signal is small compared to the eyebrows and the cheek luminance signal is high relative to the eyebrows, is learned by a machine learning method called Adaboost, and the judgment rule common to face images is Generate the described Adaboost classifier.

  When detecting a face in a newly input image, a specific rectangle is moved in the raster scan order from left to right and top to bottom, and the luminance signal in each destination rectangle is Haar-like feature Whether or not a person's face is included is determined based on whether or not the determination rule described in the Adaboost classifier generated in advance is satisfied.

  In the present invention, the detection target is not a face but a bolt, a large number of bolt images are prepared in advance as sample images, a luminance signal in a rectangular region including the bolt is learned, and a discriminator is generated. The detection unit 21 detects a partial region including the bolt from the inspection image. In addition to the above-mentioned Adaboost, an SVM (support vector machine) or the like may be adopted as the discriminator.

  The feature learning unit 22 learns the bolt characteristics according to the direction of the bolt and creates a discriminator for each direction. The component detection unit 21 detects the inspection target component in the inspection image using the discriminator for each direction. By doing so, the processing accuracy of the geometric shape fitting part 32 described later can be improved. In this case, the component detection unit 21 notifies the contour extraction unit 30 of the partial area including the detected component and the direction at the time of detection.

  A predetermined type of bolt direction is set in advance. For example, in addition to the vertical two types, the horizontal direction two left and right types, and the diagonal four types in addition to the front, there may be about nine types. In this way, the component detection unit 21 detects the shape of the partial area including the bolt in the inspection image as a rectangle or a circle, and at the same time, identifies the direction according to the type of the detected classifier. That is, in the case of a rectangular area, in addition to the four parameters of the center position x-coordinate, y-coordinate, width and height, in addition to the bolt direction, in the case of a circular area, three parameters of the center position x-, y-coordinate and radius r In addition, the direction of the bolt is determined and notified to the contour extracting unit 30.

  Next, the description will proceed to the contour extracting unit 30 and the component abnormality degree calculating unit 40, and after explaining the outline and principle, details of the processing of each unit will be described. In the present invention, the portions 30 and 40 detect a deviation between the center of gravity of the washer and the center of gravity of the nut for a bolt as an example of a component, and calculate an abnormality degree related to loosening of the bolt joint. If it is assumed that the position of the nut is fixed with respect to the bolt, the shift is equal to the shift between the center of gravity of the washer and the center of gravity of the bolt.

  FIG. 5 is a diagram for explaining the distinction between normal and abnormal bolt joints depending on the presence or absence of the deviation. Both (1) and (2) are examples viewed from the front. (1) is a normal example, and the bolt B, the nut N, and the washer W are normally arranged without being displaced from each other. (2) is an abnormal example, in which the washer W is displaced in the direction of gravity G with respect to the bolt B and the nut N. The distinction between normal and abnormal is quantified by the degree of abnormality.

  FIG. 6 is a diagram for explaining the principle of calculating the degree of abnormality in the present invention. The exploded view of (1) shows the bolt (bolt B) as the assembly in the image (photographed image) of the partial area of the bolt detected by the component detection unit 21 in the inspection image as in the explanation column (C). And a nut N and a washer W, showing the positional relationship of each element of the steel tower framework F).

  That is, the principle of calculating the degree of abnormality is based on the following relationship. As shown in the exploded view of (1) and as described in the explanation column (B), if the bolt (bolt as an assembly with nuts etc.) is fixed normally, the washer W The center of gravity of both the bolt B and the nut N coincide with each other, and each center of gravity is on the central axis A0 of the bolt B.

  However, when there is looseness, the center of gravity of the washer W is shifted below the center of gravity of the bolt B and nut N due to the influence of gravity. Therefore, by measuring this deviation amount, it is possible to calculate the degree of looseness considered to be particularly important among the abnormality of the bolt joint as the degree of abnormality.

  In FIG. 6, the center of gravity of the washer W (considered as a flat surface) is W0, the center of gravity of the upper surface side of the nut is N20, and the center of gravity of the lower surface side of the nut N (the side in contact with the washer W) is N10. The center of gravity of the bolt B is not shown as being coincident with the center of gravity of the nut N. Also, the reference numerals in FIG. 6 are commonly used in the description with reference to FIGS.

  The outline of the abnormality degree calculation process defined as described above is as follows. That is, by extracting the contour of (2) shown in parallel with the exploded view of (1) of FIG. 6, each center of gravity and the central axis A0 are determined by the contour. The extracted contours are ellipses N2 and N1 extracted by ellipse fitting on the hexagonal upper and lower surfaces of nut N shown in (2), and ellipse extracted by ellipse fitting on the washer W circle as well. W1. As shown in the column (A), the outline of the extraction processing is to estimate an ellipse to be applied after detecting an edge. In this manner, as shown in the column (C), the processing shown in the columns (A) and (B) is performed on the captured image.

  The contour extraction unit 30 includes an edge detection unit 31 and a geometric shape fitting unit 32, which will be described later.In the inspection image obtained by photographing the building as illustrated as a rectangle in (2) of FIG. The contour of the inspection target part (washer, nut, bolt) is specified from the partial region including the inspection target part detected in this way, and is passed to the subsequent part abnormality degree calculation unit 40.

  FIG. 7 is a diagram for explaining the details of the processing of the edge detection unit 31 and the geometric shape fitting unit 32 separately from (1) to (6). (1) in FIG. 7 is an example of a bolt detected in the lower right direction by the component detection unit 21, and the description will be made with the bolt as an example of a processing target.

  The edge detection unit 31 detects an edge from a partial region including the inspection target component detected by the component detection unit 21 in the inspection image obtained by photographing the building, and passes the detection result to the geometric shape fitting unit 32. . For the detection, a Laplacian filter, which is a known technique, or a Canny filter described in Non-Patent Document 2 below can be used. In this way, from (1) in FIG. 7, edge pixel groups that are candidates for the contours of the inspection target components (washers, nuts, bolts) illustrated in (2) are extracted.

  [Non-Patent Document 2] A computational approach to Edge detection, John Canny, 1986 IEEE. Mathematical Formulation of Performance Criterion.

  From the detected edge, the geometric fitting unit 32 has a predetermined contour used for calculating the degree of abnormality in the part abnormality degree calculation unit 40 in the subsequent stage in the inspection target part (washer, nut, bolt). Extraction is performed by applying a geometric shape, and the result is passed to the component abnormality degree calculation unit 40.

  For example, a known technique (Hough transform) described in Non-Patent Document 3 below can be used for fitting the geometric shape. If the component to be inspected is a bolt, as described in FIG. 6 as an outline, as an embodiment, the predetermined contour can use the hexagons on the upper and lower surfaces of the nut N and the circle of the washer W. For the geometric shape, an ellipse can be used for each contour.

  Therefore, by voting Hough transform in the parameter space that specifies the ellipse, the location where the edge pixel group is arranged in a circle or close to it in the edge image including the contour candidate as shown in (2) of FIG. 7 is estimated. The contour of the inspection target component (washer, nut, bolt) can be specified as illustrated in (3) of FIG.

  [Non-Patent Document 3] "R.O. Duda, and P.E. Hart," Use of the Hough Transformation to Detect Lines and Curves in Pictures, "Comm. ACM, Vol. 15, pp. 11-15 (January, 1972)"

  In (3) of FIG. 7, the contour W1 of the washer W, the contour N1 of the bottom surface of the nut N, that is, the back side (the side in contact with the washer W), and the contour N2 of the front side of the nut N, that is, the top surface N2 in total. An ellipse is extracted.

  If an ellipse is simply applied, the outline of the bolt tip or the thread portion of the bolt may be extracted as an ellipse from the image. Therefore, three are identified by using the direction of the component identified by the component detection unit 21. That is, in the case of the bolt detected as the lower right direction as in the example of FIG. 7, three ellipses are arranged in order from the upper left to the lower right direction in the inspection image in the opposite direction, respectively, the washer contour W and the nut back Specify the contour N1 on the side (the side in contact with the washer) and the contour N2 before the nut. At this time, since the size of the assumed contour can be estimated by taking into account the size of the partial region, it is possible to avoid erroneously extracting an ellipse caused by noise. become.

  FIG. 8 is a flowchart according to an embodiment when the geometric fitting unit 32 performs the fitting as shown in (3) of FIG. 7 on the basis of the above-described purpose, and FIG. 9 shows each step of the flowchart. It is a figure for demonstrating.

  In step S1, an edge at a high edge density is deleted from the edges in the edge image obtained by the edge detection unit 31. What is necessary is just to compare with the predetermined threshold value regarding a density, in order to determine with a high density. In (1) of FIG. 9, an example of the portion R having a high edge density is shown as a screw portion of a bolt, and such a portion is not a portion detected as a predetermined contour, but a voting for Hough transform Since it is inappropriate to participate in, delete in advance.

  In step S2, as shown in (2) of FIG. 9, the contour W1 of the washer is detected by voting for Hough transform. At this time, the size of the ellipse is approximately the size of the entire edge image, that is, the size of the partial area detected by the component detection unit 21, and / or the orientation of the ellipse is the component detection unit 21. It may be imposed as a restriction on the parameter space for voting that the direction is in accordance with the direction detected in (1). Since it appears as the largest ellipse in the contour W1 edge distribution of the washer, the W1 is obtained by the most votes.

  In step S3, in order to detect the remaining contours N1 and N2, an edge corresponding to the already detected contour W1 (for example, an edge within a predetermined distance from W1) is deleted. At this time, the main part of the remaining edge distribution is in a state configured as each side of the hexagonal column of the nut N. In (2) of FIG. 9, the vertices of the hexagonal column of the nut N are shown as points P1 to P9.

  In step S4, in order to delete the portion corresponding to the side of the nut N (and other noise-induced portions) from the edge distribution and leave the points P1 to P9, it is formed as the end point of the line segment from the remaining edge. Delete anything other than what is there. A predetermined threshold value based on the size of the ellipse W1 or the size of the edge image may be used for the length determined as the line segment.

  In step S5, ellipses N1 and N2 are detected by the Hough transform voting for the remaining edges P1 to P9. At this time, the ellipse N2 where the six points P1 to P6 exist is first obtained by voting, and then the ellipse N1 is obtained by moving the ellipse N2 to fit the three points P7 to P9. Good. In the voting of the ellipse N2, the same restriction on the parameter space as the ellipse W1 and / or the relative arrangement of the ellipse N2 with respect to the ellipse W1 imposed from the ellipse W1 and the direction of the bolt (N2 is W1 May be imposed on the parameter space based on the constraints of The ellipse N2 may be voted after the three points P7 to P9 corresponding to the ellipse N1 are specified and excluded by the above relative arrangement relationship with respect to the nine points P1 to P9 and the ellipse W1.

  When the component detection unit 21 detects the direction of the component as “front”, only the six points P1 to P6 remain, and only the ellipse N2 is obtained. Ellipse N1 does not exist. In this case, the ellipses N2 and W1 are close to circles in shape.

  The part abnormality degree calculation unit 40 measures a deviation amount of each center of gravity between inspection target parts (washers, nuts, bolts) as a part abnormality degree. Specifically, using the center of gravity of the geometric shape of the geometric fitting portion 32 as the center of gravity of each corresponding element, the distance between the center of gravity of the washer in the inspection image and the center of gravity of the nut (which can be regarded as one with the bolt) is calculated. Judged as the degree of component abnormality.

  That is, when the bolt is normally fixed as illustrated in (4) of FIG. 7, the center of gravity of the both is the same between the washer and the bolt (or nut), so the component abnormality degree is zero. On the other hand, as shown in (5), when the bolt joint is loose, it is affected by gravity due to the loosening, and the vertical distance between the center of gravity of the washer and the center of gravity of the nut (which can be regarded as one with the bolt) is separated. Therefore, the degree of component abnormality shows a large value.

  In this case, in detail, as shown in (4) and (5) of FIG. 7, a straight line A0 connecting the center of gravity N10 on the back side of the nut (the side in contact with the washer) and the center of gravity N20 in front of the nut is obtained. The vertical deviation width D of the washer's center of gravity W0 from the straight line A0 is measured. Here, the centroids N10, N20, and W0 are determined as the centers of the ellipses N1, N2, and W1, respectively.

  In addition, the straight line A0 is the central axis of the bolt B and the nut N that can be regarded as one body, as described with reference to FIG. In addition, since the inspection video input to the inspection auxiliary device 1 is usually obtained from a camera arranged substantially horizontally, the vertical direction of the image may be the vertical direction.

  However, the deviation width differs depending on whether the diameters of the bolts and washers used are large or small. For example, when the bolt is completely loose and the washer is shifted to the lowest point, the shift width is smaller when the washer diameter is smaller than when it is larger. Further, when the same bolted portion is photographed from different angles, the magnitude of the deviation width measured on the image varies. Therefore, it is preferable to normalize the deviation width independent of the diameter of the bolt and washer and the shooting angle.

As shown in (6) of FIG. 7, the normalization realizes normalization of the shift width by dividing the shift width D in the vertical direction G by the diameter R (width in the vertical direction) of the washer. The normalized deviation width D ′ is calculated as in equation (1). The part abnormality degree calculation unit 40 obtains D ′ as the part abnormality degree of the inspection target part and outputs it to the abnormality determination part 50.
D '= D / R (1)

  In addition, when the component detection direction in the component detection unit 21 is “front”, only the ellipse N2 is obtained, and thus the straight line A0 is not obtained. In this case, the deviation width D may be obtained as the distance between the center of gravity N20 before the nut and the center of gravity W0 of the washer.

  The abnormality determination unit 50 determines that the component abnormality degree of the inspection target component calculated by the component abnormality degree calculation unit 40 satisfies the predetermined threshold condition as an abnormal component. In this way, when there are parts to be inspected (as partial areas) in the inspection image, it is possible to obtain determination information as to whether or not each part is abnormal.

  Alternatively, separately from the individual determination or in addition to the individual determination, the abnormality determination unit 50 may define the degree of abnormality with respect to the entire inspection image. When multiple inspection target parts are included in the inspection image, for example, the maximum value of the part abnormality degree calculated by the part abnormality degree calculation unit 40 or a predetermined threshold value exceeded by the part abnormality degree calculation unit 40 The total number of inspection target components determined to be abnormal may be calculated as the degree of abnormality with respect to the entire inspection target image.

  When the component detection unit 21 does not detect the component in the inspection image, the abnormality determination unit 50 may output information indicating that the component has not been detected, or the abnormality degree value may be zero. The degree of abnormality may be output by giving a value corresponding to the lowest value.

  The important section determination unit 60 identifies inspection images (frames in the inspection video) in which the degree of abnormality calculated by the abnormality determination unit 50 for the entire inspection image exceeds the specified threshold value, and is particularly continuous in the inspection video. Then, the section that exceeds the predetermined threshold is determined as an important section that is considered to be highly inspected, and the determination information is passed to the synchronous reproduction section 70 and the control section 80.

  That is, the important section determination unit 60 analyzes the time series of the degree of abnormality for each frame output from the abnormality determination unit 50 on the time series of the inspection video, and sets a section exceeding a predetermined threshold in the abnormality degree time series as an important section. decide. (4) in FIG. 3 shows an example of the time series of the abnormality degree, and (5) shows examples of a plurality of important sections determined from the abnormality time series in order as important sections S1 to S9. Has been.

  The synchronous reproduction unit 70 reproduces the inspection video and uses it for the visual inspection of the user as an inspector. Specifically, the still frame images captured by the sampling unit 10 are reproduced and displayed in time order. At this time, the ancillary degree time-series data from the abnormality determining unit 50 and / or the important section information from the important section determining unit 60 is additionally displayed on the inspection video, and various reproductions from the control unit 80 are performed. Display is performed while controlling playback based on control information and the like.

  That is, along with the screen of the inspection video, the abnormality degree time-series data from the abnormality determination unit 50 and / or the important sections from the important section determination unit 60 are supplementarily displayed in a graph in time series. By displaying the current playback position of the inspection video in synchronization on the time-series graph, the user can easily visually and intuitively grasp the scene that is highly inspected in the videotaped steel tower video. It becomes like this.

  Furthermore, when displaying an important section, if the degree of abnormality is also displayed at the same time, it is determined by the important section determination unit 60 on the abnormality degree time series graph, otherwise on the line graph indicating the reproduction position. The place corresponding to the important section is visually marked and displayed. This also facilitates visual and intuitive grasp of scenes that are highly inspected in videotaped tower images.

  FIG. 3 (6) shows an example of the inspection screen reproduced in this way. That is, the video F1 at the time of the reproduction of the inspection video reproduced along the time series, the graph G1 displayed in such a manner that the important sections are further color-coded on the time series graph of the degree of abnormality, and the like An inspection screen P10 is configured including a seek bar B1 indicating the reproduction time point on the graph G1.

  The graph G1 may employ other desired display configurations such as displaying the graph G1 not on a different position from the video F1 but on the edge of the video F1. Abnormality time series information may be omitted from the graph G1, and the graph may have a simple configuration only indicating whether or not the current playback position is an important section. Further, the information of the important section may be omitted from the graph G1, and the graph may be configured only by the abnormality degree time series.

  The control unit 80 generates a reproduction control signal according to the operation of the inspector, that is, a user input, and transmits it to the synchronous reproduction unit 70 to control the reproduction so that the inspector efficiently inspects a desired position in the inspection video. It can be so. Typical controls to reduce the burden on the inspector are, for example, presenting only the time zone when the area around the bolt that is considered to be particularly susceptible to abnormalities in the tower is shown, and other abnormalities are generated. For example, fast-forwarding or skipping may be used during a time zone when a difficult part is photographed.

  The above control can be realized by manual operation of the inspector. At this time, as an input interface, it is possible to use a keyboard, a mouse, or the like provided in the test assistant device 1 included in the control unit 80 (not shown in FIG. 2) and configured as a computer. Using this interface, the bar displaying the current position on the seek bar can be dragged and dropped to be reproduced at an arbitrary position.

  Whether or not the current playback position should be inspected during manual control is determined by referring to information on the degree of abnormality and / or important section on the inspection screen as shown in (6) of Fig. 3. can do. The seek bar B1 in (6) may also be used for playback position control. In addition to controlling the playback position, the playback speed (including zero speed, that is, pause), the playback direction (forward playback or backward playback), and the like can be controlled by manual input from the user.

  The cue unit 81 stores in advance the start point and end point of each important section, and controls to skip the current playback position from the start point or end point to the point designated by the user, Control the ejection. When the user wants to confirm a certain important section again, the work can be smoothly performed by using the cue portion 81.

  For the user designation, a method of directly designating a desired location on the GUI (graphical user interface) from both ends of the important section added to the graph G1 in (6) of FIG. 3 can be adopted. . When skipping to the end point, the user can also specify reverse playback.

  Further, automatic reproduction control that does not require sequential user operation is also possible. The important section extraction unit 82 performs control so that only important sections are sequentially extracted from the inspection video and played back. In this way, by reproducing only the important section and not reproducing the other sections, the inspection can be performed in a shorter reproduction time than the original inspection video.

  Further, the speed changing unit 83 performs control so that the important section is played back at the normal speed, but skip playback is performed at a speed faster than usual by thinning out the frames at a constant interval in the other sections. Since the control by the speed changing unit 83 can be visually recognized up to the section determined not to have a high level of inspection compared to the control by the important section extracting unit 82, it is possible to confirm the validity of the determination by the important section determining unit 60. There is.

  When using the important section extracting unit 82 or the speed changing unit 83, the user may first specify that each unit is to be used. In this case, manual operations such as pausing and changing the playback position are added. You may make it add in.

  Furthermore, in addition to the control of the playback position / speed, it is also possible to control the user's editing or the like in the playback display or the like. The memo assigning unit 84 receives information such as a work memo input by the user and displays it as accompanying information on an inspection screen as shown in (6) of FIG. For example, whether each important section has been visually inspected, which part of the tower corresponds to each important section, information that the result of visual inspection was normal, information that identifies a known defect, or inspection other than the important section Information indicating that a location to be found has been found is appended to a desired location on the graph of (6) (particularly the position on the time axis to which the memo should be added). The user's work efficiency is improved by the accompanying notation. The memo information can be composed of text, a symbol such as a check mark, or any other desired format or combination thereof.

  FIG. 10 shows an example of a user's memo input via the memo assigning unit 84. That is, examples of user-input memo information M1 to M4 are shown on the abnormality time series graph in which important sections are added. Memo M1 that the bolt was found in the non-critical section between the critical sections S1 and S2, Memo M2 that the critical section S4 corresponds to the pillar A of the steel tower, the visual inspection results were normal, Memo M3 indicates that the critical section S6 has found a bolt drop corresponding to the pillar B, and Memo M4 indicates that the critical section S9 corresponds to the pillar C and has a crack at a predetermined position on the time axis. ing. In FIG. 6, the portion of the inspection video as shown as F1 in (6) of FIG. 3 is omitted, and the user replaces the graph G1 in (6) with the display of FIG. A screen is provided.

  The correction unit 85 corrects movement, expansion, reduction, addition, deletion, etc. of the important section according to the manual input from the user, and the inspection under the control by the control unit 80 described above for the corrected important section. Enables video playback. Also in the correction, the user may directly drag the end of the important section in the graph shown in (6) of FIG. The user can work efficiently using, for example, the cue unit 81 after correcting the important section and other necessary portions by the correcting unit 85.

  The capture specification unit 86 specifies that the user captures the current screen being reproduced (including the paused state) as a still image by the synchronized playback unit 70 under control using the other units of the control unit 80. Accept from the user. Further, when receiving the capture designation from the user, the capture designation unit 86 causes the frame capture unit 100 to generate a still image of the current screen as encoded data such as JPEG.

  At this time, the frame capture unit 9 may generate the entire inspection screen indicated as P10 in (6) of FIG. 3 as a still image according to the designation from the user, or only the inspection image indicated as F1 is a still image. The user-specified still image data is received from the synchronous reproduction unit 7 and encoded.

  For example, when a user who is an inspector creates a report of a steel tower inspection, the capture designation unit 86 is used to designate a desired location where a defect exists, thereby capturing captured image data such as a malfunction location to be reported. The defect can be easily obtained from the frame capture unit 90, and the defect can be explained in text by pasting a photograph of the defect part. Similarly, other related businesses can be made more efficient. Note that the interface of the capture specification unit 86 can also be configured as a GUI for buttons for specifying the capture on the inspection screen in (6) of Fig. 3, and the user can input the capture specification while grasping the current screen. Can do.

  The encoding unit 90 generates encoded data of a moving image from a plurality of still image frames. Here, MPEG-2, MPEG-4, H.264, etc. which are well-known video encoding methods disclosed in Non-Patent Documents 4 to 6 and the like can be used.

[Non-Patent Document 4] ISO / IEC 13818-2: 2000, Information technology-Coding of moving pictures and associated audio for digital storage media at up to about 1,5 Mbit / s-Video
[Non-Patent Document 5] ISO / IEC 14496-2: 2004, Information technology-Coding of audio-visual objects‐ Part 2: Visual
[Non-Patent Document 6] ITU-T H.264, Telecommunication Standardization Sector of ITU, Series H: Audio Visual and Multimedia Systems, Infrastructure of audiovisual services-Coding of moving video, Advanced video coding for generic audiovisual services

  A still image frame captured by the sampling unit 10 is sent to the encoding unit 90. At the same time, the control section 80 also receives information on the start position and end position of the important section determined to have a high necessity for inspection, and generates encoded data of the inspection video to be reproduced by the reproduction section 70. At this time, it is possible to generate encoded data of a video whose playback position is manually operated using the cue unit 81 or the like, but as a preferable summary video for the inspection video input to the sampling unit 2 The encoded data can be generated from the video that is automatically reproduced by the important section extracting unit 82 or the speed changing unit 83.

  That is, under the control of the important section extraction unit 82, only the important section is encoded, and the other sections are not encoded, so that a summary video shorter than the original inspection video can be created. Also, under the control of the speed changing unit 83, the important section is encoded at the normal playback speed, but in the other sections, the summary video is skip-played at a speed faster than normal by thinning out frames at regular intervals. Can be generated. The latter summary video has an advantage that it can be confirmed up to the legitimacy of the determination in the important section determination unit 60 because it can be visually recognized up to the section determined that the necessity of inspection is not high.

  If the photographed worker uses the inspection auxiliary device 1, the encoded data of the summary video is generated in advance from the inspection video and passed to the inspector. The inspector himself uses the inspection auxiliary device 1 directly. Even if this is not possible, the inspector can take advantage of the present invention by using an ordinary decoding / playback apparatus that is commercially available.

  As described above, according to the inspection auxiliary device 1 of the present invention, inspectors play back video images of a steel tower that has been video-captured in advance, and in the inspection for screening the steel tower normally and abnormally by visual confirmation, time series By providing information on the degree of abnormality above, it is possible to reduce the inspection burden on the inspector.

  Based on the embodiment described above, various embodiments of the present invention will be described below. An embodiment in which these are appropriately combined, or a more simplified embodiment including only an arbitrary part is possible.

  In one embodiment, the inspection assistant device 1 includes only a component detection unit 21, a feature learning unit 22, a contour extraction unit 30, and a component abnormality degree calculation unit 40. In this case, the inspection auxiliary device 1 receives each inspection image corresponding to each frame of the inspection video as an input, and outputs the degree of abnormality of each component in the single inspection image. In the case of inputting an image in which the component is mainly shown (image of the partial area itself), the component detection unit 21 and the feature learning unit 22 may be omitted.

  In one embodiment, the inspection assisting apparatus 1 includes only a component detection unit 21, a feature learning unit 22, a contour extraction unit 30, a component abnormality degree calculation unit 40, and an abnormality determination unit 50. In this case, the inspection auxiliary device 1 receives each inspection image corresponding to each frame of the inspection video as an input, and the degree of abnormality of the entire single inspection image is obtained. In the case of inputting an image in which the component is mainly shown (image of the partial area itself), the component detection unit 21 and the feature learning unit 22 may be omitted.

  In one embodiment, the sampling unit 10 is omitted, and the inspection screen is reproduced and the summary video is generated with the predetermined sampling rate set in the inspection video. Alternatively, similarly, the inspection video itself may be prepared in advance at a desired sampling rate.

  In one embodiment, the important section determination unit 60 is omitted, the important section is not displayed on the examination screen and the summary video, and the control based on the presence of the important section in the control unit 80 is omitted.

  In one embodiment, the inspection screen and / or the summary video in the synchronous reproduction unit 70 and / or the encoding unit 90 is a rectangle or a circle when the component detection unit 21 detects the inspection target component as a partial region. In addition, the detected part to be inspected may be displayed so as to surround it. In this case, information on the position of the partial area is transferred from the component detection unit 21 to the synchronous reproduction unit 70 and / or the encoding unit 90 by a flow not shown as an arrow in FIG. When the display is applied, the inspection screen has a configuration in which, for example, F1 in (6) in FIG. 3 is replaced with (3).

  In one embodiment, the inspection target is not limited to a steel tower but is a general building, and the target of visual inspection is not limited to a bolt, but is a predetermined type of component (inspection target component) arranged on the exterior of the building. In this case, the inspection image is photographed by scanning the exterior of the building continuously or intermittently in the same manner as described with reference to FIG. 1, and the feature learning unit 22 constructs a discriminator for the inspection object part. The detection unit 21 uses the discriminator.

  Even when the inspection target part is a predetermined type of part other than a bolt, the contour extraction unit 30 selects a predetermined geometric model that fits the shape of the part in the same manner as when an ellipse is applied to the case of a bolt. Just apply.

  In the case of a predetermined type of component, the component abnormality degree calculation unit 40 calculates the abnormality degree based on the deviation of the washer from the central axis in the case of a bolt, for each element of the part. In the fitted geometric model, an arrangement relationship representing a normal state is defined, and the degree of abnormality is determined based on the deviation of the arrangement relationship of each geometric model on the actually detected image from the normal state relationship. It may be calculated. As an example, the normal state may be defined as a case where a certain central axis is defined as in the case of a bolt, and a predetermined point in the geometric model coincides with the central axis.

  DESCRIPTION OF SYMBOLS 1 ... Inspection auxiliary device, 10 ... Sampling part, 21 ... Component detection part, 22 ... Feature learning part, 30 ... Contour extraction part, 31 ... Edge detection part, 32 ... Geometric shape fitting part, 40 ... Part abnormality degree calculation part, 50: Abnormality determining unit, 60: Important section determining unit, 70: Synchronous playback unit, 80 ... Control unit, 81 ... Cueing unit, 82 ... Important section extracting unit, 83 ... Speed changing unit, 84 ... Memo adding unit, 85 ... correction unit, 86 ... capture designation unit 86, 90 ... encoding unit and 100 ... frame capture unit

Claims (13)

An inspection assisting device for assisting inspection of the inspection target part disposed on the object,
A contour extraction unit for extracting a contour of the inspected component from the inspection image in which the object is photographed,
A component abnormality degree calculation unit that calculates an abnormality degree related to an installation state of the inspection target part based on the extracted outline of the inspection target part ; and
In the contour extraction unit, the contour is extracted as a plurality of contours reflecting the three-dimensional structure of the inspection target component,
The component abnormality degree calculation unit calculates the abnormality degree based on a deviation from a predetermined arrangement relationship in a normal state of a plurality of contour arrangement relationships reflecting the extracted three-dimensional structure of the inspection target component. Inspection auxiliary device characterized by that. An inspection assisting device for assisting inspection of the inspection target part disposed on the object,
A contour extraction unit for extracting a contour of the inspected component from the inspection image in which the object is photographed,
A component abnormality degree calculation unit that calculates an abnormality degree related to an installation state of the inspection target part based on the extracted outline of the inspection target part ; and
The contour extraction unit
An edge detection unit for detecting an edge from the inspection image;
A geometric fitting portion for fitting a predetermined geometric model for the inspection target part to the detected edge; and
Extracting the contour as the fitted geometric model,
The part to be inspected is an assembly bolt including washers, nuts and bolts as elements,
The geometric fitting portion fits an ellipse to the washer and the nut,
The component abnormality degree calculation unit calculates the abnormality degree based on a deviation between a center axis of the assembly bolt obtained from an ellipse fitted to the nut and a center of the ellipse fitted to the washer. An inspection auxiliary device characterized by calculating . A feature learning unit that learns the characteristics of the inspection target part from a plurality of sample images obtained by photographing the inspection target part and constructs a discriminator;
A component detector that applies the classifier to the inspection image and detects a partial region in which the inspection target component exists; and
The contour extraction unit extracts the contour for each detected partial region, and the component abnormality degree calculation unit calculates an abnormality degree of a corresponding inspection target component for each detected partial region ,
The feature learning unit learns the characteristics of the inspection target part for each direction in which the inspection target part is visible, and constructs a discriminator for each direction.
The component detection unit detects a partial region where the inspection target component exists and a direction in which the component is visible,
Inspection assisting device according to claim 1 or 2, wherein the contour extraction unit and extracting the contour using the arrow that is visible, which is the detection. When the component detection unit extracts a plurality of partial regions from the inspection image, the maximum value of the degree of abnormality calculated for each inspection target component corresponding to the component abnormality degree calculation unit, or for each inspection target component The inspection auxiliary device according to claim 3 , further comprising: an abnormality determination unit that calculates the number of abnormalities calculated in the above as a degree of abnormality for the entire inspection image. The abnormality determination unit determines whether the installation state is normal or abnormal for each inspection target part depending on whether the degree of abnormality calculated for each inspection target part satisfies a predetermined threshold condition. The inspection auxiliary device according to claim 4 , wherein A sampling unit that sequentially reads inspected images taken by continuously scanning the appearance of the object at a predetermined sampling rate;
Each sequentially read at the predetermined sampling rate is adopted as the inspection image,
The abnormality determining unit inspection assisting device according to claim 4 or 5, characterized in that calculating the error probability as the time-series data along the inspection image. The inspection auxiliary device according to claim 6 , further comprising a synchronous reproduction unit that reproduces the inspection video together with a graph of the degree of abnormality as the time-series data while displaying a current reproduction position on the graph. . An important section determining unit that determines a section that satisfies a predetermined threshold condition in the degree of abnormality calculated as the time-series data as an important section, and the synchronized playback unit displays the determined important section on the graph. The inspection auxiliary device according to claim 7 , wherein the inspection auxiliary device is reproduced while being displayed. The inspection auxiliary device according to claim 8 , further comprising a control unit that controls reproduction by the synchronous reproduction unit according to the important section. A cue unit that cues the reproduction position to any one of a start position and an end position of the important section according to an input from a user;
An important section extractor that controls to play back only the important sections sequentially;
A speed changing unit that controls playback by setting the playback speed of other sections faster than the playback speed of the important section;
A memo assigning unit that clearly indicates a designated position from the user in the important section or other sections on the graph, and displays the memo information input from the user at the specified location and controls playback,
A correction unit for correcting the important section according to an input from a user;
The inspection auxiliary device according to claim 9 , comprising at least one of the following. Further comprising an encoding unit for generating encoded data of a reproduced video under the control of the important section extracting unit or the speed changing unit , or
The control unit includes a capture designating unit that accepts from a user a designation to capture the screen currently played by the synchronous playback unit as a still image, and further, the code of the still image of the screen currently being played according to the designation The inspection auxiliary device according to claim 10 , further comprising a frame capture unit that generates the digitized data . The contour extraction unit
An edge detection unit for detecting an edge from the inspection image;
A geometric fitting portion for fitting a predetermined geometric model for the inspection target part to the detected edge; and
12. The inspection auxiliary device according to claim 1, wherein the contour is extracted as the fitted geometric model. The inspection auxiliary device according to claim 1, wherein the object is a building.