JP2019121337A - Change point detection apparatus, change point detection method, and change point detection program - Google Patents

Abstract

To highly accurately detect a change point of an appearance of a control board.SOLUTION: In a change point detection apparatus 00 that detects a change point before and after repair work of a control board, a board front surface identification unit 000 identifies a contour of a board front surface from image data obtained by imaging the board front surface before the repair work to extract a board front surface image. A board front surface identification unit 010 identifies a contour of the board front surface from image data obtained by imaging a board front surface after the repair work to extract a board front surface image. A projection conversion unit 001 converts the board front surface image obtained by the board front surface identification unit 000 to a projection conversion result image. A projection conversion unit 011 converts the board front surface image obtained by the board front surface identification unit 010 to a projection conversion result image. A difference detection unit 020 segments projection conversion result images before the work and after the work obtained by the projection conversion units 001, 011 to generate change point detection result images colored on the segmented images having a difference between the image before the work and the image after the work.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for detecting a change point of a control board from a photographed image of the control board.

  Examples of a technique for detecting a change in a subject based on an image of the subject include a change detection device disclosed in Patent Documents 1 and 2, and the like.

  The detection technology apparatus of Patent Document 1 acquires a plurality of image data groups having different shooting locations and shooting times for the same subject, and performs triangulation feature points and shooting on points serving as features of each image. Understand the positional relationship of the device and create 3D point cloud data. Then, the change point of the subject is extracted by comparing the three-dimensional point group data obtained from each image data group.

  The change detection device of Patent Document 2 attaches an AR (Augmented Reality) marker to a subject, and detects a change point of the subject for each frame of an image.

JP, 2016-110210, A JP, 2015-158880, A JP 2017-39189 A

Takafumi Aoki, Koichi Ito, Takuma Sugahara, Sei Nagauta, “High-Precision Machine Vision Based on Phase-Only Correlation-Toward Image Sensing Technology Beyond Pixel Resolution-,” IEICE Fundamentals Review, Vol. 1, No. 1, pp. 30-40, July 2007.

  The device exemplified in the control panel has a box shape of a single color and lacks feature points, and parts such as levers and switches attached to the device may have light reflection at the time of shooting because they are black and glossy. False detection of change points is likely to occur. Therefore, a change point detection technique based on three-dimensional point group data as in Patent Document 1 is not suitable for detection of a change point of a subject with a poor feature point.

  On the other hand, the detection technique of Patent Document 2 detects a change point of a subject based on an AR marker attached to the subject. However, if the subject is, for example, a device owned by a customer, attaching an AR marker to the subject may not meet the customer's request. In addition, when an AR marker is attached to a device so as to be removable like a magnet sheet, the position and the orientation of the AR marker may change during maintenance of the device, which may cause a problem in the detection of the change point. Furthermore, as described above, since devices such as control panels generally have few feature points, it is difficult to extract feature points according to AR markers and detect change points. In addition, attaching a small size AR marker to a relatively large object such as a control panel results in poor detection accuracy of the change point.

  An object of the present invention is to detect a change point of the appearance of a control panel with high accuracy in view of the above-mentioned circumstances.

  Therefore, one aspect of the present invention is a change point detection device that detects a change point before and after a repair work of a control board, and from image data obtained by photographing the front of the board before and after the work of repair A panel front identification unit that specifies the contour of the panel front and extracts a panel front image, a projection conversion unit that converts the panel front image into a projection conversion result image, and a projection conversion result image before and after the operation And a difference detection unit that generates a change point detection result image in which the divided images having differences before and after the work are colored.

  In one embodiment of the present invention, in the change point detection device, the panel front surface specifying unit extracts an area specified based on a panel front surface angular coordinate specified for the image data as the panel front surface image.

  In one embodiment of the present invention, in the change point detection device, the board front surface specifying unit extracts an area specified based on a board front surface angular coordinate specified for the image data as the board front surface image. Matching template images of the four corners of the front panel are obtained from the front corner coordinates of the front panel, and the front panel image is extracted by specifying the front panel after the operation with the matching template images of the four corners of the front panel.

  In one embodiment of the present invention, in the change point detection device, the board front surface specifying unit sets a quadrangle with the largest area among the squares included in the image data as the outline of the board front surface.

  In one embodiment of the present invention, in the change point detection device, the difference detection unit detects the difference based on an RGB threshold.

  In one embodiment of the present invention, in the change point detection device, the difference detection unit detects the difference based on an HSV threshold.

  In one embodiment of the present invention, in the change point detection device, the difference detection unit performs projective transformation result image after the work by template matching of hierarchical reduced images of the projective transformation result images before and after the work. The modified point detection result image is generated by coloring the coordinate group of the dissimilar feature points of the projective transformation result image before the work detected in step b.

  In one embodiment of the present invention, in the change point detection device, the difference detection unit regards an area having a number of adjacent coordinates equal to or more than a threshold in a group of coordinates of the dissimilar feature points as a change area.

  One aspect of the present invention is a change point detection method in which a computer detects a change point before and after a repair work of a control board, and from image data obtained by photographing the front of the board before and after the work. The front surface identification process of specifying the outline of the front surface and extracting the front image of the front surface, the projection conversion process of converting the front image of the front surface into the projection conversion result image, and the projection conversion result images before and after the operation And a difference detection step of generating a change point detection result image in which the divided images having differences before and after the work are colored.

  In one embodiment of the present invention, in the change point detection method, the panel front surface specifying step extracts a region specified based on a panel front surface coordinate specified for the image data as the panel front image.

  In one embodiment of the present invention, in the change point detection method, the board front specifying step extracts a region specified based on a board front corner coordinate specified for the image data as the board front image. Matching template images of the four corners of the front panel are obtained from the front corner coordinates of the front panel, and the front panel image is extracted by specifying the front panel after the operation with the matching template images of the four corners of the front panel.

  In one aspect of the present invention, in the change point detection method, the board front surface specifying step takes a quadrangle with the largest area among the squares contained in the image data as the outline of the board front surface.

  In one embodiment of the present invention, in the change point detection method, the difference detection step detects the difference based on an RGB threshold.

  In one aspect of the present invention, in the change point detection method, the difference detection step detects the difference based on an HSV threshold.

  In one embodiment of the present invention, in the change point detection method, the difference detection step is performed after the work on the projective transformation result image by template matching of hierarchical reduced images of the projective conversion result image before the work and after the work. The modified point detection result image is generated by coloring the coordinate group of the dissimilar feature points of the projective transformation result image before the work detected in step b.

  In one embodiment of the present invention, in the change point detection method, in the difference detection step, an area in which the number of adjacent coordinates is equal to or more than a threshold in a group of dissimilar feature points is regarded as a change area.

  One aspect of the present invention is a change point detection program that causes a computer to function as the change point detection device, or a change point detection program that causes a computer to execute the change point detection method.

  According to the present invention as described above, it is possible to detect a change in the appearance of the control panel with high accuracy.

BRIEF DESCRIPTION OF THE DRAWINGS The block block diagram which showed the outline | summary of the panel change point detection system provided with the change point detection apparatus which is 1 aspect of this invention. Explanatory drawing of the operation example of the change point detection apparatus of FIG. Explanatory drawing of the image processing process at the time of using the board change point detection system of FIG. Explanatory drawing of the board front surface specific process in Embodiment 1 of this invention. FIG. 7 is an explanatory diagram of a difference detection process in the first embodiment. FIG. 6 is an explanatory diagram of a pixel comparison process in the first embodiment. Explanatory drawing of the board front specific process by board front specific part 000 of Embodiment 2 of this invention. FIG. 18 is an explanatory diagram of a panel front surface identification process by the panel front surface identification unit 010 according to the second embodiment. Explanatory drawing of the process process of the to-be-processed image in Embodiment 2. FIG. Explanatory drawing of the board front surface identification process in Embodiment 3. FIG. Explanatory drawing of the process process of the to-be-processed image in Embodiment 3. FIG. Explanatory drawing of the extraction process of the panel front surface image based on the panel front surface angle coordinate in Embodiment 3. FIG. Explanatory drawing of the difference detection process in Embodiment 4. FIG. Explanatory drawing of the pixel comparison process in Embodiment 4. FIG. Explanatory drawing of the difference detection process in Embodiment 5. FIG. Explanatory drawing of the hierarchical feature point search process in Embodiment 5. FIG. Explanatory drawing of the coordinate range conversion process in Embodiment 5. FIG.

  Embodiments of the present invention will be described below with reference to the drawings.

Embodiment 1
(Overview)
Are unintended minor changes such as turning on / off the switch on the control panel when doing repair work on the control panel possible to lead to an accident or damage, so are the change points on the control panel after the repair work appropriate? The importance of confirmation is growing.

  The board change point detection system 1 of the present embodiment illustrated in FIG. 1 grasps the change point of the control board easily and accurately based on the image of the control board before and after the repair work.

(Example of mode of panel change point detection system 1)
The panel change point detection system 1 includes an imaging device 20, an input device 30, an image data storage device 40, a change point detection device 00, and a display device 50.

  The photographing device 20, the input device 30, the image data storage device 40, the change point detection device 00, and the display device 50 are implemented in the panel change point detection system 1 by the cooperation of hardware resources and software resources of a computer.

  The photographing device 20 photographs the control panel before and after the repair work. As the photographing device 20, a general digital camera 21 is applied. For example, when the panel change point detection system 1 is mounted on a smart terminal such as a smartphone, a tablet, or a notebook computer, the digital camera 21 is built in the smart terminal.

  The input device 30 receives the specification of the front panel angle with respect to the image of the control panel obtained by the photographing by the input operation of the user (for example, the manager of the control panel). The input device 30 explicitly receives the designation via the input unit 31. As the input unit 31, for example, when the panel change point detection system 1 is mounted on the smart terminal, an aspect of a user interface that receives the designation by an input device or a touch operation in a state displayed on the display of the smart terminal It can be mentioned.

  The image data storage device 40 stores image data obtained by photographing of the control panel.

  The change point detection apparatus 00 processes the image data to detect a change point of the object board.

  The display device 50 displays the change point detection result of the object board via the display unit 500. Examples of the display unit 500 include the display of the smart terminal.

  In addition, the panel change point detection system 1 cooperates with the hardware resource and software resource of the smart terminal to capture the photographing device 20, the input device 30, the image data storage device 40, the change point detection device 00, and the display device 50. It is also possible to configure with a single device.

(Configuration Example and Operation Example of Change Point Detection Device 00)
As illustrated in FIG. 1, the change point detection device 00 includes a panel front surface specifying unit 00, 010, a projection conversion unit 001, 011, and a difference detection unit 020.

  The front panel specifying unit 000 specifies the outline of the front of the panel from the image data 100 of the front panel image obtained by photographing the front of the panel before the repair of the control panel, and extracts the front image 101 of the panel.

  The front panel specifying unit 010 specifies the outline of the front panel surface from the image data 110 of the working rear panel image obtained by photographing the front panel after the work of the modification and extracts the front panel image 111.

  The projective transformation unit 001 transforms the front image 101 of the board into a projective transformation result image 102 by projective transformation based on the coordinates of the corners of the board front.

  The projective transformation unit 011 transforms the front panel image 111 into a projective transformation result image 112 by projective transformation based on the coordinates of the corner of the front panel.

  The difference detection unit 020 subdivides the projective transformation result image 102 before the work and the projective transformation result image 112 after the work, and detects the change point detection result in which the subdivided image having the difference between the work and the work after the work is colored An image 120 is generated.

  The flow of image data processing by the change point detection apparatus 00 will be described with reference to FIGS.

  First, before the repair work of the control board, the control board to be subjected to change point detection is photographed, and the imaging device 20 obtains image data 100. A panel front face angular coordinate group C101 is designated for the image data 100 by the operation of the user of the input device 30. When the image data 100 and the panel front angle coordinate group C101 are input to the panel front specifying unit 000, the panel front image 101 is obtained. The front panel image 101 and the front panel angular coordinate group C 101 are converted into a projection conversion result image 102 by the projection conversion unit 001. The projective transformation result image 102 is input to the difference detection unit 020.

  As described above, the input image data is converted into an image captured from the front of the control panel. The above-described front panel identification processing and projection conversion processing are performed on image data of different shooting locations and shooting times for the same shooting target, as described below.

  Then, after repair work of the control panel, photographing of the control panel is performed again, and the image data 110 is obtained by the photographing device 20. Similarly to the above, the panel front face angular coordinate group C 111 is designated for the image data 110 by the operation of the input device 30 by the user. The image data 110 and the panel front angle coordinate group C 111 are input to the panel front specifying unit 010 to obtain the panel front image 111. The front panel image 111 and the front panel angular coordinate group C 111 are converted into a projective transformation result image 112 by the projective transformation unit 011. The projection conversion result image 112 is input to the difference detection unit 020.

  Then, the difference detection unit 020 performs change point detection by comparing the image data after projective transformation obtained for each image data, and obtains a change point detection result image. That is, the difference detection unit 020 compares the projective transformation result image 102 and the projective transformation result image 112 to detect different portions, colors change points on the projective transformation result image 112, and changes the change point detection result image 120. obtain.

  By the above-described process, even if the images of the camera for photographing the object to be photographed have two different images, the change point can be detected.

(Example of operation of front panel identification unit 00, 010)
A specific operation example of the front panel specifying unit 00 0, 0 10 will be described with reference to FIG.

  The front panel specifying unit 00 0, 0 10 of this embodiment executes the following front surface specifying process.

  S101: The front panel angle to the input image (image of the board) is designated by the instruction received from the input device 30 operated by the user.

  S102: Recognize the coordinates of the designated panel front angle.

  In this process, the front panel specifying unit 000 obtains the front surface angle coordinate group C101 by the explicit specification of the front surface angle in the image data 100 from the user's input device 30.

  On the other hand, the front panel specifying unit 010 obtains the front surface angle coordinate group C111 by the explicit specification of the front surface angle in the image data 110 from the user's input device 30.

  S103: Connect the specified corners with a line to draw an outline of the front of the board.

  In the present process, the outline drawing function 200 of the board front specifying unit 000 draws the board front outline L100 by connecting the specified corner of the board with a line.

  On the other hand, the outline drawing function 200 of the board front surface specifying unit 010 draws the board front surface outline L110 by connecting the specified board corners by a line.

  S104: The image of the part surrounded by the outline of the panel front is cut out.

  In this process, the image extraction function 201 of the front panel specifying unit 000 extracts an image of only the part surrounded by the front panel outline L100 among the input images, and creates the front image 101. On the other hand, the image extracting function 201 of the front panel specifying unit 010 extracts an image of only the part surrounded by the front panel outline L110 among the input images, and creates the front image 111.

  S105: The image of the cut panel front and the corner coordinates of the panel front are input to the projective transformation units 001 and 011.

  In the present process, the front panel image 101 and the front panel angular coordinate group C 101 are input to the projective transformation unit 001. On the other hand, the front panel image 111 and the front panel angular coordinate group C 111 are input to the projective transformation unit 011.

  S106: The above processing of S101 to S105 is performed on each image before and after the work.

(Operation Example of Projective Transformation Units 001 and 0111)
A specific operation example of the projective transformation units 001 and 0111 will be described with reference to FIG.

  The projective transformation units 001 and 011 execute the following projective transformation process.

  S111: A front panel image and front panel angular coordinates are received from the front panel identification unit 00 0,010.

  In this process, the projective transformation unit 001 receives the board front image 101 and the board front corner coordinate group C101 from the board front identifying unit 000.

  On the other hand, the projective transformation unit 011 receives the front panel image 111 and the front panel angle coordinate group C 111 from the front panel identification unit 010.

  S112: Projective transformation is performed on the front image of the board into an image taken from the front of the board.

  In this process, the projective transformation unit 001 transforms the board front surface image 101 into an image captured from the front of the board by a general method based on the projective transformation based on the board front surface coordinate group C101, and the projective transformation result image 102 Get

  On the other hand, the projective transformation unit 011 transforms the board front surface image 111 into an image photographed from the front of the board by a general method according to the projective transformation based on the board front surface coordinate group C 111 and obtains the projective transformation result image 112 .

  S113: The projection conversion result is input to the difference detection unit 020.

  In this process, the projection conversion result image 102 and the projection conversion result image 112 are input to the difference detection unit 020.

  S114: The above is performed on the images before and after the operation (front panel images 101 and 111).

  In the illustrated case, the image format of the projective transformation result image 102 or the projective transformation result image 112, the image size, the image resolution, the area of the board front surface drawn in the projective transformation result image, and the position of the board front surface in the image are input. It is fixed regardless of the front panel image.

(Operation example of difference detection unit 020)
A specific operation example of the difference detection unit 020 will be described with reference to FIG.

  The processing of the difference detection unit 020 executes the following difference detection process.

  S121: The projective transformation result images 102 and 112 before and after the work are received.

  In this process, the projective transformation result image 102 before the work is received from the projective transformation unit 001. Further, the projective transformation result image 112 after the work is received from the projective transformation unit 011.

  S122: Each image is divided into meshes and subdivided, and the coordinates of each subdivided image are stored.

  In this process, the image subdivision function 300 of the difference detection unit 020 first subdivides the projective transformation result image 102 obtained from the projective transformation unit 001, and the respective subdivisions in the subdivision image group 103 and the projective transformation result image 102. A coordinate group C103 of the image conversion image is obtained.

  On the other hand, the image subdivision function 301 of the difference detection unit 020 first subdivides the projective transformation result image 112 obtained from the projective transformation unit 011 and divides each of the subdivision image group 113 and the projective transformation result image 112 A coordinate group C113 of the image is obtained.

  At this time, all the subdivided images in the subdivided image group 103 and the subdivided image group 113 have the same size and the same resolution.

  S123: The divided images at the same coordinates are compared in the RGB color space one pixel at a time.

  In this process, the segmented image comparison function 302 of the difference detection unit 020 compares the segmented images at the same coordinates in pixel units, and sets the number of pixels having the pixel number threshold S1 or more set in the change point detection device 00 in advance. If there is a difference, the process is considered to be a change.

  S124: If there is a pixel having a difference greater than or equal to the RGB threshold, the segmented image is recognized as a portion having a difference if the pixel number threshold or more exists, and the coordinates are recorded.

  In this process, the RGB color space is referred to for comparison in pixel units. When each pixel in the subdivided image of the same coordinates in the subdivided image group 103 and the subdivided image group 113 has a difference greater than or equal to the RGB threshold group S2 set in advance in the change point detection device 00, that pixel is Recognize that there has been a change. The RGB threshold value group S2 has threshold values S2R, S2G, and S2B of differences between RGB values for R (red), G (green), and B (blue). Even if the RGB value of each pixel in the subdivided image of the subdivided image group 113 is at least one with respect to the RGB value of each pixel in the subdivided image of the subdivided image group 103 having the same coordinates, ± S2R, ± S2G If it is out of the range of ± S2B, it is considered that the pixel is changed.

  If the number of pixels having differences in each subdivided image does not satisfy the pixel number threshold S1, the pixel comparison in the subdivided image is continued. If the number of pixels having differences even if all the pixels are compared is less than the pixel number threshold S1, it is considered that there is no change in the segmented image, the comparison of the segmented images is completed, and the next segmented image Make a comparison of

  Further, in the comparison processing, when the difference in the number of pixels of the pixel number threshold S1 or more with respect to the pixels in the subdivided image is confirmed, as illustrated in FIG. The comparison of the segmented image is completed. After that, the coordinates of the segmented image are recorded, and the next segmented image is compared.

  S125: The above process is performed on all the segmented images.

  In this process, the above comparison process is repeated to compare all subdivided images of the subdivided image group 103 and the subdivided image group 113, and the coordinates of the subdivided image whose change point is confirmed are recorded as a change point coordinate group C120. .

  S126: Only the portion of the subdivided image having a difference is colored and output in the projective transformation result image 112 after the operation.

  In this process, a region of the same size as the subdivided image is colored in each coordinate of the change point coordinate group C 120 in the projective transformation result image 112. Thereby, the change point detection result image 120 is obtained.

(Effect of Embodiment 1)
The prior art such as Patent Document 1 is a method of performing change point detection using a three-dimensional shape restoration technique. However, in general, the board has few feature points, and the accuracy when performing three-dimensional shape restoration is low. When the technique described in this document is used for a board, feature points on the board can not be captured during three-dimensional shape restoration, or three-dimensional shape restoration can not be performed, or three-dimensional shape restoration of the board can be performed However, the accuracy is low, and a part not changed may be recognized as a change point.

  Moreover, the prior art like patent document 2 was a method of attaching an AR marker to imaging | photography object, and detecting the change point of a to-be-photographed object. When the AR marker is retrofitted to the board, the position and the orientation of the AR marker may change during the work of remodeling or repairing the board. In that case, the positions of the components such as switches and lamps on the board with reference to the position of the AR marker are recognized as being moved, and the change point can not be detected. Also, even if the AR marker does not move during the remodeling or repair work of the board, it is generally accurate to install a small AR marker for objects of a size such as a control board to recognize the positional relationship. As a result, there is a case where a portion not changed is recognized as a changed point.

  On the other hand, the board change point detection system 1 according to the present embodiment recognizes the board front by the board front identification unit 0000 and 1010, and generates an image of the board front photographed from the front by the projective transformation units 001 and 011, The difference detection unit 020 determines the change point on the front of the board based on the RGB value difference. That is, when comparing the front image of the panel before and after the work, the color of the same coordinates on the front surface of the panel on the basis of the front surface angle of the panel is not compared with the feature point on the front surface of the panel or the AR marker. Compare. This makes it possible to recognize the front of the board with higher accuracy and to recognize the change points with higher accuracy.

  As described above, according to the panel change point detection system 1 of the present embodiment, the control panel which is generally considered to have few characteristic points is the imaging target, and the image to be remodeled under different imaging conditions and illumination conditions It is possible to detect the change point of the board and the repaired board with high accuracy. In addition, the changed point can be detected with high accuracy without installing a special illustration that is a feature such as an AR marker.

Second Embodiment
(Overview)
The first embodiment adopts a method in which the user explicitly sets in the input unit 31 by the operation of the input device 30 when setting the corner coordinates of the front surface of the board in the image data 100 and the image data 110.

  On the other hand, in the second embodiment, setting of the panel front face angular coordinates of the image data 110 is automated by utilizing the image processing by the hardware resources of the computer in which the change point detection device 00 is mounted.

  In the panel change point detection system 1 of the second embodiment, the panel front surface identification unit 000 of FIG. 2 of the change point detection device 00 is changed to the processing function of FIG. The fourth embodiment is the same as the first embodiment except that the processing function of FIG. 4 is changed to the processing function of FIG.

(Example of operation of front panel identification unit 00, 010)
The flow of image processing according to the present embodiment will be described with reference to FIGS.

  The front panel specifying unit 00 0, 0 10 according to the second embodiment executes the following front surface specifying process.

  S201: The user designates a front panel angle with respect to the input before-operation panel image.

  In this process, as shown in FIG. 7, the panel front surface identification unit 000 first receives the panel front surface coordinate group C101 of the input image data 100 in the input unit 31 by the operation of the input device 30 by the user. Explicitly specified.

  S202: Recognize the coordinates of the front panel angle.

  S203: Connect specified corners with a line to draw an outline of the front of the board.

  In this process, the outline drawing function 210 of the front side specifying unit 000 draws the front side outline L101 by connecting the specified front side angular coordinate group C101 with a line.

  S204: The image of the part surrounded by the outline of the panel front is cut out.

  In this process, the image extraction function 211 of the front panel specifying unit 000 extracts an image of only the part surrounded by the front panel outline L101 in the input image, and obtains the front image 101. The front panel image 101 is input to the projection transformation unit 001 together with the front panel angular coordinate group C101.

  S205: Cut out an image around a corner of the panel front (four corners of the panel front) and register it as a matching template image.

  In the present process, the image extraction function 212 of the front panel specifying unit 000 extracts an image around the front panel angle coordinate group C 101 from the front panel image 101 to obtain a matching template image group 104. When the peripheral image of the panel front angle coordinate group C101 is extracted from the panel front image 101, the image is surrounded by the panel front edge and the image edge on the premise that the panel front edge around the panel front angle is included. By extracting the area, a matching template image group 104 obtained by extracting the vicinity of the front panel angle of the front panel is obtained. The matching template image group 104 is input to the front panel specifying unit 010.

  S206: Template matching is performed on the input image of the board after the work.

  In this process, as shown in FIG. 8, first, the template matching function 213 of the front panel identification unit 010 uses the matching template image group 104 as a matching template for the input image data 110 and performs template matching. Do. Among the obtained matching results, the image portion closest to each matching template image on the image data 110 is selected, and the corner edge of the image portion is set as the panel front surface corner coordinate group C111.

  S207: Recognize the coordinates of the front panel angle.

  S208: Connect the detected corner of the board with a line, and draw the outline of the board front.

  As in the example of FIG. 9, which corner edge is used as the panel front face coordinate differs depending on the matching template image group 104, that is, which matching template image of the four matching template images matches. . Therefore, in the present process, the outline drawing function 214 of the board front specifying unit 010 draws the board front outline L111 by connecting the board front corner coordinate group C111 with a line.

  S209: The image of the part surrounded by the outline of the panel front is cut out.

  In this process, the drawing and extracting function 215 of the front panel specifying unit 010 extracts an image of only the part surrounded by the front panel outline L111 in the input image to obtain the front image 111.

  S210: The cut-off panel front image and the corner coordinates of the panel front are input to the projective transformation unit.

  In the present process, the panel front surface image 111 and the panel front surface angular coordinate group C 111 are input to the projection conversion unit 011.

  As described above, according to the panel change point detection system 1 of the present embodiment, as in the first embodiment, the panel front image 101 and the panel front angle coordinate group C 101 obtained above are included in the projection conversion unit 001 in the panel front surface. The image 111 and the panel front face angular coordinate group C 111 are input to the projective transformation unit 011. Then, the projective transformation result image 102 and the projective transformation result image 112 obtained by the projective transformation unit 011 are input to the difference detection unit 020, and the change point detection result image 120 is obtained.

  In particular, the present embodiment is not limited to the template matching method described above, and can be realized, for example, by applying the arrangement detection technique of Patent Document 3.

  As described above, according to the panel change point detection system 1 of the present embodiment, the setting of the panel front face angular coordinates in the image data 110 is automated by utilizing the image processing of the computer. In addition to the effects of the above, the data processing can be made more efficient.

Third Embodiment
(Overview)
In the first embodiment described above, the user is explicitly set by the change point detection apparatus 00 when designating the angular coordinates of the front surface of the board in the image data 100 and the image data 110.

  In the second embodiment, when the user explicitly sets the panel front surface coordinate of the image data 100, the panel front surface coordinate of the image data 110 is automatically recognized by the change point detection device 00.

  On the other hand, in the third embodiment of FIG. 11, it is assumed that the front of the board is the largest in the photographed picture, and the user does not set the board front angle coordinates and which part of the front of the board Even when remodeling or repair is performed, the change point detection device 00 automatically recognizes the front panel angle.

  The panel change point detection system 1 of the third embodiment does not include the input device 30 of the first embodiment, and the processing contents of the panel front surface identification unit 000 and the panel front surface identification unit 010 of the change point detection device 00 substitute for FIG. This embodiment is the same as the embodiment 1 except that the contents of FIG.

(Example of operation of front panel identification unit 00, 010)
The flow of image processing according to the present embodiment will be described with reference to FIGS.

  The front panel identification unit 0000 and 1010 of the third embodiment execute the following front panel identification process.

  S301: Perform straight line detection and corner detection on the input panel image.

  In this process, the straight line detection function 220 of the front panel identification unit 000 first extracts a group of straight component formulas E100 present in the input image data 100 using a general method based on Hough transform. On the other hand, the straight line detection function 220 of the front panel identification unit 010 first extracts the straight component formula group E110 present in the input image data 110 using a general method based on Hough transform.

  Further, the corner detection function 221 of the front panel specifying unit 000 detects a corner present in the image data 100 using a general method based on corner detection, and obtains an angular coordinate group C100. On the other hand, the corner detection function 221 of the front panel identification unit 010 detects a corner present in the image data 110 using a general method based on corner detection, and obtains an angular coordinate group C110.

  S302: If the detected angular coordinates exist on the detected straight line, an image is drawn in which lines are drawn only between the coordinates.

  In this process, the all linear contour drawing function 222 of the front panel specifying unit 000 focuses on each linear component expression of the linear component expression group E100, and when there are a plurality of angular coordinates on the line of the linear component expression, the corners thereof Draw a straight line between the coordinates (FIG. 12). As a result, a straight line contour image 105 in which only the entire straight line contour present in the image data 100 is drawn is obtained.

  On the other hand, the full linear contour drawing function 222 of the front panel specifying unit 010 focuses on each linear component expression of the linear component expression group E110, and when there are a plurality of angular coordinates on the line of the linear component expression, Draw a straight line on the (Fig. 12). As a result, a straight line contour image 115 is obtained in which only the entire straight line contour present in the image data 110 is drawn.

  S303: Calculate the area of all the quadrangles surrounded by the drawn line, and create an image in which only the largest quadrilateral of the drawn quadrilaterals is drawn.

  In this process, the maximum area plane selection function 223 of the front panel specifying unit 000 calculates the areas of all the squares of the straight line contour image 105, and selects the square that is the maximum area plane. As a result, the maximum area planar outline image 106 in which only the maximum area plane of the linear outline image 105 is drawn is obtained.

  On the other hand, the maximum area plane selection function 223 of the panel front surface specification unit 010 calculates the area of all the squares of the straight line contour image 115, and selects the square that is the maximum area plane. Thereby, the largest area planar outline image 116 which drew only the largest area plane of the linear outline image 115 is obtained.

  S304: An image in which a quadrangle of the largest area is drawn and an original input image are superimposed, and an image is extracted only within the range surrounded by the quadrilateral. As a result, only the front of the board is extracted.

  In this process, the image extraction function 224 of the front panel specifying unit 000 superposes the image data 100 on the maximum area planar contour image 106, and cuts out only the image portion in the maximum area planar contour. Thereby, the front panel image 101 is obtained.

  On the other hand, the image extraction function 224 of the front panel identification unit 010 superimposes the image data 110 on the maximum area planar contour image 116, and cuts out only the image portion in the maximum area planar contour. Thereby, the front panel image 111 is obtained.

  S305: The corner coordinates are superimposed on the image in which the quadrangle of the largest area is drawn, and the corner coordinates of the quadrangle of the largest area are selected. Thereby, the corner coordinates of the front of the board are recognized.

  In this process, the panel front surface angle coordinate selecting function 225 of the panel front surface specification unit 000 superposes the group of angular coordinates C100 on the maximum area planar contour image 106, and selects the angular coordinates of the maximum area plane from the group of angular coordinates C100. Thereby, the board front surface angular coordinate C101 is obtained.

  On the other hand, the panel front surface angular coordinate selecting function 225 of the panel front surface specification unit 010 superposes the angular coordinate group C110 on the largest area planar contour image 116, and selects the angular coordinates of the largest area plane from the angular coordinate group C100. Thereby, the board front surface angular coordinate C111 is obtained.

  S306: The cut-off panel front image and the corner coordinates of the panel front are input to the projective transformation unit.

  In the present process, the front panel image 101 and the front panel angular coordinate group C101 obtained in S305 are input to the projective transformation unit 001. On the other hand, the panel front surface image 111 and the panel front surface angular coordinate group C111 obtained in S305 are input to the projective transformation unit 011.

  S307: The above processing of S301 to S306 is performed on each image before and after the work.

  As described above, according to the panel change point detection system 1 of the present embodiment, as in the first embodiment, the panel front image 101 and the panel front angle coordinate group C 101 obtained above are included in the projection conversion unit 001 in the panel front surface. The image 111 and the panel front face angular coordinate group C 111 are input to the projective transformation unit 011. Then, the projective transformation result image 102 and the projective transformation result image 112 obtained by the projective transformation unit 011 are input to the difference detection unit 020, and the change point detection result image 120 is obtained.

  Therefore, according to the panel change point detection system 1 of the present embodiment, in addition to the effects of the inventions of the first and second embodiments, the user does not have to set the panel front surface angle coordinates, and the speed of on-site work is increased. It is possible. In addition, even if the panel front corner portion is changed in the remodeling and repair work, the panel front corner can be recognized.

Fourth Embodiment
(Overview)
When comparing the subdivided image group 103 and the subdivided image group 113, the difference detection unit 020 of the fourth embodiment shown in FIG. 13 refers to the HSV color space instead of referring to the RGB color space. In the HSV color space, hue (H), saturation (S) and lightness (V) are treated as individual parameters.

  The panel change point detection system 1 of the fourth embodiment is the same as that of the first embodiment except that the processing content of the difference detection unit 020 of the change point detection device 00 is the content of FIG. 13 instead of FIG. It is.

(Operation example of difference detection unit 020)
The flow of difference detection according to the present embodiment will be described with reference to FIGS.

  The difference detection unit 020 of the fourth embodiment executes the following difference detection process.

  S401: Receive the projective transformation result image 102 before the work from the projective transformation unit 001. On the other hand, the projective transformation result image 112 after the work is received from the projective transformation unit 011.

  S402: Each image is divided into meshes and subdivided, and the coordinates of each subdivided image are stored.

  In this process, first, the difference detection unit 020 subdivides the projective transformation result image 102 obtained from the projective transformation unit 001, and sets the coordinate group C103 of each subdivided image of the subdivided image group 103 and the projective transformation result image 102. obtain. On the other hand, the projective transformation result image 112 obtained from the projective transformation unit 011 is subdivided, and a coordinate group C 113 of each subdivided image of the subdivided image group 113 and the projective transformation result image 112 is obtained.

  S403: Compare the hue in the HSV color space one pixel at a time in the subdivided image of the same coordinates.

  In this process, all subdivided images of the subdivided image group 103 and the subdivided image group 113 have the same size and resolution. The divided images at the same coordinates are compared in units of pixels, and if there is a difference in the number of pixels equal to or larger than the pixel number threshold S1 set in advance in the change point detection device 00, processing is performed as a changed part.

  Pixel-by-pixel comparison refers to the hue in the HSV color space. For each pixel in the subdivided image of the same coordinates in the subdivided image group 103 and the subdivided image group 113, based on the hue threshold value S3 set in advance on the alteration point detection device 00, the presence or absence of change for each pixel Make a judgment on

  The hue threshold value S3 in the fourth embodiment indicates the threshold value (HSV threshold value) of the hue difference in the HSV color space. The HSV color space is a color space consisting of three elements of hue (H), saturation (S), and lightness (V), and is characterized in that the hue is expressed by one parameter. As a result, even if the lightness of the subject board image before and after the work is different, it is possible to compare only the hue of the subject board by comparing only the hue H values.

  When two colors having the same hue and different lightness are represented by RGB values in the RGB color space, the three parameters of the R value, the G value, and the B value are different although they have the same hue. Therefore, when the lightness of the subject board image before and after the work is different, it is difficult to determine the hue difference in the RGB color space as in the first embodiment. There is a possibility that the part which is not changed is misrecognized as a change point.

  In the fourth embodiment, by comparing only the hue H value in the HSV color space, it is easier to set the threshold when determining the hue difference even if the lightness of the photographed board image before and after the work is different. As a result, false recognition due to false setting of the threshold is less likely to occur.

  S404: If there is a pixel having a difference greater than or equal to the hue threshold value, the segmented image is recognized as a difference portion if the pixel number threshold or more exists, and the coordinates are recorded.

  In this process, the hue value of each pixel in the subdivided image of the subdivided image group 103 is H1, and the hue value of each pixel in the subdivided image of the subdivided image group 113 at the same coordinates is H2, | H1 If the smaller of -H2 | and (360- | H1-H2 |) is S3 or more, it is considered that the pixel has been changed. If the number of pixels having differences in each subdivided image does not satisfy the pixel number threshold S1, the pixel comparison in the subdivided image is continued. If the number of pixels having differences even if all the pixels are compared is less than the pixel number threshold S1, it is considered that there is no change in the segmented image, the comparison of the segmented images is completed, and the next segmented image Make a comparison of Further, in the above comparison processing, when a difference in the number of pixels of the pixel number threshold S1 or more with respect to the pixels in the subdivided image is confirmed, it is regarded as a portion having a change, and the comparison of the subdivided image is ended at that time. (Figure 14). After that, the coordinates of the segmented image are recorded, and the next segmented image is compared.

  S405: The above process is performed on all segmented images.

  The comparison processing of S403 and S404 is repeated, all the subdivision images of the subdivision image group 103 and the subdivision image group 113 are compared, and the coordinates of the subdivision image whose change point is confirmed are recorded as a change point coordinate group C120.

  S406: Only the portion of the segmented image having a difference is colored and output in the projective transformation result image after the operation.

  In this process, in the projection conversion result image 112, an area of the same size as the subdivided image is colored at each coordinate of the change point coordinate group C120. Thereby, as shown in FIG. 3, the change point detection result image 120 is obtained.

  If the lightness of the photographed board image is different before and after the work, it may be recognized as a change even in a place where it has not been changed. This is because, when two colors having the same hue but different lightness are represented by RGB values, all three parameters of R value, G value, and B value may be different. From this, if the setting of the RGB threshold value group S2 is incorrect, depending on the lightness of the board to be photographed, there is a possibility that a portion not changed is erroneously recognized as a change point.

  On the other hand, according to the panel change point detection system 1 of the present embodiment, the segmented image group 103 and the segmented image group 113 are compared only by the hue by setting the threshold only for the hue (H). It becomes easy. Therefore, according to the present embodiment, in addition to the effects of the inventions of the first to third embodiments, it is possible to perform change point recognition with high accuracy even when the brightness of both image groups is different.

Fifth Embodiment
(Overview)
In the first and fourth embodiments, the projective transformation result image 102 and the projective transformation result image 112 are divided into the divided image group 103 and the divided image group 113, and the change points are performed by comparing colors in the divided images of the same coordinates. To detect

  On the other hand, when comparing the projective transformation result image 102 and the projective transformation result image 112, the difference detection unit 020 of this embodiment shown in FIG. 15 does not compare the color of the same coordinates of the two, instead of comparing them. Feature points of the projective transformation result image 102 are found from the projective transformation result image 112. As a result, even if the position coordinates of the panel component included in the projective transformation result image 102 and the projective transformation result image 112 are locally different, it is possible to perform change point recognition with high accuracy.

(Operation example of difference detection unit 020)
The flow of difference detection according to the present embodiment will be described with reference to FIGS.

  The difference detection unit 020 of the fifth embodiment executes the following difference detection process.

  S501: The projection transformation result image 102 before the work is received from the projection transformation unit 001. On the other hand, the projective transformation result image 112 after the work is received from the projective transformation unit 011.

  S502: The reduced images 102a and 112a are created by reducing the image size of the projective transformation result images 102 and 112 before and after the work.

  In this process, the image reduction function 320 of the difference detection unit 020 reduces the image size of the projective transformation result image 102 obtained from the projective transformation unit 001 to obtain a reduced image 102a. On the other hand, the image reduction function 321 of the difference detection unit 020 reduces the image size of the projective transformation result image 112 obtained from the projective transformation unit 011 to obtain a reduced image 112a.

  S503: The reduced images 102b and 112b are generated by further reducing the image size of the reduced images 102a and 112a.

  In this process, the image reduction function 322 of the difference detection unit 020 further reduces the image size of the reduced image 102a to obtain a reduced image 102b as shown in FIG. On the other hand, the image reduction function 323 of the difference detection unit 020 further reduces the image size of the reduced image 112a and obtains the reduced image 112b as shown in the figure.

  S504: A hierarchical feature point search process is performed based on the projective transformation result images before and after the work and the reduced images obtained in the process of S502 to S503.

  In this process, the hierarchical feature point search processing function 324 of the difference detection unit 020 hierarchically applies the projective transformation result image 102, the projective transformation result image 112, the reduced image 102a, the reduced image 112a, the reduced image 102b, and the reduced image 112b. Perform feature point search processing.

  The hierarchical feature point search process first performs template matching of the lowest resolution reduced image 102b and the reduced image 112b. The feature points determined to be different in this process are stored. In addition, template matching is performed on the reduced image 102 a and the reduced image 112 a having high resolution in the area determined to have no difference. The feature points determined to be different in this process are stored. Further, template matching of the projective transformation result image 102 and the projective transformation result image 112 is further performed on the area determined to have no difference. The feature points determined to have a difference in this process are stored. In the above process, in the projective transformation result image 102 and the projective transformation result image 112, a coordinate group determined to have a difference is obtained. As such hierarchical feature point search processing, a general method called hierarchical template matching, hierarchical matching or the like may be applied (for example, non-patent document 1).

  S505: The coordinate group of feature points with low similarity obtained by the hierarchical feature point search process is obtained.

  In this process, the hierarchical feature point search processing function 324 regards feature points whose similarity is equal to or more than the similarity threshold S4 as locations that are not changed, and coordinate groups of feature points whose similarity is lower than the similarity threshold S4 are not It is obtained as a coordinate group C130 of similar feature points.

  S506: Among the coordinate group of the change point detection area obtained by the coordinate range conversion process, an area where adjacent coordinates exist is regarded as a change range.

  In this process, as shown in FIG. 17, the coordinate range conversion processing function 325 of the difference detection unit 020 is a circle whose radius is the coordinate adjacent distance R1 with a certain coordinate as a center in the coordinate group C130 of dissimilar feature points. The area is regarded as a changed area only when there are five or more adjacent coordinate number threshold values S in the area, and the area is regarded as a change point detection area. In the example of the figure, the areas shown in (1) and (3) are detected as the change point detection area. The above-described coordinate range conversion processing is performed on all the coordinates of the coordinate group C130 of dissimilar feature points, whereby a change point detection area group A130 is obtained.

  S507: The predetermined change range in the projective transformation result image 112 is colored and output.

  In this process, the change point detection area coloring function 326 of the difference detection unit 020 obtains the change point detection result image 120 by coloring the areas of the change point detection area group A 130 in the projective transformation result image 112.

  At the time of shooting the front image 100 and the rear image 110 of the work, if the positional relationship of the imaging device with respect to the board to be photographed is largely different, even if it is changed to the image photographed from the front of the board by projection conversion processing There are cases where the position coordinates of the board component in the image 102 and the projective transformation result image 112 differ. This causes the change point detection accuracy of the change point detection system to be lowered.

  On the other hand, when comparing the projective transformation result image 102 and the projective transformation result image 112, the difference detection unit 020 according to the present embodiment does not compare the color of the same coordinates, but projects by hierarchical feature point search processing. Feature points are found from the projective transformation result image 112 from the transformation result image 102.

  In particular, according to the hierarchical feature point search processing, template matching is performed sequentially from the reduced image 102 b and the reduced image 112 b having a low resolution, with the search window size fixed. The feature points in are found.

  As described above, according to the present embodiment, the positional coordinates of the panel components in the projective transformation result image are local, as in the first and fourth embodiments, due to the difference in the positional relationship of the imaging device with respect to the target board. Even if they are different, it is possible to perform change point detection with high accuracy. Further, in the hierarchical feature point search processing of the present embodiment, the search range is limited for each hierarchy and searched, and the presence or absence of a feature point having a high degree of similarity in the designated search range is used as a determination criterion. Therefore, even when compared with the methods of Embodiments 1 and 4 in which the color of the same coordinate is compared in the projective conversion result image, the method is more robust against differences in local position coordinates of board components in the projective conversion result image. Become. In particular, it is suitable for feature point search of a board in which feature points are similarly arranged. In addition, it can implement | achieve by using a general method, without limiting regarding a hierarchical feature point search processing method.

[Another Example of the Present Invention]
Another embodiment of the present invention is a change point detection program which causes a computer to execute the above-mentioned change point detection apparatus 00 or a change point detection method. This program can be provided via a known computer readable recording medium or a network such as the Internet.

  The present invention is not limited to the embodiments described above, and can be implemented in various modes within the scope of the claims of the present invention.

DESCRIPTION OF SYMBOLS 1 ... panel change point detection system 30 ... input device 40 ... image data storage device 00 ... change point detection device, 0000, 010 ... panel front surface identification unit, 001, 011 ... projection conversion unit, 020 ... difference detection unit 50 ... display device

Claims (17)

A change point detection device for detecting a change point before and after a repair work of a control panel,
A front panel identification unit that identifies an outline of the front of the panel from image data obtained by photographing the front of the panel before and after the work of the modification and extracts an image of the front of the panel;
A projection conversion unit that converts the front panel image into a projection conversion result image;
Providing a difference detection unit for dividing the projective transformation result images before and after the work and generating a change point detection result image in which the divided images with differences between before and after the work are colored Characteristic change point detection device.   2. The change point detection device according to claim 1, wherein the board front surface specifying unit extracts an area specified based on a board front surface coordinate specified for the image data as the board front surface image.   The front panel specifying unit extracts an area specified based on front panel angular coordinates specified for the image data as the front panel image, and from the front panel angular coordinates, a matching template of four front corners of the front panel. 2. The change point detection device according to claim 1, wherein the image on the front side of the board is extracted by obtaining an image and specifying the front side of the board after work with matching template images at the four corners of the front side of the board.   The change point detection apparatus according to any one of claims 1 to 3, wherein the board front surface specifying unit sets a quadrangle having the largest area among the squares included in the image data as an outline of the board front surface. .   The said difference detection part detects the said difference based on a RGB threshold value, The change point detection apparatus of any one of Claim 1 to 4 characterized by the above-mentioned.   The said difference detection part detects the said difference based on a HSV threshold value, The change point detection apparatus of any one of Claim 1 to 4 characterized by the above-mentioned.   The difference detection unit is a projective transformation result image of the work before the work detected in the projective transformation result image after the work by template matching of hierarchical reduced images of the projective transformation result images before and after the work. The change point detection apparatus according to claim 1, wherein a change point detection result image in which a group of dissimilar feature points is colored is generated.   8. The change point detection device according to claim 7, wherein the difference detection unit regards an area in which the number of adjacent coordinates is equal to or more than a threshold value in the group of dissimilar feature points as a changed area. A change point detection method in which a computer detects a change point before and after a repair work of a control panel,
A front panel specifying process of specifying an outline of the front of the panel from image data obtained by photographing the front of the panel before and after the work of the modification and extracting an image of the front of the panel;
Projective transformation process for transforming the front panel image into a projective transformation result image;
Change point detection process including subdivision of the projective transformation result images before and after the operation to generate a change point detection result image in which the divided images having differences before and after the operation are colored Method.   10. The change point detection method according to claim 9, wherein the board front specifying step extracts an area specified based on a board front corner coordinate specified for the image data as the board front image.   The front panel specifying step extracts a region specified based on front panel angular coordinates specified for the image data as the front panel image, and from the front panel angular coordinates, a matching template of four front corners of the front panel. 10. The change point detection method according to claim 9, wherein the image on the front side of the board is extracted by obtaining an image and specifying the front side of the board after the operation with matching template images at the four corners of the front side of the board.   12. The change point detection method according to any one of claims 9 to 11, wherein the board front specifying step sets a quadrangle having the largest area among the quadrangles included in the image data as the outline of the board front. .   The change point detection method according to any one of claims 9 to 12, wherein the difference detection step detects the difference based on an RGB threshold.   The change point detection method according to any one of claims 9 to 12, wherein the difference detection step detects the difference based on an HSV threshold.   The difference detection process is performed by using a template matching of hierarchical reduced images of the projective transformation result images before and after the work with respect to the projective transformation result image before the work and detected in the projective transformation result image after the work. 10. The change point detection method according to claim 9, generating a change point detection result image in which a group of dissimilar feature points is colored.   The method according to claim 15, wherein in the difference detection step, an area having a number of adjacent coordinates equal to or larger than a threshold in the group of dissimilar feature points is regarded as an area having a change.   A change point detection program characterized by causing a computer to function as the change point detection device according to any one of claims 1 to 8.