JP2019159375A - Information processing apparatus, superposition display program and superposition display method - Google Patents

Abstract

To provide an information processing apparatus, a superposition display program and a superposition display method for improving an extraction efficiency of a ridge line of design data and expanding a scope of application of the superposition display technique.SOLUTION: A first group of lines corresponding to a ridge of an object is extracted based on a picked-up image of the object (A1). A first project image is generated based on design data of the object and a first external parameter (viewpoint position and visual line direction (A3). A second group of lines corresponding to a ridge of the object is extracted based on the first project image (A4). The second group of lines are converted to a third group of lines based on the first external parameter (A5). A second external parameter that is a view point position and a visual line direction same as the picked-up image is calculated based on a correspondence between the first group of lines and the third group of lines (A12). A second project image is rendered to be superposed on the picked-up image based on the design data and the second external parameter (A13).SELECTED DRAWING: Figure 8

Description

  The present invention relates to an information processing apparatus, program, and method for superimposing and displaying a computer graphics (CG) of a perspective projection image drawn from design data of an object and an actual captured image.

  In recent years, display technology using AR (Augmented Reality) technology has been proposed in product inspection in the manufacturing field. In other words, this is a technology that makes it easy to grasp the difference between the actual product and the design drawing by superimposing and displaying the captured image of the product (object) and the perspective projection image rendered from the design data of the product. . As an example of such a technique, the edge line is extracted by performing edge detection processing on the captured image, the ridge line of the design data is extracted, and the operator inputs the correspondence between these two, thereby the edge There is a method to generate CG where the ridge line overlaps the line. By displaying the captured image and the perspective projection image so as to overlap with high accuracy, the accuracy and work efficiency of product inspection can be improved (see Patent Documents 1 to 3).

  In order to draw a perspective projection image that accurately overlaps the captured image, not only the design data of the drawing target but also the relative position and orientation of the camera with respect to the drawing target (that is, the perspective position and the line-of-sight direction of the perspective projection) Is preferably taken into account. Such information is called “external parameters” as opposed to “internal parameters” such as focal length and sensor size defined by the internal structure of the camera. For example, the external parameter may be manually set by the user with reference to the captured image, or a user who selects a perspective projection image similar to the captured image from a plurality of preset external parameter groups. May be selected.

  On the other hand, a method has also been proposed in which an external parameter equivalent to a captured image is estimated by associating a linear element (edge line) included in the captured image with a linear element (ridge line) included in the subject design data (non-existing). Patent Document 1). By increasing the estimation accuracy of the external parameters, it is possible to calculate where the 3D coordinates on the design data are to be perspectively transformed (mapped) in the captured image, and the overlay accuracy between the captured image and the perspective projection image Can be improved.

JP 2017-091078 JP 2007-207251 A Japanese Unexamined Patent Publication No. 06-168341

Lilian Zhang, Chi Xu, Kok-Meng Lee, Reinhard Koch, "Robust and Efficient Pose Estimation from Line Correspondences", Asian Conference on Computer Vision, Computer Vision-ACCV 2012, Springer, p.217-230, (2013)

  Edge lines in the design data are extracted by searching for line features representing the external features of the 3D CAD (Computer-Aided Design) model from the line segment information (polyline information, polygon information) included in the design data. . However, if the outer shape of the 3D CAD model has a curved surface (including a pseudo curved surface that connects not only a plane but also a plurality of minute planes), extract an appropriate edge line. Is difficult. For example, when an edge with a fillet (roundness or round) is provided in the design data, the edge is locally formed on a curved surface with a constant curvature (or a curved surface with a gradually changing curvature). In other words, it is difficult to extract a specific part as a ridge line. A similar problem can occur when the curved surface in the design data is defined as a parametric curved surface. Even in the case of a three-dimensional CAD model having no curved surface, the calculation time for extracting the edge line is prolonged as the number of polygons included in the design data increases. For example, it may take several minutes to several tens of minutes to extract a ridge line from design data of millions of polygons, and it is desirable to improve calculation efficiency.

  In one aspect, an object is to improve the extraction efficiency of edge lines in design data and improve the accuracy of superimposed display.

  In one embodiment, the information processing apparatus includes a first extraction unit, a generation unit, a second extraction unit, a three-dimensional conversion unit, a calculation unit, and a superimposed display unit. The first extraction unit extracts a first straight line group corresponding to a ridge line of the object based on a captured image of the object. The generation unit generates a first projection image of the object based on data defining a three-dimensional shape of the object and a first external parameter representing a viewpoint position and a line-of-sight direction when the object is perspective-projected. The second extraction unit extracts a second straight line group corresponding to the ridge line based on the first projection image. The three-dimensional conversion unit converts the second line group into a three-dimensional third line group based on the first external parameter. The calculation unit calculates a second external parameter representing a viewpoint position and a line-of-sight direction equivalent to an imaging position and an imaging direction of the captured image based on a correspondence relationship between the first straight line group and the third straight line group. The superimposed display unit renders a second projected image of the object superimposed on the captured image based on the data and the second external parameter.

  In one aspect, the extraction efficiency of the ridgeline of design data can be improved, and the accuracy of superimposed display can be improved.

(A) is an example of a camera image of a product, (B) is an example of a perspective projection image of design data of the product, and (C) is an example of a superimposed image of (A) and (B). (D) is an example of a perspective projection image of design data drawn from an angle different from (C), and (E) is an example of a superimposed image of (A) and (D). It is a figure which shows the hardware constitutions of information processing apparatus. It is a figure which shows the software structure of a superimposition display program. It is an example of a display screen of a display device (input of a straight line pair). (A) is a perspective image example of design data, and (B) to (D) are a normal image example, a depth image example, and a grayscale image example. (E), (F) is a figure which shows the 2nd straight line group and the 3rd straight line group. (A)-(E) are the figures for demonstrating the division | segmentation and recombination of a 3rd straight line group. It is an example of a display screen (superimposed display) of a display device. It is a flowchart which illustrates the flow of the superimposition display method.

[1. Overview]
Hereinafter, a superimposed display program and a superimposed display method executed by the information processing apparatus 1 and the information processing apparatus 1 as embodiments will be described with reference to the drawings. The apparatus, the program, and the method of the present case are superimposing that accurately superimpose and display a captured image 50 (actually captured camera image) of a product and a CG of a perspective projection image drawn from the design data of the product. Provide a display function. FIG. 1A is an example of the former captured image 50, and FIG. 1B is an example of the latter perspective projection image. By superimposing these with high accuracy, a superimposed image as shown in FIG. Therefore, it becomes easy for the user to quickly find the difference between the former and the latter visually, and the accuracy and work efficiency of product inspection are improved.

  On the other hand, FIG. 1D is a perspective projection image generated from the same design data at a different viewpoint position and line-of-sight direction from FIG. Since the viewpoint position and the line-of-sight direction in FIG. 1D do not match the camera position and orientation in FIG. 1A, even if they are superimposed, as shown in FIG. . Therefore, in order to improve the accuracy and work efficiency of product inspection, it is preferable to generate a perspective projection image that can be accurately superimposed on the captured image 50, and to accurately grasp external parameters equivalent to the captured image 50. It is important.

[2. Hardware configuration]
FIG. 2 is a block diagram for explaining the information processing apparatus 1 (computer) that performs the above superimposed display and peripheral devices connected thereto. A camera 2, a storage 3, an input device 4, and a display device 5 are connected to the information processing apparatus 1. The connection form may be wired connection or wireless connection, and may be connected via a network (not shown). When the information processing apparatus 1 is a notebook personal computer, a smartphone, a tablet terminal, or the like, the camera 2, the storage 3, the input device 4, the display apparatus 5, and the like built in the information processing apparatus 1 may be used.

  The camera 2 is a digital camera (imaging device), an image scanner, a video camera, or the like that incorporates an image sensor such as a CCD (Charge-Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). 50 is acquired. Information of the captured image 50 acquired by the camera 2 is input to the information processing apparatus 1. Note that the captured image 50 here includes not only an image taken as a single independent still image by operating a shutter button, but also an image corresponding to one frame in a moving image.

  The storage 3 stores design data of a three-dimensional CAD model that defines the three-dimensional shape of the product. For example, a storage device such as a hard disk drive (HDD), a solid state drive (SSD), a nonvolatile memory, or a removable medium. It is. The three-dimensional CAD model of the product includes vertex information, edge information (line information), surface information, and the like for defining the three-dimensional shape of the product. The storage 3 can store not only design data but also information of the captured image 50 acquired by the camera 2. That is, the captured image 50 may be stored in the storage 3 in advance and the superimposed display may be performed using the captured image 50. In this case, the camera 2 can be omitted.

  The input device 4 is an input device of the information processing apparatus 1 and is, for example, a keyboard, a mouse, a track pad, or the like. The input device 4 includes a panel type touch sensor (touch panel) attached to the surface of the display device 5. The display device 5 is an output device of the information processing device 1, and is, for example, a liquid crystal display (LCD) or an organic EL display (Organic Electro-Luminescence Display, OELD).

  The information processing apparatus 1 incorporates a processor 41 (central processing unit), a memory 42 (main memory, main storage device), an auxiliary storage device 43, an interface device 44, a recording medium drive 45, and the like, via an internal bus 46. They are communicably connected to each other. The processor 41 is a central processing unit that incorporates a control unit (control circuit), an arithmetic unit (arithmetic circuit), a cache memory (register group), and the like. The memory 42 is a storage device that stores programs and various data during work, and includes, for example, a ROM (Read Only Memory) and a RAM (Random Access Memory). The auxiliary storage device 43 is a memory device that stores various data and firmware that are held for a longer time than the memory 42, and includes, for example, a nonvolatile memory such as a flash memory and an EEPROM.

  The interface device 44 controls input / output (I / O) between the information processing apparatus 1 and the outside. The information processing apparatus 1 is connected to the camera 2, the storage 3, the input device 4, the display device 5, and the like via the interface device 44. Further, the recording medium drive 45 is a reading device (or a device having a function of reading information recorded on and stored in a recording medium 47 (removable medium) such as an optical disk or a semiconductor memory (portable flash drive conforming to the Universal Serial Bus standard). Reading / writing device). The program executed by the information processing apparatus 1 may be recorded and stored in the memory 42, or may be recorded and stored in the auxiliary storage device 43 and the recording medium 47.

[3. Software configuration]
FIG. 3 is a block diagram showing a functional configuration of the superimposed display program 6 executed by the information processing apparatus 1. The superimposed display program 6 includes a camera image processing unit 10, a design data processing unit 20, and a superimposition processing unit 30. The camera image processing unit 10 is in charge of processing related to the captured image 50 of the product imaged by the camera 2, and the design data processing unit 20 is in charge of processing related to the perspective projection image generated from the design data of the product. is there. The superimposition processing unit 30 is in charge of superimposing display processing of the former captured image 50 and the latter perspective projection image.

  Each of these elements shows the functions of the superimposed display program 6 for convenience, and each element may be described as an independent program, or described as a composite program having these functions. May be. The superimposed display program 6 is recorded and stored in the memory 42 or the auxiliary storage device 43. Alternatively, the superimposed display program 6 is recorded and stored on the recording medium 47, and the superimposed display program 6 written on the recording medium 47 is read into the information processing apparatus 1 via the recording medium drive 45 and executed.

[3-1. Camera image processing unit]
The camera image processing unit 10 includes a control unit 11, an image storage unit 12, an image display unit 13, and a first extraction unit 14. The control unit 11 acquires the captured image 50 of the product by controlling the shooting state of the camera 2. The image storage unit 12 temporarily stores the data of the captured image 50. In addition, when using the information of the captured image 50 preserve | saved at the storage 3, you may memorize | store the information (photograph data) in the image memory | storage part 12 instead of the data of the captured image 50 temporarily. Hereinafter, the name “captured image 50” is simply used as a general term for these.

  The image display unit 13 displays the captured image 50 on the display device 5 based on the data of the captured image 50 stored in the image storage unit 12. The display by the image display unit 13 is not intended to be superimposed and intended to allow the user to confirm the captured image 50. Here, the captured image 50 is displayed on the display device 5 with a size and a position that allow the user to grasp at least the state of the product shown in the captured image 50. A screen of the display device 5 on which the captured image 50 is displayed is illustrated in FIG.

  The first extraction unit 14 extracts the first straight line group 54 corresponding to the ridgeline of the product based on the captured image 50. The first straight line group 54 is a set of straight lines (straight lines in a two-dimensional plane) defined in the plane coordinate system in the captured image 50. The first straight line group 54 is, for example, subjected to known edge detection processing (Sobel processing, Laplacian processing, Canny processing, etc.) on the captured image 50 to generate an edge image, and then known public line detection processing (Hough transform or probabilistic Hough transform). Etc.) can be extracted. The information of the first straight line group 54 extracted here is transmitted to the superimposition processing unit 30.

[3-2. Design data processing unit]
The design data processing unit 20 includes a reading unit 21, a generation unit 22, a data display unit 23, a change unit 24, a second extraction unit 25, a three-dimensional conversion unit 26, a division unit 27, and a two-dimensional conversion unit 28. The reading unit 21 acquires design data information stored in the storage 3. The production | generation part 22 produces | generates the 1st projection image 51 of a product based on the design data and 1st external parameter of a product. The first external parameter is one of external parameters representing the viewpoint position and the line-of-sight direction when the product is perspectively projected. The initial value of the first external parameter is set in advance, but the value can be changed as will be described later. As shown in FIG. 5A, the first projection image 51 here may be a perspective projection diagram of a polygon from which hidden lines are erased, or a perspective projection diagram expressed only by a wire frame. Also good.

  The generation unit 22 has a function of generating a first projection image 51 generated from each of the normal image, the depth image, and the grayscale image. These normal image, depth image, and grayscale image may be generated using, for example, a scan line method (scanning method), or may be generated (calculated) using a ray tracing method or a ray casting method. 5B to 5D are drawing examples of the normal image, the depth image, and the gray image of the first projection image 51, respectively. A normal image is colored by associating red, green, and blue primaries with the x, y, and z coordinate values of the normal vector on the surface of the subject (product). It is an image that was made. A depth image is an image colored according to the distance from the viewpoint position. A grayscale image is a monochromatic multi-tone image.

  The data display unit 23 displays the first projection image 51 generated by the generation unit 22 on the display device 5. The data display unit 23 of the present embodiment causes the display device 5 to display a perspective projection view as shown in FIG. 5A among the various first projection images 51. Instead of this perspective projection diagram, normal images, depth images, and grayscale images as shown in FIGS. 5B to 5D may be displayed on the display device 5. The display by the data display unit 23 is not intended to be superimposed in the same manner as the display by the image display unit 13, but is intended to allow the user to confirm the perspective projection view. A screen of the display device 5 on which this perspective projection view is displayed is illustrated in FIG.

  The changing unit 24 changes the first external parameter referred to when the generating unit 22 generates the first projection image 51. The changing unit 24 changes the first external parameter based on a preset rule or based on a user input operation. The change unit 24 of the present embodiment provides a UI (User Interface) that allows the user to input and change the first external parameter. For example, the user can intuitively change the viewpoint position and the line-of-sight direction of the first projection image 51 by operating a mouse or a keyboard.

  A mouse pointer 53 in FIG. 4 indicates that mouse input for changing the position, scale (enlargement / reduction ratio), rotation direction, rotation amount, and the like of the first projection image 51 is possible. For example, the display position is changed by dragging the first projection image 51 with the mouse (moving the position of the mouse pointer 53 while left-clicking). It is assumed that the scale is changed by dragging. In addition, when the mouse is right-clicked and dragged in the vertical direction in a blank area other than the first projection image 51, the first projection image 51 is rotated in the vertical direction, and the mouse is right-clicked and dragged in the left-right direction. Thus, it is assumed that the first projection image 51 rotates in the left-right direction. The value of the first external parameter changed here is reflected in real time on the first projection image 51 generated by the generation unit 22.

  The second extraction unit 25 extracts the second straight line group 55 corresponding to the three-dimensional ridgeline of the product based on the first projection image 51 generated by the generation unit 22. The second straight line group 55 is a set of straight lines (straight lines in a two-dimensional plane) defined in the plane coordinate system in the first projection image 51. The second straight line group 55 is, for example, generated by performing known edge detection processing (Sobel processing, Laplacian processing, Canny processing, etc.) on the first projection image 51 and then generating known edge detection processing (Hough transform or probability). It is possible to extract by performing a general Hough transform. FIG. 5E is a diagram showing the second straight line group 55 extracted from the normal image shown in FIG.

  The second straight line group 55 of the present embodiment extracts the second straight line group 55 using at least one of a normal image, a depth image, and a grayscale image from the first projection image 51 generated by the generation unit 22. . Note that with respect to a grayscale image, a portion where an edge cannot be detected (a portion where the edge detection accuracy is significantly reduced) may occur depending on the illumination position at the time of rendering. Therefore, when using a grayscale image, it is preferable to extract the second straight line group 55 using another image (normal image or depth image) in combination. The same applies to the normal image and the depth image, and it is preferable to use a plurality of images in combination rather than using a single image. By using the three images of the normal image, the depth image, and the gray image together, the extraction accuracy of the second straight line group 55 is remarkably improved.

  The second straight line group 55 can also be extracted by inputting the first projection image 51 to the first extraction unit 14. Therefore, one of the first extraction unit 14 and the second extraction unit 25 may be omitted, and the two functions may be integrated into the remaining one. Alternatively, by providing both the first extraction unit 14 and the second extraction unit 25, the extraction of the first straight line group 54 and the extraction of the second straight line group 55 may be processed simultaneously in parallel.

  The three-dimensional conversion unit 26 converts the second straight line group 55 into a three-dimensional third straight line group 56 based on the first external parameter. The conversion from the second straight line group 55 (two-dimensional) to the third straight line group 56 (three-dimensional) can be realized by, for example, back projection processing using the first external parameter. The three-dimensional conversion unit 26 of the present embodiment sets two points located on the straight lines included in the second straight line group 55. Further, the three-dimensional coordinates of each point are calculated from the coordinates (two-dimensional coordinates) of each point in the planar coordinate system in the first projection image 51, the depth information of each point, and the first external parameter, and a three-dimensional straight line is obtained. Identify. By repeating such calculation for each straight line, information on the third straight line group 56 including a plurality of three-dimensional straight lines can be obtained.

  In the conversion from the second straight line group 55 to the third straight line group 56, a plurality of points (two-dimensional) on the two-dimensional straight lines included in the second straight line group 55 are converted into three-dimensional points by back projection conversion. The third straight line group 56 may be obtained while dividing the straight line based on the shape change rate of the line segment connecting them. By referring to the shape change rate of the line segment, it becomes easy to recognize the curved ridge line as a plurality of straight lines (as a broken line).

  The dividing unit 27 removes, from the third straight line group 56, a part of the straight line included in the third straight line group 56 that is separated from the surface of the actual product (a part not corresponding to the ridgeline of the actual product). Is. Here, for example, inappropriate parts derived from erroneous extraction by the second extraction unit 25 are excluded from the third straight line group 56. As shown in FIG. 6 (A), a case where the direction of the arrow is the line-of-sight direction will be described for a cubic product in which a recess is formed on one side that forms the boundary between the top surface and the side surface. The normal directions of the top surface of the cube and the bottom surface of the dent are assumed to be substantially the same. In the first projection image 51 (normal image) shown in FIG. 6B, the shape of the dent appears to be smaller than actual. As a result, the ridge line that is actually divided into two via the dent can be erroneously extracted as one straight line 61.

  Based on such a problem, the dividing unit 27 divides a straight line included in the third straight line group 56 into a plurality of points 62 (or line segments), and whether or not design data exists in the vicinity of each point 62. Determine whether. If the design data does not exist in the vicinity, the point 62 is removed, and if it exists, the point 62 is left. At this time, if the distance from the point 62 to the design data is equal to or greater than the predetermined distance, it is determined that “the design data does not exist in the vicinity”. Thereby, as shown in FIG. 6D, the point 62 of the part corresponding to the dent is deleted. Thereafter, the continuity of the plurality of remaining points 62 is evaluated to recombine with a straight line. The straight line 63 in FIG. 6 (E) corresponds to the recombination of the point 62 located on the right side of the deleted portion, and the straight line 64 is the recombination of the point 62 located on the left side. It corresponds to a thing. By repeating such an operation, a plurality of three-dimensional straight lines according to the model shape of the design data can be acquired.

  The two-dimensional conversion unit 28 projects and converts the third straight line group 56 from which inappropriate parts are removed by the dividing unit 27 into a two-dimensional straight line group again. The two-dimensional straight line group is two-dimensionalized for display on the display device 5, and is substantially the same as the third straight line group 56. Here, the third straight line group 56 is converted into a two-dimensional straight line group based on the first external parameter. Information on the two-dimensional line group calculated here is transmitted to the superimposition processing unit 30.

[3-3. Superimposition processing unit]
The superimposition processing unit 30 includes a ridge line display unit 31, an acquisition unit 32, a calculation unit 33, and a superimposition display unit 34. The ridge line display unit 31 displays the first straight line group 54 and the third straight line group 56 on the display device 5 in a selectable state. In the present embodiment, a two-dimensional line group that is substantially the same information as the third line group 56 is displayed on the display device 5. For example, as shown in FIG. 4, the first line group 54 is displayed so as to be superimposed on the captured image 50, and the third line group 56 (two-dimensional line group) is displayed so as to be superimposed on the first projected image 51. Thereby, the ridge line in the captured image 50 and the ridge line in the first projection image 51 are emphasized, and the user can easily select the correspondence.

  The acquisition unit 32 acquires a correspondence relationship between the first straight line group 54 and the third straight line group 56 based on a user's selection operation. The acquisition unit 32 allows the user to input which straight line included in the first straight line group 54 corresponds to which straight line included in the third straight line group 56 (which is a straight line pair). I will provide a. For example, the mouse pointer in FIG. 4 is moved to any straight line in the first line group 54 and left-clicked, and then the mouse pointer is moved to any straight line in the corresponding third line group 56. By left-clicking, the relationship between a pair of straight lines is specified. The acquisition unit 32 of the present embodiment acquires at least four or more sets of correspondences in order to grasp where the three-dimensional coordinates on the design data are perspective-transformed in the captured image 50. By specifying at least four correspondence relationships, the perspective transformation can be calculated. The information on the correspondence acquired here is transmitted to the calculation unit 33.

  Based on the correspondence acquired by the acquisition unit 32, the calculation unit 33 calculates a second external parameter that represents a viewpoint position and a line-of-sight direction equivalent to the imaging position and imaging direction of the captured image 50. The perspective projection image based on the design data and the second external parameter is an image that matches the captured image 50. Therefore, by calculating the second external parameter, it is possible to accurately grasp to which coordinate of the captured image 50 the three-dimensional coordinates on the design data are to be perspectively transformed. The information of the second external parameter calculated here is transmitted to the superimposed display unit 34.

  The superimposed display unit 34 generates a second projected image 52 of the product based on the design data and the second external parameter, and renders the superimposed image on the captured image 50. As described above, by grasping the viewpoint position and line-of-sight direction equivalent to the imaging position and imaging direction of the captured image 50, a perspective projection image that matches the captured image 50 can be generated. FIG. 7 is a diagram illustrating a captured image 50 and a second projection image 52 superimposed thereon. By such superposition display, it becomes easy for the user to quickly and visually find the difference between the captured image 50 and the second projection image 52, and the accuracy and work efficiency of product inspection are improved.

[4. flowchart]
FIG. 8 is a flowchart illustrating the procedure of the superimposed display method in the superimposed display program 6. First, in the camera image processing unit 10, the first straight line group 54 corresponding to the ridgeline of the product is extracted based on the captured image 50 (step A1). On the other hand, the design data processing unit 20 reads the initial value of the first external parameter (step A2), and generates the first projection image 51 based on the first external parameter and the design data (step A3). . In the present embodiment, as shown in FIGS. 5A to 5D, a plurality of first projection images 51 are generated. Of these first projection images 51, the perspective perspective view of the polygon shown in FIG. 5A is displayed to assist the user in inputting and changing the first external parameter.

  Of the first projection image 51, the normal image, the depth image, and the gray image shown in FIGS. 5B to 5D are used for extracting the second straight line group 55. That is, the second straight line group 55 corresponding to the ridgeline of the product is extracted based on the first projection image 51 (step A4). Thereafter, the two-dimensional second straight line group 55 is converted into the three-dimensional third straight line group 56 by back projection processing using the first external parameter (step A5). Thereby, a three-dimensional straight line corresponding to the ridgeline on the design data is grasped.

  In addition, among the straight lines included in the third straight line group 56, the part separated from the surface of the product is excluded from the third straight line group 56, and the remaining part is recombined (step A6). Thereby, an inappropriate part derived from erroneous extraction by the second extraction unit 25 is removed from the third line group 56, and the extraction accuracy of the third line group 56 is improved. Thereafter, the third straight line group 56 from which inappropriate portions have been removed is projected and converted into a two-dimensional straight line group (step A7), and the first straight line group 54 and the third straight line group 56 are selectable in the display device 5 in a selectable state. This is displayed (step A8), and the input by the user is accepted (step A9). The user can change the first external parameter and select the correspondence relationship between the first straight line group 54 and the third straight line group 56 while comparing the first straight line group 54 and the third straight line group 56. .

  Subsequently, it is determined whether or not the first external parameter has been changed by the user (step A10). When this condition is satisfied, the flow returns to step A3, and the first projection image 51 is regenerated. As a result, the first projected image 51 having a different viewpoint position and line-of-sight direction is drawn, and the second straight line group 55 and the third straight line group 56 are recalculated accordingly. Further, regarding the correspondence relationship between the first straight line group 54 and the third straight line group 56, it is determined whether or not at least four pairs of correspondence relationships have been selected by the user (step A11).

  If this condition is not satisfied, the flow returns to step A3. On the other hand, when this condition is satisfied, the second external parameter is calculated (step A12), and the second projected image 52 is drawn based on the design data and the second external parameter (step A13). At this time, the viewpoint position and the line-of-sight direction of the second projection image 52 are equivalent to the imaging position and the imaging direction of the captured image 50. Therefore, the second projection image 52 is displayed in a superimposed manner in a state that substantially matches the captured image 50.

[5. Action, effect]
(1) In the above-described embodiment, the first projection image 51 is generated based on the design data, the second line group 55 corresponding to the ridge line is extracted from the first projection image 51, and the three-dimensional third line group 56 is extracted. Has been converted. By performing such processing, the extraction efficiency of the ridge lines in the design data can be improved. For example, it is possible to extract a ridge line even at an edge with a fillet (roundness, round) that cannot be detected as a ridge line by an existing method. Therefore, the accuracy of superimposed display can be improved.

  In addition, the correspondence between the straight line portion (two-dimensional) in the captured image 50 and the ridge line portion (three-dimensional) in the design data can be appropriately specified. That is, it is possible to perform superimposed display even on an object for which it has been difficult to extract a ridge line with the existing technology, and it is possible to expand the application target of the superimposed display technology. Furthermore, even when the number of polygons included in the design data is large, the calculation load for extracting the second straight line group 55 from the two-dimensional first projection image 51 is the calculation load for three-dimensional ridge line extraction. Small compared to Therefore, the calculation time can be shortened and the calculation efficiency can be improved as compared with the existing method.

  (2) In the above-described embodiment, as shown in FIG. 4, the first straight line group 54 and the third straight line group 56 are displayed in a selectable state, and the user is allowed to select a correspondence (pair). By adopting such a method, it becomes easy to obtain an appropriate correspondence between the straight line groups, and the estimation accuracy of the second external parameter can be improved. In addition, it is possible to provide a user-friendly UI that is intuitive and easy to understand, improving convenience, and improving the accuracy and work efficiency of product inspection.

  (3) In the above-described embodiment, the first projection image 51 is updated each time the first external parameter is changed, and the third straight line group 56 is updated each time. Thereby, while rotating the 1st projection image 51, the correspondence of the 1st straight line group 54 and the 3rd straight line group 56 can be confirmed, and while being able to improve convenience, the precision and work efficiency of product inspection Can be increased.

  (4) In particular, in the above-described embodiment, the user can input the first external parameter of the first projection image 51. Thereby, it is possible to easily identify the correspondence relationship between the first straight line group 54 and the third straight line group 56. For example, when the first projection image 51 is displayed in the viewpoint position / line-of-sight direction that is accurately superimposed on the captured image 50, if the straight line to be selected is located in the vicinity of another straight line, the straight line is selected well It is thought that it cannot be done. Even in such a case, the user can slightly rotate the first projection image 51 to facilitate selection of a desired straight line. Therefore, the convenience can be improved and the accuracy and work efficiency of product inspection can be improved.

  (5) In the above-described embodiment, as shown in FIG. 4, not only the third straight line group 56 but also the three-dimensional shape of the product corresponding to the first external parameter is displayed. As a result, the result of changing the first external parameter (how the first projected image 51 looks) can be easily shown to the user, and the viewpoint position and the line-of-sight direction can be accurately changed. it can. Further, the shape from which the third straight line group 56 is extracted can be displayed in an easy-to-understand manner for the user, and the viewpoint position and the line-of-sight direction can be changed with high accuracy.

  (6) In the above-described embodiment, as shown in FIGS. 5B to 5D, the first projection image 51 is generated as at least one of a normal image, a depth image, and a grayscale image. Thereby, the second straight line group 55 can be extracted quickly and accurately with a simple calculation configuration, and the extraction accuracy of the third straight line group 56 can be improved.

  (7) In the above-described embodiment, the part separated from the surface of the product is excluded from the third straight line group 56 for the straight lines included in the third straight line group 56. Thereby, the straight line included in the third straight line group 56 becomes a shape along the three-dimensional shape in the data, and the correspondence relationship between the first straight line group 54 and the third straight line group 56 can be correctly selected.

  (8) In the above-described embodiment, the straight line included in the third straight line group 56 is divided into a plurality of points, points where the distance to the object is equal to or greater than a predetermined distance are removed, and the remaining points are recombined. The As a result, as shown in FIG. 6E, a plurality of three-dimensional straight lines according to the model shape of the design data can be acquired, and the extraction accuracy of the third straight line group 56 can be further improved.

[5. Modified example]
This embodiment is merely an example, and there is no intention to exclude various modifications and technical applications that are not explicitly described in the above embodiment. That is, the above-described embodiment can be variously modified (for example, by combining the embodiment and the modified examples) without departing from the spirit of the embodiment.

  For example, in the above-described embodiment, the control for extracting the second straight line group 55 and the third straight line group 56 based on the first external parameter set by the user has been described in detail, but the setting and selection of the first external parameter are automated. May be. For example, it is conceivable that the user sets an initial value of the first external parameter, and thereafter, a plurality of external parameters having different values of the first external parameter are automatically generated and presented to the user. In this case, a normal line image, a depth image, and a grayscale image may be generated for each of a plurality of automatically generated external parameters, and the second straight line group 55 may be extracted. By automating the setting of the first external parameter, the user's workload can be reduced.

  In addition, the data display unit 23 in the above-described embodiment causes the display device 5 to display a perspective projection view as shown in FIG. 5A among the various first projection images 51. In addition, A normal image, a depth image, a grayscale image, or the like may be displayed on the display device 5 (or alternatively). That is, the data display unit 23 only needs to display the first projection image 51 of the object or the three-dimensional shape of the object according to the first external parameter.

[6. Addendum]
The following additional notes are disclosed regarding the embodiment including the above-described modification.
(Appendix 1-8: Information processing device)
(Appendix 1)
A first extraction unit that extracts a first straight line group corresponding to a ridgeline of the object based on a captured image of the object;
A generating unit that generates a first projection image of the object based on data defining a three-dimensional shape of the object and a first external parameter representing a viewpoint position and a line-of-sight direction when the object is perspective-projected;
A second extraction unit that extracts a second straight line group corresponding to the ridge line based on the first projection image;
Based on the first external parameter, a three-dimensional conversion unit that converts the second line group into a three-dimensional third line group;
Based on the correspondence between the first straight line group and the third straight line group, a calculation unit that calculates a second external parameter that represents a viewpoint position and a line-of-sight direction equivalent to the imaging position and imaging direction of the captured image;
An information processing apparatus comprising: a superimposed display unit that renders a second projection image of the object superimposed on the captured image based on the data and the second external parameter.

(Appendix 2)
A ridge line display unit for displaying the first straight line group and the third straight line group on a display device in a selectable state;
The information processing apparatus according to claim 1, further comprising: an acquisition unit that acquires a correspondence relationship between the first straight line group and the third straight line group based on a user's selection operation.

(Appendix 3)
A changing unit for changing the first external parameter;
The information processing apparatus according to appendix 1 or 2, wherein the generation unit updates the first projection image every time the first external parameter is changed.

(Appendix 4)
The information processing apparatus according to appendix 3, wherein the changing unit has a function of causing the user to input the first external parameter.

(Appendix 5)
5. The information processing apparatus according to claim 1, further comprising a data display unit configured to display a first projected image of the object or a three-dimensional shape of the object according to the first external parameter. .

(Appendix 6)
The information processing apparatus according to any one of appendices 1 to 5, wherein the generation unit generates the first projection image using at least one of a normal line image, a depth image, and a grayscale image.

(Appendix 7)
In any one of appendices 1-6, the straight line included in the third straight line group includes a dividing unit that excludes the part separated from the surface of the object from the third straight line group. The information processing apparatus described.

(Appendix 8)
The dividing unit divides a straight line included in the third straight line group into a plurality of line segments, removes the line segments whose distance to the three-dimensional shape is equal to or greater than a predetermined distance, and reuses the remaining line segments. The information processing apparatus according to appendix 7, wherein the information processing apparatus is combined.

(Appendix 9-16: Superimposition display program)
(Appendix 9)
Based on the captured image of the object, a first straight line group corresponding to the ridgeline of the object is extracted,
Based on data defining the three-dimensional shape of the object and a first external parameter representing a viewpoint position and a line-of-sight direction when the object is perspectively projected, a first projection image of the object is generated,
Based on the first projection image, a second straight line group corresponding to the ridge line is extracted,
Based on the first external parameter, the second line group is converted into a three-dimensional third line group,
Based on the correspondence between the first straight line group and the third straight line group, a second external parameter representing a viewpoint position and a line-of-sight direction equivalent to the imaging position and imaging direction of the captured image is calculated,
A superimposed display program for causing a computer to execute a process of rendering a second projected image of the object on the captured image based on the data and the second external parameter.

(Appendix 10)
The first straight line group and the third straight line group are displayed on a display device in a selectable state,
The superimposed display program according to appendix 9, wherein the computer executes a process of acquiring a correspondence relationship between the first straight line group and the third straight line group based on a user's selection operation.

(Appendix 11)
Changing the first external parameter;
The superimposed display program according to appendix 9 or 10, which causes a computer to execute a process of updating the first projection image each time the first external parameter is changed.

(Appendix 12)
The superimposed display program according to appendix 11, which causes a computer to execute a process for causing the user to input the first external parameter.

(Appendix 13)
The superimposed display program according to any one of appendices 9 to 12, which causes a computer to execute a process of displaying a first projection image of the object or a three-dimensional shape of the object according to the first external parameter.

(Appendix 14)
14. The superimposed display program according to any one of appendices 9 to 13, which causes a computer to execute a process of generating the first projection image using at least one of a normal image, a depth image, and a grayscale image.

(Appendix 15)
The superposition according to any one of appendices 9 to 14, which causes a computer to execute a process of excluding a part separated from the surface of the object from the third straight line group for the straight line included in the third straight line group. Display program.

(Appendix 16)
A computer that divides a straight line included in the third straight line group into a plurality of line segments, removes the line segments whose distance to the three-dimensional shape is a predetermined distance or more, and recombines the remaining line segments. The superimposed display program according to appendix 15, which is executed by the program.

(Appendix 17-24: Superimposition display method)
(Appendix 17)
Based on the captured image of the object, a first straight line group corresponding to the ridgeline of the object is extracted,
Based on data defining the three-dimensional shape of the object and a first external parameter representing a viewpoint position and a line-of-sight direction when the object is perspectively projected, a first projection image of the object is generated,
Based on the first projection image, a second straight line group corresponding to the ridge line is extracted,
Based on the first external parameter, the second line group is converted into a three-dimensional third line group,
Based on the correspondence between the first straight line group and the third straight line group, a second external parameter representing a viewpoint position and a line-of-sight direction equivalent to the imaging position and imaging direction of the captured image is calculated,
A superimposed display method comprising: rendering a second projection image of the object superimposed on the captured image based on the data and the second external parameter.

(Appendix 18)
The first straight line group and the third straight line group are displayed on a display device in a selectable state,
18. The superimposed display method according to appendix 17, wherein a correspondence relationship between the first straight line group and the third straight line group is acquired based on a user's selection operation.

(Appendix 19)
Changing the first external parameter;
19. The superimposed display method according to appendix 17 or 18, wherein the first projection image is updated each time the first external parameter is changed.

(Appendix 20)
The superimposed display method according to appendix 19, wherein the user inputs the first external parameter.

(Appendix 21)
The superimposed display method according to any one of appendices 17 to 20, wherein the three-dimensional shape of the object corresponding to the first projection image of the object or the first external parameter is displayed.

(Appendix 22)
The superimposed display method according to any one of appendices 17 to 21, wherein the first projection image is generated by at least one of a normal image, a depth image, and a grayscale image.

(Appendix 23)
The superimposed display according to any one of appendices 17 to 22, wherein a part of the straight line included in the third straight line group is excluded from the third straight line group. Method.

(Appendix 24)
Dividing a straight line included in the third straight line group into a plurality of line segments, removing the line segments whose distance to the three-dimensional shape is a predetermined distance or more, and recombining the remaining line segments. The superimposed display method according to attachment 23.

1 Information processing equipment (computer)
2 camera 3 storage 4 input device 5 display device 6 superimposed display program 10 camera image processing unit 11 control unit 12 image storage unit 13 image display unit 14 first extraction unit 20 design data processing unit 21 reading unit 22 generation unit 23 data display unit 24 change unit 25 second extraction unit 26 three-dimensional conversion unit 27 division unit 28 two-dimensional conversion unit 30 superimposition processing unit 31 ridge line display unit 32 acquisition unit 33 calculation unit 34 superimposition display unit 50 captured image 51 first projection image 52 second Projected Image 54 First Line Group 55 Second Line Group 56 Third Line Group

Claims (10)

A first extraction unit that extracts a first straight line group corresponding to a ridgeline of the object based on a captured image of the object;
A generating unit that generates a first projection image of the object based on data defining a three-dimensional shape of the object and a first external parameter representing a viewpoint position and a line-of-sight direction when the object is perspective-projected;
A second extraction unit that extracts a second straight line group corresponding to the ridge line based on the first projection image;
Based on the first external parameter, a three-dimensional conversion unit that converts the second line group into a three-dimensional third line group;
Based on the correspondence between the first straight line group and the third straight line group, a calculation unit that calculates a second external parameter that represents a viewpoint position and a line-of-sight direction equivalent to the imaging position and imaging direction of the captured image;
An information processing apparatus comprising: a superimposed display unit that renders a second projection image of the object superimposed on the captured image based on the data and the second external parameter. A ridge line display unit for displaying the first straight line group and the third straight line group on a display device in a selectable state;
The information processing apparatus according to claim 1, further comprising: an acquisition unit configured to acquire a correspondence relationship between the first straight line group and the third straight line group based on a user's selection operation. A changing unit for changing the first external parameter;
The information processing apparatus according to claim 1, wherein the generation unit updates the first projection image every time the first external parameter is changed. The information processing apparatus according to claim 3, wherein the changing unit has a function of causing the user to input the first external parameter. 5. The information processing according to claim 1, further comprising a data display unit that displays a first projection image of the object or a three-dimensional shape of the object according to the first external parameter. apparatus. The information processing apparatus according to claim 1, wherein the generation unit generates the first projection image using at least one of a normal line image, a depth image, and a grayscale image. The straight line included in the third straight line group is provided with a dividing unit that excludes the part separated from the surface of the object from the third straight line group. The information processing apparatus described in 1. The dividing unit divides a straight line included in the third straight line group into a plurality of points, removes the points where the distance to the object is a predetermined distance or more, and recombines the remaining points. The information processing apparatus according to claim 7, wherein the information processing apparatus is characterized. Based on the captured image of the object, a first straight line group corresponding to the ridgeline of the object is extracted,
Based on data defining the three-dimensional shape of the object and a first external parameter representing a viewpoint position and a line-of-sight direction when the object is perspectively projected, a first projection image of the object is generated,
Based on the first projection image, a second straight line group corresponding to the ridge line is extracted,
Based on the first external parameter, the second line group is converted into a three-dimensional third line group,
Based on the correspondence between the first straight line group and the third straight line group, a second external parameter representing a viewpoint position and a line-of-sight direction equivalent to the imaging position and imaging direction of the captured image is calculated,
A superimposed display program for causing a computer to execute a process of rendering a second projected image of the object on the captured image based on the data and the second external parameter. Based on the captured image of the object, a first straight line group corresponding to the ridgeline of the object is extracted,
Based on data defining the three-dimensional shape of the object and a first external parameter representing a viewpoint position and a line-of-sight direction when the object is perspectively projected, a first projection image of the object is generated,
Based on the first projection image, a second straight line group corresponding to the ridge line is extracted,
Based on the first external parameter, the second line group is converted into a three-dimensional third line group,
Based on the correspondence between the first straight line group and the third straight line group, a second external parameter representing a viewpoint position and a line-of-sight direction equivalent to the imaging position and imaging direction of the captured image is calculated,
A superimposed display method comprising: rendering a second projection image of the object superimposed on the captured image based on the data and the second external parameter.