JP6262610B2 - Information registration device, information continuation registration device, method and program - Google Patents

Description

  In order to present the operation / inspection / repair method of the target device to the worker using the display and augmented reality technology, the information related to the operation / inspection / repair method is registered in association with the target device. The present invention relates to a suitable information registration device, information continuation registration device, method and program.

  When an operator operates, inspects, or repairs equipment, there is a need to work while confirming information (such as a manual) on the target equipment. On the other hand, there is a conventional method in which the work is completely relied upon only by hand while sequentially referring to a paper-based manual related to the target device. On the other hand, since the method is excessively dependent on human resources, information on the target device is stored as electronic data, and a device (for example, a display) different from the target device is used to display the target device on the display. A work support system that provides information superimposed on an image of a work site has been proposed.

  In particular, when targeting communication devices such as routers, there are often multiple ports with similar shapes, so that operators can visually check, repair, and repair in addition to referencing them manually. It is desirable to be able to confirm the work site.

  Patent Document 1 uses a device having a camera and a display, which is different from the target device, identifies the target device and the work site by reading a marker pasted in advance with the camera, and further displays the display. Discloses a method in which augmented reality display data (operation method and the like) registered in advance for a work site of a target device is expressed by superimposing a real image.

JP 2013-258703 A (Information Providing System)

  The prior art of Patent Document 1 realizes recognition of a work site of a target device by using an image recognition technology, and visually displays a transition of the movement of the device with respect to the operation of the target device. However, there has been no presentation of a specific method regarding a registration method for registering augmented reality display data or the like for presenting the operation method of the target device in association with the target device.

  Regarding the registration method, there are various situations that cause the following problems.

  First, as a premise, it is assumed that the target device is basically already in operation. Furthermore, it is assumed that a marker such as a QR code (registered trademark) is pasted later to determine a reference point for calculating the relative position of the augmented reality display data to be presented to the target device.

  In this case, in order to present the work site in augmented reality, it is necessary to calculate and store the relative position of the augmented reality information with respect to the marker. As one method for accurately calculating the relative position between the marker and the device, there is a method in which the marker is actually attached to the apparatus, the marker and the work site are photographed, and geometrically calculated from the image. At this time, the following problems (1) and (2) occur.

  (1) Since the target device is currently used, a plurality of other devices may already be connected to the device. In this case, since a part of the work part for displaying the augmented reality information is hidden, it may be difficult to specify the work part by the method of calculating geometrically from the image.

  (2) In the case as described above, if the two-dimensional position of the marker and the work part is simply saved, there is a difference between the relative position of the marker and the work part when observed from a viewpoint other than the saved imaging position. And augmented reality information may not be correctly superimposed on the work site.

  FIG. 1 is a photograph example (with description) of a target device that causes the problems (1) and (2). The target device is a communication device as an example, and a marker is attached to the surface of the casing as shown in (a). Since it is already in operation, as shown in (b), in the area that is the work target such as the switching of the wiring in the port, the wiring is already applied in the port that can be the work target. There are ports that are partially hidden, ports that are not wired themselves, but that are blocked by a portion of the portion that is visible when another portion of the wiring crosses.

  As described above, since the portion that has already been operated and hidden by wiring and the like appears, the problem (1) occurs.

  Furthermore, FIG.2 and FIG.3 is a figure for demonstrating the subject (2) typically. FIG. 2 is a schematic diagram when the communication device as shown in FIG. 1 is viewed from the front. The marker M1 is pasted on the surface P1 of the housing, and the port areas W1, W2, W3 to be worked on. It is shown. In FIG. 2 (and FIG. 3), the wiring that blocks the port region is not shown.

  Here, the port areas W1, W2, and W3 to be worked on can be treated as two-dimensional areas when viewed from the front as shown in FIG. 2, but actually protrude from the plane P1. It is a three-dimensional shape. This is shown in FIG. FIG. 3 is a view of FIG. 2 as viewed from the front when viewed from an oblique direction.

  As shown in FIG. 3, the port areas W1, W2, and W3 are rectangular parallelepipeds, for example, and it is necessary to specify the upper surface portion that protrudes from the surface P1 on which the marker M1 is placed as the work area. However, if it is registered in the form of two-dimensional position information in the plane P1 as shown in FIG. 2, when viewed from an oblique direction as described above, it is geometrically determined from the registered two-dimensional position information. Even if it is calculated, the augmented reality display cannot be presented at the correct position. For example, in the case where it looks like FIG. 3, when augmented reality display is performed using the two-dimensionally registered position information, the plane P1 is not formed on the upper surface portion of the port areas W1, W2, W3 but on the lower surface thereof. The display is performed for the part in contact with.

  That is, the position information is registered on the premise that the work area exists in the plane P1 of the marker M1 in the framework of the prior art, but the actual work target is a three-dimensional object such as the ports W1, W2, and W3. Therefore, the augmented reality display based on the position information registered on the assumption that it exists on the plane P1 when viewed from an angle that makes a solid with respect to the plane P1 cannot be used as an inappropriate display. turn into.

  As described above, in the problem (2), it is desired to register the position information on the plane P1 as the upper surface portions of the ports W1, W2, W3, etc. are on the plane away from the plane P1, but in the conventional technique, Registration as a dimensional coordinate is a premise, and no such registration framework is provided. The above-mentioned problem (1) can occur as a further problem when trying to solve the problem (2).

  As described above, there is another problem (3) in addition to the problems (1) and (2).

  That is, consider a case where there are a plurality of the same (same type) target devices. In this case, it is desirable that the marker is a different marker for identifying each target device. However, since the presentation information such as the operation method of the device may be the same, the registration information of the operation method is the same. It is desirable that can be diverted. This causes the following problem (3).

  (3) When a marker is pasted on a plurality of devices later, augmented reality display data is an actual work site because it is difficult to accurately paste the marker at a specific position on the apparatus. May not superimpose well.

  In other words, when displaying the augmented reality data or the like by using the already registered location information for the target device of the same type as it is for the new target device, it has already been registered for the new target device to be used. It is necessary to attach a marker at the same position as the marker that is pasted on the target device. However, especially when the marker is attached manually, there is a limit to accuracy, and accurate attachment is impossible, and the attachment position will be deviated to some extent. As a result, the augmented reality data or the like presented in the diverted new target device is not presented at the original position due to the influence of the pasting deviation.

  FIG. 4 is a diagram for schematically explaining the problem (3). The result of registering the position information of the port areas W1, W2, and W3 with respect to the marker M1 in FIG. The deviation which occurs when diverting to equipment is shown. The marker should be pasted at position M1, but as a result of sticking to position M10 with a shift to the upper right side and a counterclockwise rotation, it shifted from the original port area W1, W2, W3. Port regions W11, W12, and W13 are recognized as positions. The work area is also shifted from P1 to P10. Even if an attempt is made to display augmented reality data or the like using this result, the display is made at a shifted position.

  In the present invention, in relation to the above problems (1) and (2), when registering the position of the work site of the target equipment in order to display augmented reality data or the like, the work site deviates from the surface of the target equipment. The first object of the present invention is to provide an information registration apparatus, method and program capable of registering the position of a work part even in a situation where a work part may be partially hidden .

  In addition, the present invention relates to the above-mentioned problem (3), and the information continuation registration apparatus and method for enabling the result already registered by the information registration apparatus or method to be used in the registration of another target equipment of the same type. The second purpose is to provide a program.

  In order to achieve the above first object, the present invention relates to a target facility in which the work site in the target facility is deviated from the surface of the target facility and there may be a structure that partially hides the work site. An information registration device for registering the position of a work part for display, acquiring an image obtained by photographing a target facility to which a marker is attached in association with a depth map, and using a camera reference corresponding to each pixel position in the image An imaging unit that associates spatial coordinates, a marker recognizing unit that recognizes the position of the marker from the image, an instruction input unit that receives an input of a range of a work part in the image, and each point of the input range The three-dimensional position calculation unit for classifying the input data based on the spatial coordinates, and by causing the user to select one corresponding to the work site from among the classified results, A relative position determination unit for registering the position of the working part in the selected image as the position of the working part with reference to the recognized marker, as a spatial coordinate of the range of the working part in the selected image is present at the selected location. To do.

  Further, the present invention relates to the position of the work part for the augmented reality display with respect to the target equipment in which the work part in the target equipment is deviated from the surface of the target equipment and there may be a structure that partially hides the work part. An image registration step for acquiring an image obtained by photographing a target facility to which a marker is attached in association with a depth map, and for associating a camera-based spatial coordinate corresponding to each pixel position in the image; A marker recognition stage for recognizing the position of the marker from the image, an instruction input stage for accepting an input of a range of a work part in the image from a user, and each point of the input range based on its spatial coordinates In the input image, the three-dimensional position calculation stage to be classified and the user selects the one corresponding to the work site from the classified results. As business sites ranging spatial coordinates are present in the selected portion, characterized in that it comprises a relative position determining step of registering as the position of the work site relative to the said recognized marker.

  Further, the present invention is a program for causing a computer to execute the information registration method.

  In order to achieve the second object described above, the present invention provides information on a work site that has already been registered by the information registration device with respect to the first target facility to which a marker has been assigned. An information continuation registration device for diverting to target equipment and registering for display of augmented reality, a registration information storage unit for storing information on the already registered work site, and a second target equipment with a marker attached An image obtained by associating with a depth map, a photographing unit for associating a camera-based spatial coordinate corresponding to each pixel position in the image, a marker recognizing unit for recognizing the position of the marker from the image, A registration information display unit that displays the range of the already registered work site on the image with the recognized marker plane as a reference, and the already registered unit displayed by the registration information display unit. An instruction input unit that accepts information for correcting the range of the work site to the range of the work site in the image from the user, and for each point of the range input as the information to be corrected, based on the spatial coordinates, And a three-dimensional position adjustment unit that identifies a work site to be registered by identifying a range determined to match the spatial coordinates of the image.

  Further, the present invention applies the information on the work site that has already been registered by the information registration method with respect to the first target facility provided with the marker to the second target facility of the same type as the first target facility. An information continuation registration method for registering for display, wherein a registration information storing step for storing information on the already registered work part is associated with an image obtained by imaging a marker-attached second target facility with a depth map. And taking a camera-based spatial coordinate corresponding to each pixel position in the image, a marker recognition stage for recognizing the marker position from the image, and the recognized marker plane as a reference A registration information display stage for displaying a range of the already registered work parts on the image, and a range of the already registered work parts displayed by the registration information display stage. An instruction input step for accepting information to be corrected to the range of the work site in the image from the user, and for each point in the range input as the information to be corrected, the spatial coordinates of the image are matched with the spatial coordinates. And a three-dimensional position adjustment step of specifying a work site to be registered by specifying a range determined to be correct.

  Furthermore, the present invention is a program for causing a computer to execute the information continuation registration method.

  According to the information registration device or method or program of the present invention, the camera-referenced spatial coordinates corresponding to each pixel position are acquired by acquiring an image of the target facility to which the marker is attached in association with the depth map, and the marker position In addition, the range of the work site is set on the image in response to the user input, and each point within the set range is classified based on the spatial coordinates. Even if the part deviates from the surface of the target equipment or there is a structure that partially hides the work part, the original work part can be cut out by the classification, and the original cut out by the user Can be selected, and augmented reality data or the like can be registered for the selected portion specified by the marker criterion.

  According to the information continuation registration apparatus or method or program of the present invention, when registering information on a work part already registered by the information registration apparatus or method with respect to the same type of target equipment, the target equipment to be registered In this image, the already registered work part is displayed on the basis of the marker plane, the range of the work part in the image is received in a form that is corrected by the user for the display, and the range of the range that is input in the form of the correction By identifying the range determined to match the coordinates in the spatial coordinates of the image of the target equipment with respect to the coordinates, the target equipment is based on the coincidence of the shape of the work site reflected in the spatial coordinates The work site in can be specified. Therefore, from the user's standpoint, the work site information can be diverted to another target facility only by a correction input for fine adjustment.

It is a photograph example (with description) of the object apparatus which produces a subject (1) (2). It is a figure explaining a subject (2) typically, and is a typical figure at the time of seeing an object apparatus from the front. It is a figure explaining a subject (2) typically, and is a mimetic diagram showing the three-dimensional effect in a work area when the object apparatus is seen from the slant. It is a figure which illustrates subject (3) typically. It is a functional block diagram of the information registration device concerning a first embodiment. It is a figure which shows the marker detected with respect to the example of the image of FIG. It is a figure which shows the example of writing of the four corners which a user designates in an apparatus range input part. It is a figure which shows the example which a user writes and designates the area | region of a work part by an instruction | indication input part. FIG. 9 is an enlarged view of the vicinity of the work site specified in FIG. 8. FIG. 6 (and FIG. 7 and FIG. 8) is a conceptual diagram of a plan view (viewed from above) when the device is viewed as an elevation view (corresponding photograph), and the clustering result is schematically shown. It is a figure for demonstrating. It is a figure which shows the example of the screen at the time of selecting a plane candidate with the arbitrary viewpoints which a user sets. It is a functional block diagram of the information continuation registration apparatus concerning a second embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. First, a first embodiment related to issues (1) and (2) will be described.

  FIG. 5 is a functional block diagram of the information registration apparatus according to the first embodiment. The information registration device 10 includes an imaging unit 11, a marker recognition unit 12, a device range input unit 13, an instruction input unit 14, a three-dimensional position calculation unit 15, a relative position determination unit 16, and an information registration unit 17. In FIG. 5, the information acquired by the functional unit upstream of the arrow between the functional units can be used by any downstream functional unit.

  Details of each part are as follows.

  The imaging unit 11 captures the target equipment that is provided with the marker and includes the work site, and acquires the image data. At this time, a depth map is also acquired. As shown in the example of FIG. 1 (a), the target equipment to be photographed has markers attached in advance at appropriate positions.

Here, the depth map can be acquired by a known method such as a method based on measurement of an existing infrared sensor or a method based on corresponding point detection of a stereo camera in which two or more visible light cameras are arranged in parallel. As a specific device capable of acquiring the depth map, for example, a tablet disclosed in the following URL equipped with a camera and a depth sensor can be used.
http://techcrunch.com/2014/06/05/googles-project-tango-tablet-is-a-1024-7-inch-tegra-k1-powered-device-with-depth-sensing/

  Further, at the time of acquisition by the photographing unit 11, a correspondence between the image by the visible light camera and the position (u, v) on the pixel of the depth map is reserved. That is, for an arbitrary pixel (u, v) (pixel), in addition to the color pixel value (RGB), the depth map d (u, v) is converted, so that the actual value with the camera center as the reference value is obtained. The position (x, y, z) (mm) on the space coordinates can be used. For example, x is rightward with respect to the camera plane, y is upward with respect to the camera plane, and z is coordinate information regarding the depth with respect to the camera plane. For example, the RGBDemo method described in Non-Patent Document 1 below can be used for associating the coordinates (x, y, z) with the reference to the visible light camera and the depth map d (u, v). Note that z = d (u, v).

[Non-Patent Document 1]: RGBDemo
http://labs.manctl.com/rgbdemo/index.php/Documentation/KinectCalibrationTheory

Subsequently, the marker recognizing unit 12 detects the marker from the image acquired by the photographing unit 11. As the marker detection method, for example, a method using ARToolKit described in Non-Patent Document 2 below can be used.
[Non-Patent Document 2]: ARToolKit: http://www.hitl.washington.edu/artoolkit/

  FIG. 6 shows the detected markers for the example image of FIG. For the detected marker, for example, position information (ui, vi, xi, yi, zi; i = 1, 2, 3, 4; position on a two-dimensional image and three-dimensional position from the camera) is stored. To do. Note that marker detection is possible only with the value of the pixel (u, v), and the camera linked in advance by the photographing unit 11 at the positions of the four corners of the marker detected as the position of the pixel (u, v). By adding the reference three-dimensional position (x, y, z), the above information (ui, vi, xi, yi, zi) can be obtained.

  The device range input unit 13 allows the user to input the entire position of the device (target equipment) with respect to the marker for the image recognized by the marker recognition unit 12, and the two-dimensional position (u, v) on the pixel. ) And the corresponding three-dimensional position (x, y, z). Here, the input is performed by the user via the pixel (u, v), and the corresponding three-dimensional position (x, y, z) linked in advance by the photographing unit 11 is automatically acquired.

  Specifically, for example, an image is displayed on a display, and the user is allowed to specify four corners of the device by using a touch panel or a mouse. FIG. 7 shows an example of writing at the four corners designated by the user. In FIG. 7, four corners are designated by a key shape written by the user as shown in [1] to [4] in the image detected by the marker shown in [0] shown in FIG. 6. Note that the device range input unit 13 may be omitted.

  The instruction input unit 14 obtains the marker recognition unit after obtaining it by the imaging unit 11 in order to designate a work part desired by the user (or set in advance in a work manual or the like) as a part for displaying augmented reality or the like. An input of a relative position with respect to the marker is received from the user for the image that has undergone the processing of 12 or the like.

  Specifically, an image is displayed on the display, and the user can write position information by designating a series of pixels (u, v) corresponding to the work site using a touch panel or a mouse. Since the marker position has already been recognized, the relative position to the marker is automatically acquired by the designation.

  In order to specify the work site, a predetermined geometric model such as a rectangle, a circle, an ellipse or the like is available, and the user selects a desired geometric model and performs various edits. A work site as a series of areas is designated on the pixel (u, v). As editing, translation, rotation, enlargement, projective transformation, and the like can be used, and the parameters can be adjusted by the user with a slide bar or the like. In addition, it is possible to perform editing such as deletion (cancellation from the designation target) from the already designated area. If the shape of the work site is determined in advance, it is preferable to prepare the shape as a geometric model whose parameters can be adjusted.

  FIG. 8 shows an example (an example corresponding to a series of examples from FIG. 6 onward) in which the user writes and designates the region of the work site by the instruction input unit 14. As shown in [5], the work site is specified here in the form of a rectangular frame. The rectangular frame may be prepared and input in advance as a geometric model, or it may be input by designating the entire large rectangle and then deselecting the small rectangle from the inside. Alternatively, a combination of the four sides of the frame that are input as long and thin rectangles may be combined.

  In the example of FIG. 8, it is assumed that the port of the work site has a shape in which the outer shape is rectangular and the edge of the rectangular frame is formed by having a rectangular recess inside. In addition, the work site is designated by a hollow rectangular frame. If the work site does not have such a hollow structure, the work site may be designated as a rectangle without a hole, for example, and may be designated according to the shape of each individual specific work site.

  FIG. 9 is an enlarged view of the vicinity of the work site specified in FIG. As can be seen from FIG. 9, the above-mentioned problem (1) occurs. That is, the presence of a cable in front of a port within the specified range hides part of the port, and the entire port surface is not photographed. In addition, since such a port is a three-dimensional shape that cannot be seen because it is taken from the front in FIG. 9, the above-described problem (2) also occurs.

  In this way, in order to appropriately find a port area as a work part having a three-dimensional structure that may be partially hidden, the three-dimensional position calculation unit 15 uses a user writing area acquired by the instruction input unit 14. Cluster classification is performed on a series of three-dimensional points (x, y, z) represented by (u, v). Cluster classification for spatial coordinates (x, y, z) makes it possible to separate the parts separated by the three-dimensional structure in the user writing area (u, v).

  As a cluster classification method, classification can be performed automatically using threshold classification or the K-means method of a general unsupervised cluster classification method. The number K of classifications may be fixed (for example, K = 4) or may be selected by the user. Alternatively, a plurality of K may be tried, and a statistically calculated value such as K in which a value obtained by dividing the sum of variances between classes of each cluster by the sum of variances within the class of each cluster may be used.

  In addition, when the port is a type having an irregularity such as a male type female type as in the examples of FIGS. 8 and 9, the point on the rectangular frame region that fits the type can be accurately used. The 3D position of the part can be registered. Even in the case of other shapes, the three-dimensional position of the target part can be accurately registered by using a shape that fits appropriately.

  At the time of the clustering, in one embodiment, the three-dimensional position calculation unit 15 calculates a marker plane from the three-dimensional position of the marker position previously detected by the marker recognition unit 12, and a group of planes parallel to the marker plane (in parallel) Plane group that can be determined). That is, classification is performed in such a way that when a series of points at the classified three-dimensional position are plane-fitted, it can be determined that they are parallel to the marker plane on a predetermined basis.

  In this embodiment, the work site for which the position information is to be registered is on a plane parallel to the plane (a plane that can be determined to be parallel based on a predetermined reference) that is separated from the plane of the target facility on which the marker is attached. It is assumed that In addition, it is assumed that the marker plane matches the surface of the target facility by being attached (can be determined to match on a predetermined standard).

  As an embodiment for classifying into a plane group parallel to the marker plane, for example, the depth value z is clustered as a value representing each point instead of the value of the three-dimensional coordinates (x, y, z) (only the depth value z). (Based on clustering), it can be classified into a plane group parallel to the marker plane. Here, as a premise, it is assumed that the photographing unit 11 attaches a marker to the front surface of the target facility and photographs from the front.

  Further, as an embodiment for classifying into a plane group parallel to the marker plane, clustering may be performed according to the position in the direction of the axis perpendicular to the marker plane. That is, the three-dimensional coordinates (x, y, z) of each point are projected onto an axis perpendicular to the marker plane (projected by moving in a horizontal direction on the marker plane), and based on the position on the axis Each point may be clustered. In addition, when the imaging | photography part 11 image | photographs the marker plane from the front, the said embodiment corresponds with embodiment clustered by said depth value z.

  As described above, in each embodiment classified into the plane group parallel to the marker plane, the plane detection method may be, for example, that the plane is statistically estimated from the three-dimensional positions of the four corners of the marker using the least square method or the like. Alternatively, the plane may be statistically estimated using a least square method or the like using a series of three-dimensional positions in the marker region.

  In addition, when there is an input of the device area by using the device range input unit 13, statistics are obtained by using the least-squares method or the like from the three-dimensional positions of the four corners or from the whole of the four corners as described above. Alternatively, the result of estimating the plane may be used as the marker plane.

  Although the marker plane is not particularly used, the following is possible as an embodiment in which clustering is performed in consideration of the spatial distribution of each point of the three-dimensional coordinates (x, y, z).

  In one embodiment, principal component analysis (PCA) is performed on a point cloud of three-dimensional coordinates (x, y, z) to calculate a two-dimensional axis that is a principal component, with the first axis as a reference. You may make it cluster the point group of a three-dimensional coordinate. The clustering in the predetermined axial direction may be performed in the same manner as described in the clustering in the direction of the axis perpendicular to the marker plane described above for the direction. Note that PCA makes the first axis and the second axis vertical.

  In one embodiment, the clustering may be performed in the axial direction of the designated angle after the user designates a clustering angle by referring to a plan view or the like by manual work. The clustering in the predetermined axial direction may be performed in the same manner as described in the clustering in the direction of the axis perpendicular to the marker plane described above for the direction. As for the plan view, the same thing as that shown in FIG. 10 described below may be presented to the user. Or you may show a user the figure of the arbitrary viewpoints like FIG. 11 mentioned later instead of a top view. The user can directly specify a desired direction with reference to the presented three-dimensional diagram.

  FIG. 10 is a conceptual diagram of a plan view (viewed from above) when FIG. 6 (and FIGS. 7 and 8) is regarded as an elevation view of the device (a corresponding photograph). 6 corresponds to a state in which the device is seen straight from the front surface to be worked on, whereas FIG. 10 shows the device viewed from directly above so that it can be viewed in a direction parallel to the front surface of the device. It is a conceptual diagram of the state which is in order to explain a clustering result typically. Here, the case where it classify | categorizes into the plane group parallel to a marker plane is taken as an example.

  In FIG. 10, a marker plane is shown in [1], and a portion surrounded by dotted circles [2] to [4] represents the attribution of each of a series of three-dimensional coordinates to be clustered, [5] Shows an explanation of the clustering result. The 3D coordinate points corresponding to the rectangular framework entered by the user shown in [3] are actually the device plane in [2] (which matches the marker plane because the marker is pasted) and the original port in [3] And the cable part that prevents the original port part of [4] from being seen, and as shown in [5] Are properly classified into the original parts reflecting the three-dimensional structure (planes C1, C2, and C3 of the clustering result).

  That is, as shown in [5], with reference to the marker plane, the plane is classified into three planes (planes C1, C2, and C3 of the clustering result) parallel to the marker plane, and the result is output. The standard value (initial value) of the output may be a plane that is the shortest distance from the marker plane (here, the plane C1 that actually matches the marker plane P1).

  Note that the classification result as schematically shown in FIG. 10 can be obtained by designating a certain extent area on the pixel (u, v) by the user input in the instruction input unit 14 with respect to the original work area to be extracted. Will be obtained.

  The relative position determination unit 16 presents a plurality of work site plane candidates output from the three-dimensional position calculation unit 15 to the user, and allows the user to select a plane of the work site. Determine the plane that will be the work site. When selection by the user is omitted, the standard value (initial value) of the output by the above-described three-dimensional position calculation unit 15, that is, the plane having the shortest distance from the marker plane may be selected.

  When the user selects a plane, the relative position determination unit 16 displays the range of the work site that the user inputs as the range on the image (u, v) in the instruction input unit 14 on the marker plane as in the registration in the prior art. Instead, the spatial coordinates (x, y, z) of the range of the work site are determined as being on the plane selected by the user. The determined spatial coordinates (x, y, z) are registered in the information registration unit 17, and augmented reality display or the like can be performed based on the position information.

  Here, the spatial coordinates (x, y, z) of the determined range of the work site are set with reference to the marker plane, but are not on the marker plane. For example, in the example of FIG. 3, when the front surface of the port W1 protruding from the plane P1 is registered as the work site range, the position is determined with reference to the plane P1 (spatial coordinates determined by the marker M1). Information will be set. If the plane selected by the user matches the marker plane, as a result, it is registered that the work site is on the marker plane (the work site is on the marker plane) as in the prior art. Will be.

  It should be noted that the marker plane and the marker position (such as the four corners thereof) are calculated by the three-dimensional position calculation unit 15 and the like as those in the camera-based spatial coordinates (x, y, z), and the plane selected by the user is the same. Thus, the relative position determination unit 16 may calculate the value within the camera-based spatial coordinates (x, y, z). Using the information on the two planes, the spatial coordinates (x, y, z) of the range of the work site corresponding to the area input as the area on the pixel (u, v) in the instruction input unit 14 is referred to the marker plane. In addition, it can be set as being on the plane selected by the user on the basis of the marker to enable the augmented reality display.

  When the user makes the selection, various display formats can be used. For example, in the form of a plan view as shown in FIG. 10, the user can select from the planes C1, C2, and C3 as shown in [5]. In this case, the fact that the plane C1 is coincident with the marker plane can also be grasped by the user by displaying as shown in FIG. Note that the plan view as shown in FIG. 10 can be created from the camera reference coordinate values (x, y, z) acquired by the photographing unit 11.

  For example, when the plane C2 of FIG. 10 is selected for the port (working part) input as the rectangular frame region (u, v) as shown in FIGS. 8 and 9, the rectangular frame region (u, v) is selected. ) Are all on the plane C2, and their three-dimensional coordinates (x, y, z) are registered. The same applies when another shape (u, v) is entered as the work site. The three-dimensional coordinates (x, y, z) of the registered work part are given as being on some plane parallel to the marker plane according to the marker reference coordinates, so that the region of the registered work part The augmented reality display and the like can be performed.

  If the marker plane and the plane of the work part are not parallel, the user may arbitrarily designate the three-dimensional position of the work part based on the cluster classification result that can be confirmed from the above plan view. The relative position between the marker plane and the work site is stored as a position in the three-dimensional space, for example, by setting a three-dimensional space with the upper left corner of the marker plane as the origin.

  In addition to the plan view format as shown in FIG. 10, the clustering result can be selected by the user by actually arranging a plurality of classified plane candidates in the space according to their three-dimensional coordinate values. You may do it. That is, the plan view corresponds to a state where the front surface of the target facility is viewed from directly above as a specific viewpoint, but the plane candidate may be selected by the user according to a state viewed from an arbitrary viewpoint that can be set by the user. Good. The display according to the state viewed from the arbitrary viewpoint is well known in the field of 3D CAD and the like, and the user can freely move the arbitrary viewpoint to grasp each plane candidate in the space.

  FIG. 11 shows an example of a screen when plane candidates are selected from the arbitrary viewpoint set by the user. The example of FIG. 11 corresponds to the example of FIG. In FIG. 11, the planes C1, C2, and C3 of the clustering result are spatially displayed. The plane C1 is spatially displayed to be common with the marker plane P1 (the plane C1 is in the marker plane P1).

  Further, when the planes C1, C2, and C3 displayed at an arbitrary viewpoint as shown in FIG. 11 are presented, pixels (pixels as RGB values or the like) acquired by the corresponding photographing unit 11 are pasted on the surfaces. Thus, the user may be able to grasp what each plane corresponds to on the actual photograph. In this way, the user can actually confirm the photograph of the front part of the target equipment on the three-dimensional plane C1, the photograph of the port part on the plane C2, and the photograph of the cable part on the plane C3. The plane C2 can be selected with certainty. In addition, a photograph of the front part of the target facility may be pasted on the marker plane P1 so that confirmation can be made more reliably.

Note that, as a technique for pasting a portion of a photograph corresponding to the surface, a technique disclosed in Non-Patent Document 3 below may be used.
[Non-Patent Document 3]: PCL-Point Cloud Library (PCL) [URL; pointcloud.org]

  The information registration unit 17 is selected by the relative position determination unit 16, and the work site is determined by user input with respect to the work site whose spatial relative position (and range) is determined with reference to the marker. Register work instructions and other information related to work. In the registration, information to be displayed may be registered in a form that enables augmented reality display or the like. Information related to work may be automatically acquired from a separate server etc. by entering only the work ID etc. instead of being manually input by the user. Good.

  In the above, each part of FIG. 5 was demonstrated. When a user performs registration regarding each work site of one actual target facility, the following may be performed. That is, after acquiring an image with a depth map for the target equipment by the imaging unit 11 and performing marker recognition by the marker recognition unit 12, for each work site to be registered for one image recognized by the marker The instruction input unit 14, the three-dimensional position calculation unit 15, the relative position determination unit 16, and the information registration unit 17 may be executed. The device range input unit 13 can be used in common in each of the work sites to register the information if it is executed once for one image recognized by the marker.

  In the information registration device 10, (Restriction 1) When the work site in the target facility deviates from the surface of the target facility, and (Restriction 2) When there is a structure that partially hides the work site, Although it is possible to register information on the work site, even if only one of the constraints 1 and 2 is imposed, or if neither of the constraints 1 and 2 is imposed, It is possible to register work site information by a similar operation.

  When only one of the constraints 1 and 2 is imposed, the number of clustering results is reduced as compared with the case where both are imposed. If neither of the constraints 1 and 2 is imposed, the number of clustering results is further reduced.

  Next, a second embodiment related to the problem (3) will be described. As described in FIG. 4 regarding the problem (3), the second embodiment assumes the following. That is, by the information registration apparatus 10 of the first embodiment, the relative position with respect to the marker M1 attached to one or more work parts Ri (i = 1, 2,..., N) of the existing target equipment E1 is RM1i. (i = 1, 2,..., n) is determined, and information related to work or the like is registered at the determined position.

  And in 2nd embodiment, it aims at the next realization. In other words, in the target equipment E2 of the same type (the manufacturer model number is the same and the same shape) as the registered target equipment E1, it is almost the same as when the user manually pasted the marker M1 to the target equipment E1. When the same type of marker M2 is attached to the position, the information of the already determined relative position RM1i (i = 1, 2,..., N) is used.

  At this time, if the information of the relative position RM1i (i = 1, 2,..., N) is used as it is, as described in FIG. 4, an error when the marker M2 is manually attached to the equipment E2 (the marker M1 is installed in the equipment). (The error with the position pasted on E2) is reflected as it is, and the position for performing augmented reality display or the like shifts. Therefore, the information of the relative position RM1i (i = 1, 2,..., N) is diverted to make use of the contents of the already registered work so as to eliminate the error (positional deviation) associated with the diversion. This is the second embodiment.

  FIG. 12 is a functional block diagram of the information continuation registration apparatus according to the second embodiment. The information continuation registration device 20 includes an imaging unit 21, a marker recognition unit 22, a registration information display unit 29, a device range comparison unit 23, an instruction input unit 24, a three-dimensional position adjustment unit 25, a relative position determination unit 26, and an information registration unit 27. And a registration information storage unit 28. In FIG. 12, the information acquired by the functional unit upstream of the arrow between the functional units can be used by any downstream functional unit.

  As described above with reference to the information registration apparatus 10 in FIG. 5, the information continuation registration apparatus 20 in FIG. 12 applies to one or more work parts Ri (i = 1, 2,..., N) of the existing target equipment E1. Thus, the relative position with respect to the attached marker M1 is determined as RM1i (i = 1, 2,..., N), and it is used on the assumption that information related to work or the like is registered at the determined position. When pasting the marker M2 on the already registered relative position RM1i (i = 1, 2,..., N) and the corresponding information registered in the information registration unit 17 on another equipment E2 of the same type as the equipment E1 In addition, the registration information storage unit 28 stores the information so that it can be used.

  In the functional blocks of the information continuation registration apparatus 20 in FIG. 12 and the information registration apparatus 10 in FIG. 5, those with reference numbers corresponding to each other (the last one digit is common) are the same or corresponding functions. Take on. The same device (such as a computer) can also function as both the information registration device 10 and the information continuation registration device 20.

  Details of each part of the information continuation registration apparatus 20 are as follows.

  The imaging unit 21 is the same as the imaging unit 11 in terms of its function, and acquires image data including a work site in the target facility E2 to which the marker M2 is attached as an image acquisition target. Similarly, the depth map d (u, v) is also acquired at the same time, and the camera-based spatial coordinates (x, y, z) are obtained.

  Subsequently, the marker recognizing unit 22 detects a marker region from the visible light image obtained by the photographing unit 21 by the same function as the marker recognizing unit 12.

  Further, the device range comparison unit 23 presents an instruction input screen to the user when the device range is input by the device range input unit 13 of the corresponding information registration device 10, and registers information also for the facility E2. The input of the target device range is accepted in the same manner as the processing in the device 10. When the device range is not set by the device range input unit 13, this operation is omitted.

  In the registration information display unit 29, the position information (x, y, z) (position information as a spatial coordinate based on the marker M1 plane) previously registered in the information registration apparatus 10 for the equipment E1 is used as the information. For each work site read from the saved registration information storage unit 28 and using the marker M2 plane as a reference, the work site on the pixel (u, v) on the image of the target equipment E2 acquired by the imaging unit 21 By displaying this area, it is displayed to the user. The display is for user input in the instruction input unit 24 described below.

  Here, in the registration information holding unit 28, since the three-dimensional position of the work site is stored with the marker M1 plane as a reference point, by applying a well-known perspective projection transformation, each work site with respect to the marker M2 plane as a reference Displays the saved 3D position as the area occupied by the pixel (u, v) on the screen (screen for displaying to the user, which shows the image of the target equipment E2 acquired by the imaging unit 21) it can. That is, if the marker M1 reference work part of the equipment E1 stored in the registered information storage unit 28 is assumed to have the position and orientation of the marker M2 in the target equipment E2, the position and orientation are linked to each other. Since the transformation relationship that the image looks like this by changing is given by the perspective projection transformation, the display as described above is possible.

  In addition, when the target equipment E2 newly provided with the marker M2 is photographed by the photographing unit 21, and when the target equipment E1 provided with the marker M1 stored in the registration information storage unit 28 is photographed by the photographing unit 11, In addition to the difference in the positions of the markers M1 and M2 assigned to the facilities E1 and E2, there is a difference in the camera position and orientation at the time of shooting. The display using the perspective projection conversion in the registration information display unit 29 naturally absorbs the difference in camera position and orientation, and the difference in camera position and orientation does not cause a problem. On the other hand, the problem due to the difference in the positions of the markers M1 and M2 is solved by the following processing after the instruction input unit 24.

  The instruction input unit 24 allows the user to correct the display position of the work site displayed by the registration information display unit 29 in the image where the target equipment E2 is captured.

  Note that FIG. 4 described above is also a diagram illustrating a schematic example of the screen displayed on the registration information display unit 29 when the user makes the correction using the instruction input unit 24. In FIG. 4, for example, when the work site W1 is already stored in the registration information storage unit 28 based on the marker M1, the registration information display unit 29 adds a new marker M10 (corresponding to the marker M2 in the above description). The region W11 is displayed as a work site with reference to. In this case, the user performs an input via the instruction input unit 24 to correct the displayed area W11 so as to match the area W1 that is an actual work site area as much as possible.

  The correction of the display position received as user input by the instruction input unit 24 is mainly based on translation and rotation on the marker plane, and can be adjusted by the user himself. In this way, the position information (shifted) read from the registered information storage unit 28 and displayed with reference to the marker M2 is adjusted by the user so that it overlaps the work site on the image of the equipment E2. The user finishes correcting the work site.

  With this correction, the instruction input unit 24 uses the positional relationship on the pixel (u, v) instead of the three-dimensional coordinates (x, y, z) as information on the corrected region of the work part based on the marker M2. You will receive it. Note that the adjustment and determination of the position of the three-dimensional coordinates (x, y, z) of the work part with reference to the marker M2 as the finally necessary information are performed in the three-dimensional position adjustment unit 25 and later described later. .

  FIG. 4 schematically shows the position on the plane and the shift due to the rotation on the plane when the marker is applied on the surface of the target facility. The markers M1 and M2 are manually attached to the same location on the corresponding surfaces of the same type of target equipment E1 and E2, but there is a displacement due to the position on the plane and rotation between them. It will be. The instruction input unit 24 corrects such a deviation (with an accuracy possible by user input).

  Further, when the device range is set by the device range comparison unit 23, for example, the least square method is used based on the three-dimensional position information based on the respective markers M1 and M2 of the respective devices E1 and E2. Thus, the fitting between the three-dimensional position information of the devices E1 and E2 may be performed, and the instruction input unit 24 may replace the manual correction by the user. As is clear from the fact that the same markers M1 and M2 are attached to the same shape surface (device range surface), the fitting of the three-dimensional position information between the devices E1 and E2 is performed between the markers M1 and M2. This information is consistent with the fitting of the dimensional position information, and reflects the difference in position / orientation where the markers M1 and M2 are pasted and the difference in shooting position and orientation in the shooting units 11 and 21 that shot them.

  The instruction input unit 24 acquires correction information based on user input on the pixel (u, v), but instead of acquiring correction information in the device range comparison unit 23, as described above, The correction information is acquired directly on the three-dimensional coordinates (x, y, z), not on the pixel (u, v). Information on the position fitting between the markers M1 and M2 is obtained in the form of perspective projection transformation.

  In the three-dimensional position adjustment unit 25, the three-dimensional position of the target part of the target equipment E2 is compared with the augmented reality information position that has been aligned two-dimensionally (on the pixel position (u, v)) by the instruction input unit 24. Adjust the position (x, y, z).

  From the depth map data (that is, camera reference coordinates (x, y, z) data) acquired by the imaging unit 21 for the target device E2 to be registered by diversion, the device E1 is registered in advance and stored in the registration information storage unit 28. The depth map is subjected to filter processing by using the position information of the work part with reference to the marker M1 (mainly, the distance of the three-dimensional position between the plane of the marker M1 and the position information of the work part). For example, if the position information of the previously registered work part is present at a distance d (if there is a distribution, the average value may be taken) with respect to the marker M1 plane, the marker is obtained from the depth map data of the target device E2 to be registered. Only depth information at a distance of d ± ε with respect to the M2 plane is selected and used by filtering. ε is an arbitrarily small value.

  Fitting is performed between the depth map information having a value after the filtering process and a series of spatial coordinates (x, y, z) corresponding to the position information (u, v) adjusted by the instruction input unit 24. The range occupied by the three-dimensional coordinates (x, y, z) in the fitted part of the depth map information that has a value after the filtering process is the work area of the marker M2 reference in the device E2 to be finally obtained. It becomes a range.

  Since a predetermined shape (for example, FIG. 9) of the work site is input in the instruction input unit 14 of the information registration apparatus 10 for the original device E1 (device diverted by the device E2), the predetermined shape is It is stored in the registration information storage unit 28, and it is expanded by further fitting and assigning the stored predetermined shape to the range occupied by the three-dimensional coordinates (x, y, z) obtained by the fitting. A display area for actual display or the like may be determined.

  For example, an ICP (iterative closest point) algorithm is used for fitting the depth map information and a series of spatial coordinates (x, y, z). As a result, if the plane where the markers M1 and M2 are pasted is the same between the devices E1 and E2, it can be easily positioned even if there is a slight misalignment in the marker pasting position (position misalignment in the direction parallel to the marker plane). The position of information can be corrected, and information registration work is facilitated.

Note that when the device range comparison unit 23 is used in place of the instruction input unit 24, the above processing is the same, and the fitting target may be (B) instead of (A) below.
(A) “A series of spatial coordinates (x, y, z) corresponding to position information (u, v) adjusted by the instruction input unit 24”
(B) “Regarding the position information (x, y, z) of the work part marker M1 stored in the registration information storage unit 28 for the device E1, between the markers M1 and M2 obtained by the device range comparison unit 23 Position information (x, y, z) based on the marker M2 of the work site of device E2 obtained by applying the fitting information.

  This processing is depth map data, that is, camera reference coordinate (x, y, z) data after narrowing down the part corresponding to the original work site by the distance d (reference distance), and the distribution is the work By applying the ICP algorithm to data reflecting the shape of the part, the position is corrected to the part where the shape matches, and the original work part position information is acquired. .

  In particular, the three-dimensional position adjustment unit 25 as the fine movement adjustment is made by adjusting the coarse movement to a position that does not overlap with other work parts by the instruction input unit 24 (or the device range comparison unit 23) in advance. Will function normally. Therefore, if the deviation between the markers M1 and M2 attached to the devices E1 and E2 is sufficiently small, the instruction input unit 24 (or device range comparison unit 23) for coarse motion adjustment is omitted, and the third order from the beginning. The position information can be adjusted even if the original position adjustment unit 25 is made to function.

  The relative position determination unit 26 performs a final confirmation for the user with respect to the position adjusted by the three-dimensional position adjustment unit 25, and if the confirmation that the alignment is complete is obtained, Registration of the position information about is ended. If it is confirmed as a result of the user confirmation that the position is wrong, the registration of the work site may be performed again from the processing of the instruction input unit 24. Alternatively, when it is determined by the user judgment that registration is difficult due to correction, the information registration device 10 may be used to register again from the beginning instead of the information continuation registration device 20. Alternatively, the information continuation registration device 20 may be used again after the position of the pasted marker M2 is adjusted and pasted again.

  Similar to the information registration unit 17, the information registration unit 27 registers work instructions related to the work site and other information related to the work in association with each other. In addition, work instructions and other information for the work site already registered for the device E1 are stored in the registration information storage unit 28, and automatically registered without receiving user input when registering to the device E2. The information registration unit 27 can perform registration. When it is desired to register new information or to correct already registered information, the information registration unit 27 can accept user input and register corresponding information.

  For the device E2, the information continuation registration device 20 can also register the position information and the like for each work site. In this case, the following embodiments are possible. In one embodiment, the position information corrected from the position information registered for the device E1 is registered for a certain work site, and the device E1 in the already registered one work site is registered for the other work sites. The information of the corrected part from the position information may be used as it is and registered by correcting the position information in the same manner.

  In another embodiment, for each work part of the device E2, the information continuation registration device 20 performs registration in a form in which the position information is corrected from the already registered device E1, and then the position information of each work part is stored. An average of corrections may be adopted as final position information registered for each work site of the device E2. In this case, by averaging, the error in the individually corrected position information in each work site is absorbed, so that the position information can be registered robustly.

  The present invention can also be provided as a program that causes a computer to function as the information registration device 10 or the information continuation registration device 20. The computer can have a normal hardware configuration including a CPU (Central Processing Unit), a memory, and various I / Fs, and the CPU executes the functions of each unit of the information registration device 10 or the information continuation registration device 20. be able to.

  DESCRIPTION OF SYMBOLS 10 ... Information registration apparatus, 11 ... Image pick-up part, 12 ... Marker recognition part, 13 ... Device range input part, 14 ... Instruction input part, 15 ... Three-dimensional position calculation part, 16 ... Relative position determination part, 17 ... Information registration part , 20 ... Information continuation registration device, 21 ... Imaging unit, 22 ... Marker recognition unit, 23 ... Device range comparison unit, 24 ... Instruction input unit, 25 ... Three-dimensional position adjustment unit, 26 ... Relative position determination unit, 27 ... Information Registration part, 28 ... Registration information storage part, 29 ... Registration information display part

Claims (11)

Information registration apparatus for registering the position of a work part for augmented reality display with respect to the target equipment in which the work part in the target equipment is deviated from the surface of the target equipment and there may be a structure that partially hides the work part Because
An image that captures the image of the target facility to which the marker is attached is acquired by associating it with the depth map, and an imaging unit that associates the camera-based spatial coordinates corresponding to each pixel position in the image,
A marker recognition unit for recognizing the position of the marker from the image;
An instruction input unit that receives an input of a range of a work part in the image from a user;
A three-dimensional position calculation unit that classifies each point of the input range based on its spatial coordinates;
By letting the user select one corresponding to the work part from among the classified results, the spatial coordinates of the range of the work part in the input image are present at the selected part, And a relative position determination unit that registers the position of the work part with the recognized marker as a reference.   The three-dimensional position calculation unit further calculates a marker plane from the recognized marker position, and classifies each point in the input range so as to form a plane group that can be determined to be parallel to the marker plane. The information registration apparatus according to claim 1.   The said three-dimensional position calculation part classify | categorizes each point of the said input range based on the component of the depth direction with respect to a camera plane among the components of the said camera reference | standard space coordinate. The information registration device described. From the user, further comprising a device range input unit that receives input of the range of the surface of the target equipment in the image,
The information according to claim 2 or 3, wherein the three-dimensional position calculation unit calculates a surface of the target facility whose range is input in the image in space coordinates and replaces the calculated marker plane. Registration device. The information on the work site that has already been registered by the information registration device according to any one of claims 1 to 4 with respect to the first target facility provided with the marker is diverted to a second target facility of the same type as the first target facility. An information continuation registration device that registers for augmented reality display,
A registered information storage unit for storing information on the already registered work site;
An image obtained by linking the image obtained by imaging the second target facility with the marker attached to the depth map, and a camera reference space coordinate corresponding to each pixel position in the image,
A marker recognition unit for recognizing the position of the marker from the image;
On the basis of the recognized marker plane, a registration information display unit that displays the range of the already registered work site on the image;
An instruction input unit that receives from the user information for correcting the range of the already registered work site displayed by the registration information display unit to the range of the work site in the image;
For each point in the range input as the information to be corrected, the work site to be registered is specified by specifying the range determined to match the spatial coordinates of the image based on the spatial coordinates An information continuation registration apparatus comprising: a three-dimensional position adjustment unit. The augmented reality display by diverting the information on the work part already registered by the information registration device according to claim 4 to the second target equipment of the same type as the first target equipment regarding the first target equipment to which the marker is attached An information continuation registration device for registering for
A registered information storage unit for storing information on the already registered work site;
An image obtained by linking the image obtained by imaging the second target facility with the marker attached to the depth map, and a camera reference space coordinate corresponding to each pixel position in the image,
A marker recognition unit for recognizing the position of the marker from the image;
The input of the surface range of the target equipment in the image is received from the user, and the difference in spatial arrangement between the input surface range and the range input to the device range input unit provided in the information registration apparatus is obtained. A device range comparison unit;
The setting of the range of the work part in the image is set by applying the difference in the obtained spatial arrangement as a correction to the information on the already registered work part with reference to the plane of the recognized marker. For each point in the range set in the form of the correction, the work site to be registered is specified by specifying the range determined to match the spatial coordinate of the image based on the spatial coordinates An information continuation registration device comprising: a three-dimensional position adjustment unit.   The three-dimensional position adjustment unit, based on the reference distance between the marker plane and the work part in the already registered work part information, out of the spatial coordinates of the image, the distance between the marker plane and the reference distance 7. The information continuation registration apparatus according to claim 5, wherein the range determined to be matched is specified only for a case where the difference falls within a predetermined threshold criterion. An information registration method for registering the position of a work part for displaying an augmented reality with respect to the target equipment in which the work part in the target equipment is deviated from the surface of the target equipment and there may be a structure that partially hides the work part. Because
An image that captures the image of the target facility to which the marker is attached is obtained by associating it with the depth map, and an imaging stage that associates the camera-based spatial coordinates corresponding to each pixel position in the image,
A marker recognition stage for recognizing the position of the marker from the image;
An instruction input stage for accepting an input of a range of a work part in the image from a user;
A three-dimensional position calculation step of classifying each point of the input range based on its spatial coordinates;
By letting the user select one corresponding to the work part from among the classified results, the spatial coordinates of the range of the work part in the input image are present at the selected part, And a relative position determination step of registering as a position of a work part with the recognized marker as a reference. Augmented reality display by diverting the information on the work site already registered by the information registration method according to claim 8 to the second target facility of the same type as the first target facility with respect to the first target facility to which the marker is attached An information continuation registration method for registering for
A registration information storage step for storing information on the already registered work site;
An image obtained by linking an image obtained by photographing the marker-attached second target facility with a depth map, and linking a camera-based spatial coordinate corresponding to each pixel position in the image;
A marker recognition stage for recognizing the position of the marker from the image;
A registration information display step for displaying a range of the already registered work site on the image with reference to the plane of the recognized marker;
An instruction input step for receiving from the user information for correcting the range of the already registered work site displayed by the registration information display step to the range of the work site in the image;
For each point in the range input as the information to be corrected, the work site to be registered is specified by specifying the range determined to match the spatial coordinates of the image based on the spatial coordinates And a three-dimensional position adjustment step. An information registration method for registering the position of a work part for displaying an augmented reality with respect to the target equipment in which the work part in the target equipment is deviated from the surface of the target equipment and there may be a structure that partially hides the work part. A program for causing a computer to execute
An image that captures the image of the target facility to which the marker is attached is obtained by associating it with the depth map, and an imaging stage that associates the camera-based spatial coordinates corresponding to each pixel position in the image,
A marker recognition stage for recognizing the position of the marker from the image;
An instruction input stage for accepting an input of a range of a work part in the image from a user;
A three-dimensional position calculation step of classifying each point of the input range based on its spatial coordinates;
By letting the user select one corresponding to the work part from among the classified results, the spatial coordinates of the range of the work part in the input image are present at the selected part, A program for causing a computer to execute a relative position determination step of registering as a position of a work part based on the recognized marker. The information on the work site that has already been registered by causing the computer to execute the program according to claim 10 with respect to the first target facility provided with the marker is diverted to the second target facility of the same type as the first target facility. A program for causing a computer to execute an information continuation registration method for registering for augmented reality display,
A registration information storage step for storing information on the already registered work site;
An image obtained by linking an image obtained by photographing the marker-attached second target facility with a depth map, and linking a camera-based spatial coordinate corresponding to each pixel position in the image;
A marker recognition stage for recognizing the position of the marker from the image;
A registration information display step for displaying a range of the already registered work site on the image with reference to the plane of the recognized marker;
An instruction input step for receiving from the user information for correcting the range of the already registered work site displayed by the registration information display step to the range of the work site in the image;
For each point in the range input as the information to be corrected, the work site to be registered is specified by specifying the range determined to match the spatial coordinates of the image based on the spatial coordinates A program for causing a computer to execute a three-dimensional position adjustment step.