JP7390978B2 - Annotation support device and method - Google Patents

Description

TECHNICAL FIELD The present invention relates to an annotation support device and method suitable for annotating, for example, parts of a building that need to be preserved.

BACKGROUND ART Conventionally, 3D scanners have been used to acquire three-dimensional data in projects such as preservation and restoration plans for historical buildings. This saves the state of buildings and components before work as three-dimensional three-dimensional shape data, which can be referenced for comparison after restoration, and also allows three-dimensional structures such as curved ceilings and beams to be saved in three-dimensional shape data. It is acquired as data and used for conservation and restoration plans.

For example, when acquiring data on a historical chapel, the entire space is scanned in several batches using a large 3D scanner, and small objects such as reliefs are scanned using a handheld 3D scanner.

In addition, in the preservation and utilization plan for cultural property buildings, work is being done to determine the parts of the building that need to be preserved and to set protection policies for each part based on five-stage criteria.

On the other hand, a technique described in Patent Document 1, for example, is known as a technique that allows a construction worker to perform construction work on a mirror-blown concrete surface while visually checking the ground condition. In Patent Document 1, an area including the face is photographed with a digital camera to generate a face information image, and the face is mirror-blown with concrete, and the face information image is projected onto the mirror-blown concrete surface. After that, construction information such as the position of the charging hole is determined based on the projected image, display information of the determined construction information is added to the face information image to generate a face information update image, and the face information update image is displayed on the face. It is something that is projected.

Further, as a pointing position detection technique, for example, the technique described in Patent Document 2 is known. This patent document 2 discloses a two-dimensional image display screen, a two-dimensional image light irradiation device that irradiates image light onto the back surface of the two-dimensional image display screen, a pointer that irradiates light onto the front surface of the two-dimensional image display screen, It is equipped with a photographing device that photographs the back side of the dimensional image display screen, and a control device into which the imaging results from the photographing device are input, and the control device detects the light emitted from the photographic results as the pointing position. ing.

Japanese Patent Application Publication No. 2018-150720 Patent No. 4814700

By the way, when determining the preservation of the above-mentioned cultural heritage buildings, the survey staff evaluate the parts on site and record them in field notebooks, which are then matched with materials such as photographs and drawings, and compiled into reports. It required a lot of work to put together, and it took a lot of effort. On the other hand, when obtaining shape data using a 3D scanner, data such as the position and texture of members can be obtained, but it is not possible to annotate parts of a building. That is, it is not possible to add related information (metadata) to data as annotations. For this reason, there is a need for technology that allows annotation work to be performed while observing objects on site.

The present invention has been made in view of the above, and an object of the present invention is to provide an annotation support device and method that can annotate an object while observing it on-site.

In order to solve the above-mentioned problems and achieve the purpose, an annotation support device according to the present invention is a support device for supporting the work of annotating a predetermined part of an object, and includes three parts of the object. A three-dimensional data acquisition means for acquiring dimensional data, an image generation means for generating an image of an object based on the acquired three-dimensional data, and an annotation for annotating a predetermined part of the generated image by a user's operation. and a projection means for projecting the generated image onto the surface of the corresponding object, the image generation means generates an annotated image in which a predetermined part is annotated, and the projection means It is characterized by projecting images with annotations.

In addition, another annotation support device according to the present invention, in the above-described invention, further includes pointing means for pointing to a predetermined part of the object by a user's operation, and the three-dimensional data acquisition means includes a three-dimensional It is characterized by acquiring data.

Further, the annotation support method according to the present invention is a support method for supporting the task of annotating a predetermined part of a target object, and includes a step of acquiring three-dimensional data of the target object, and a step of acquiring three-dimensional data of the target object; a step of generating an image of the object based on the data; a step of annotating a predetermined part in the generated image by a user's operation; generating an annotated image in which the predetermined part is annotated; It is characterized by projecting the generated annotated image onto the surface of the corresponding object.

Further, another annotation support method according to the present invention is characterized in that, in the above-described invention, a predetermined part of the object is pointed by a user's operation, and three-dimensional data of the pointed part is acquired.

According to the annotation support device according to the present invention, it is a support device for supporting the work of annotating a predetermined part of an object, and includes a three-dimensional data acquisition means for acquiring three-dimensional data of the object; An image generating means that generates an image of a target object based on the acquired three-dimensional data; an annotation adding means that annotates a predetermined part of the generated image according to a user's operation; and an object to which the generated image corresponds. The image generating means generates an annotated image in which a predetermined part is annotated, and the projecting means projects the generated annotated image, so that the image can be projected onto the surface of the target object on-site. This has the effect that annotations can be added while observing.

Further, according to another annotation support device according to the present invention, the device further includes pointing means for pointing to a predetermined part of the target object by a user's operation, and the three-dimensional data acquisition means acquires three-dimensional data of the pointed part. Therefore, the annotation work can be carried out while visually checking the pointed part on-site.

Further, according to the annotation support method according to the present invention, there is provided a support method for supporting the work of annotating a predetermined part of a target object, which includes a step of acquiring three-dimensional data of the target object, and a step of acquiring three-dimensional data of the target object; A step of generating an image of the object based on the three-dimensional data, a step of annotating a predetermined part in the generated image by a user's operation, and generating an annotated image in which the predetermined part is annotated. However, since the generated annotated image is projected onto the surface of the corresponding object, it is possible to add annotations while observing the object on-site.

Furthermore, according to another annotation support method according to the present invention, a predetermined part of a target object is pointed by a user's operation, and three-dimensional data of the pointed part is obtained, so that the pointed part can be visually confirmed on site. This has the effect that annotation work can be performed while

FIG. 1 is a schematic configuration diagram showing an embodiment of an annotation support device according to the present invention. FIG. 2 is an explanatory diagram of the pointing method, in which (1) is the first embodiment, (2) is the second embodiment, and (3) is the third embodiment. FIG. 3 is a schematic diagram showing an example of an overall image of annotated image data. FIG. 4 is a schematic flow diagram showing an embodiment of the annotation support method according to the present invention. FIG. 5 is a schematic flow diagram regarding the pointing method. FIG. 6 is a schematic flow diagram showing a modification of the embodiment of the annotation support method according to the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of an annotation support apparatus and method according to the present invention will be described in detail below with reference to the drawings. Note that the present invention is not limited to this embodiment.

(Annotation support device)
First, an embodiment of an annotation support device according to the present invention will be described.
As shown in FIG. 1, an annotation support device 10 according to the present embodiment is a support device for supporting the task of adding annotations (for example, ratings, notes, etc.) to a predetermined portion of an object. This annotation support device 10 includes an input/output section 12, a pointing device 14, a PC 16 (personal computer), a processing section 18, and a storage section 20.

The input/output unit 12 is a portable unit that integrates a 3D scanner and a projector. A 3D scanner is a three-dimensional data acquisition means that scans the surroundings and acquires 3D data (three-dimensional data) regarding the position and shape of a target object. This 3D scanner is capable of acquiring 3D data indicating a region pointed with the pointing device 14 (pointing position). A projector is a projection device that projects a predetermined image (including an annotated image) onto the surface of an object.

The pointing device 14 is a pointing means for an operator to specify a region on the object S. This pointing device 14 is equipped with operation buttons such as a registration button and an instruction button (not shown). For example, when the registration button is clicked with a region specified, a signal indicating that the region has been designated for registration is sent to the processing unit 18. do. Further, when the registration is completed, a signal to that effect is received from the processing unit 18.

Any device can be used as the pointing device 14 as long as it is possible to specify a part of the object S on-site. For example, the region may be designated with the tip of a pointer or pencil-type device as shown in FIG. 2(1), or a device integrated with a smartphone or tablet terminal may be used. Alternatively, as shown in FIG. 2(2), a laser beam may be emitted to a desired location in order to point to a location that cannot be reached, such as a ceiling, on site. Further, pointing by the pointing device 14 is not limited to the method of specifying one point. For example, as shown in FIG. 2(3), the enclosed area R may be specified by dragging a button or the like.

The PC 16 is a personal computer having a display device such as a display and an operation interface such as a keyboard and a mouse, and functions as an annotation adding means. The PC 16 may be configured with a tablet terminal, for example. This PC 16 acquires image data from the storage unit 20 and displays the acquired image on a display device. The user can refer to the image displayed on the display device and add an annotation to a predetermined portion of the image using the operation interface. However, it is preferable to carry out this annotation work on-site, as it is impossible to determine which annotation should be added to which part without seeing the condition of the part on-site. For example, judgments are made on-site, such as how far to consider one part, the extent to which paint is floating within one part, and the numbering of parts when dividing one part into two parts based on the degree of deterioration. It's difficult to tell without looking at the condition of the parts.

The added annotation is stored in the storage unit 20 as annotation data associated with 3D data indicating the position of the region within the image. Note that the PC 16 may be used to add and edit annotations.

The processing section 18 is connected to the input/output section 12, the pointing device 14, and the PC 16, and performs various processes by transmitting and receiving data to and from these, and functions as an image generating means. Specifically, the processing unit 18 performs a process of generating an image of the object S based on the 3D data transmitted from the 3D scanner of the input/output unit 12 , while transmitting the generated image data to the input/output unit 12 . The process of transmitting the data to the projector and the PC 16 is performed. The processing unit 18 also performs a process of generating an annotated image in which a predetermined part (for example, a position corresponding to the pointing position) is annotated based on the annotation data transmitted from the PC 16.

Further, upon receiving a signal from the pointing device 14 indicating that a region has been designated for registration, the processing section 18 transmits a 3D scan command to the 3D scanner of the input/output section 12 . In this way, the 3D scanner is caused to perform a scan to obtain 3D data indicating the pointing position. Then, using 3D data indicating the pointing position transmitted from the 3D scanner of the input/output unit 12, a process of detecting the pointing position is performed. The pointing position can be detected using the difference between the 3D data and the 3D data before receiving a signal from the pointing device 14 indicating that the region has been designated for registration. Note that the processing unit 18, the input/output unit 12, the pointing device 14, and the PC 16 may be connected by a direct connection method or by a connection method via a network or the like.

The storage unit 20 stores various data sent to and received from the processing unit 18. For example, 3D data transmitted from the 3D scanner of the input/output unit 12, image data generated by the processing unit 18, annotation data transmitted from the PC 16, etc. are stored.

The operation and effect of the above configuration will be explained by taking as an example the work of assigning a member number (annotation) to each member (part) of a target object.
First, as shown in FIG. 2(1), the input/output unit 12 is installed in a space where the object S is located. Next, a 3D scanner provided in the input/output unit 12 acquires surrounding 3D data. The processing unit 18 generates an image based on the 3D data.

Next, when the user clicks the registration button while pointing at the target region with the pointing device 14, the processing unit 18 detects the pointing position (target region) through the 3D scanner. The processing unit 18 refers to the annotation data in the storage unit 20 and generates an annotated image in which the pointing position is annotated. A projector included in the input/output unit 12 projects an annotated image toward a target region to perform projection mapping. By doing this, a number (5 in the illustrated example) is projected and displayed on the target region.

Note that the 3D data of this projected image may be reacquired using a 3D scanner. By doing so, it is possible to specify which position on the 3D data the pointing position is and hold it as data. By repeating this process, numbers are sequentially assigned to each location on the 3D data. Additionally, since the numbers are projected in real time on-site, you can easily check the work progress. Through this series of operations, annotated image data as shown in FIG. 3, for example, can be obtained as a whole.

Further, in a later step or in parallel with the above operation, it may be possible to add and edit annotations to each registered pointing position using the PC 16 or the like. Further, on the display device of the PC 16, a plurality of points may be designated in the area R using, for example, a lasso tool (see FIG. 2(3)), so that batch editing can be performed.

Furthermore, before the user clicks the registration button while pointing at the target region with the pointing device 14, the user may be looking at the image displayed on the PC 16 and assigning numbers to predetermined members in the image. good. In this case, the assigned number is associated with the member and stored in the storage unit 20 as annotation data.

In addition, after obtaining the entire annotated image data, the number displayed on the 3D data is output as an image, and an annotation linking the name of the member and the reference is displayed at the position indicating each member. You can also easily obtain explanatory diagrams.

In the above embodiment, the case where annotations are added/edited on the PC 16 has been described as an example, but the present invention is not limited to this, and the processing unit 18 uses editing tools for adding/editing annotations. It may also have an output interface function. Even in this case, the same effects as described above can be achieved.

According to the present embodiment, it is possible to generate an image using 3D data and specify its part while checking the pointing location on-site, making it easy to generate data for a building with annotations added to each part. be able to. Additionally, the progress of annotation work can be fed back in real time through projection mapping. Therefore, the worker can proceed with the pointing work while visually checking for omissions in the annotation work on site. In addition, pointing work can be performed by multiple workers. Therefore, the efficiency of pointing work on site can be improved.

(Annotation support method)
Next, an embodiment of the annotation support method according to the present invention will be described.
As shown in FIG. 4, the annotation support method of this embodiment sets the input/output unit 12 in the local space (step S11). Next, the surrounding area is 3D scanned and 3D data is held (steps S12, 13). If the previous 3D data already exists in the storage unit 20 (Yes in step S14), the previous data and the current 3D data are integrated (step S22), and the process proceeds to step S15. On the other hand, if there is no previous 3D data (No in step S14), the process directly advances to step S15.

Next, the user registers the body part using the registration button of the pointing device 14 (step S15). The processing unit 18 generates an annotated image in which numbers (annotations) are assigned to the positions of parts on the 3D data (step S16), and the projector performs projection mapping on this image (step S17). If the next part is registered with the pointing device 14 (Yes in step S18), the process returns to step S15 and repeats the process. If it is not registered (No in step S18), the process directly advances to step S19.

Next, it is determined whether to move to the next scanning position (step S19). If the scanning position is to be changed (Yes in step S19), the input/output section 12 is moved (step S23), and the process returns to step S11 to repeat the process. If the scanning position is not changed (No in step S19), it is determined whether to additionally register or modify the scan position (step S20). If additional registration/correction is desired (Yes in step S20), 3D data is drawn on the PC 16 (step S24), a number is registered/corrected for the part on the screen (step S25), and the process proceeds to step S21. If no additional registration or correction is to be made (No in step S20), the process directly advances to step S21.

If serial numbers are to be assigned again (Yes in step S21), the assigned numbers are sorted (step S26), and the process ends. If the serial number is not to be re-assigned (No in step S21), the process ends.

In this way, if you are unable to obtain 3D data for the entire building in a single scan and have to scan it multiple times, or if you want to scan detailed areas separately using a handy 3D scanner, you can integrate the 3D data first. It is recommended that you cover the entire building before starting the pointing operation.

Next, the processing flow in the pointing work in steps S15 to S17 will be described.
As shown in FIG. 5, first, when the registration button of the pointing device 14 is clicked (step S1), the processing unit 18 recognizes the point position in the image generated from 3D data (step S2). The projector projects an image toward the recognized point position (step S3). Thereafter, the state in which the image is projected onto the target object is rescanned using a 3D scanner (step S4). By doing so, the point position can be held on the 3D data (step S5).

According to the present embodiment, it is possible to generate an image using 3D data and specify its part while checking the pointing location on-site, making it easy to generate data for a building with annotations added to each part. be able to. Additionally, the progress of annotation work can be fed back in real time through projection mapping. Therefore, the worker can proceed with the pointing work while visually checking for omissions in the annotation work on site. In addition, pointing work can be performed by multiple workers. Therefore, the efficiency of pointing work on site can be improved.

(Modified example)
Next, a modification of the annotation support method according to this embodiment will be described. Even with this modification, the same effects as described above can be achieved.

As shown in FIG. 6, first, the input/output unit 12 is installed in the local space, and the surrounding area is 3D scanned (step S31). Subsequently, the user clicks the instruction button on the pointing device 14 (step S32), and the position of the pointing device 14 is rescanned with the 3D scanner to acquire (image) 3D data (step S33).

Next, the projector projects an image toward the position indicated by the pointing device 14 (step S34). If the pointing device 14 has been moved (Yes in step S35), the process returns to step S33 and repeats the process. If you do not want to move the pointing device 14 (No in step S35), click the registration button on the pointing device 14 (step S36), rescan the projection mapped image, and acquire 3D data (imaging). (Step S37). Next, the position of the projected image is specified on the 3D data based on the difference between the data in step S31 and the data in step S37 (step S38), and the position of the projected image is stored in correspondence with the position of the object shape (step S39). After this, if there is a next point (Yes in step S40), the process returns to step S32 and repeats the process. If there is no next point (No in step S40), the process ends.

As explained above, the annotation support device according to the present invention is a support device for supporting the work of annotating a predetermined part of a target object, and is a support device for supporting the work of annotating a predetermined part of a target object. dimensional data acquisition means; image generation means for generating an image of an object based on the acquired three-dimensional data; and a projection means for projecting the image onto the surface of the corresponding object, the image generation means generates an annotated image in which a predetermined part is annotated, and the projection means projects the generated annotated image. Therefore, annotations can be added while observing the object on-site.

Further, according to another annotation support device according to the present invention, the device further includes pointing means for pointing to a predetermined part of the target object by a user's operation, and the three-dimensional data acquisition means acquires three-dimensional data of the pointed part. Therefore, annotation work can be carried out while visually checking the pointed area on site.

Further, according to the annotation support method according to the present invention, there is provided a support method for supporting the work of annotating a predetermined part of a target object, which includes a step of acquiring three-dimensional data of the target object, and a step of acquiring three-dimensional data of the target object; A step of generating an image of the object based on the three-dimensional data, a step of annotating a predetermined part in the generated image by a user's operation, and generating an annotated image in which the predetermined part is annotated. Since the generated annotated image is then projected onto the surface of the corresponding object, it is possible to add annotations while observing the object on-site.

Furthermore, according to another annotation support method according to the present invention, a predetermined part of a target object is pointed by a user's operation, and three-dimensional data of the pointed part is obtained, so that the pointed part can be visually confirmed on site. You can perform annotation work while

As described above, the annotation support device and method according to the present invention are useful for preservation and restoration planning of historical buildings, and are particularly suitable for annotating objects while observing them on-site.

10 Annotation support device 12 Input/output unit (three-dimensional data acquisition means, projection means)
14 Pointing device (pointing means)
16 PC (annotation adding means)
18 Processing unit (image generation means)
20 Storage part S Object

Claims (4)

A support device for supporting the task of annotating a predetermined part of a target object, the support device comprising:
3D data acquisition means for acquiring 3D data of the object; image generation means for generating an image of the object based on the acquired 3D data; and annotation at a predetermined part of the generated image by user operation. an annotation adding means for adding an annotation, and a projection means for projecting the generated image onto the surface of the corresponding object,
An annotation support device characterized in that the image generation means generates an annotated image in which a predetermined part is annotated, and the projection means projects the generated annotated image. further comprising pointing means for pointing at a predetermined part of the object according to a user's operation,
2. The annotation support device according to claim 1, wherein the three-dimensional data acquisition means acquires three-dimensional data of the pointed part. A support method for supporting the task of annotating a predetermined part of a target object, the method comprising:
a step of acquiring three-dimensional data of the target object; a step of generating an image of the target object based on the acquired three-dimensional data; and a step of annotating a predetermined part of the generated image by a user's operation; An annotation support method comprising: generating an annotated image in which a predetermined part is annotated; and projecting the generated annotated image onto the surface of a corresponding object. 4. The annotation support method according to claim 3, further comprising pointing at a predetermined part of the object by a user's operation, and acquiring three-dimensional data of the pointed part.

Images