WO2017163384A1

WO2017163384A1 - Data processing device, data processing method, and data processing program

Info

Publication number: WO2017163384A1
Application number: PCT/JP2016/059480
Authority: WO
Inventors: 川浦　健央; 隆博加島
Original assignee: 三菱電機株式会社
Priority date: 2016-03-24
Filing date: 2016-03-24
Publication date: 2017-09-28
Also published as: JPWO2017163384A1; JP6293386B2; TW201734954A

Abstract

According to the present invention, a world coordinate setting unit (7) acquires point group data representing the three-dimensional shape of an object and comprising a plurality of points, for each of which three-dimensional coordinates have been set. Further, the world coordinate setting unit (7) extracts, from the plurality of points in the point group data, points corresponding to image feature points included in a captured image of the object, and associates the three-dimensional coordinates set for the extracted points with the image feature points.

Description

Data processing apparatus, data processing method, and data processing program

The present invention relates to an image processing technique.

The AR (Augmented Reality) display system generates a subject development image from 3D (3 dimensional) shape data and texture data of a subject, checks the image feature points of the subject development image, and tracks the subject.

Patent Document 1 discloses an AR display system including an image input unit, a development view feature representation unit, a database, a database feature representation unit, a collation unit, and a display unit.
The image input unit inputs an RGB image obtained by photographing with a camera or the like.
The development view feature expression unit generates a development image from the 3D model and texture of the subject input via the image input unit, extracts image feature points, and calculates an image feature amount.
The database stores in advance images taken from arbitrary positions (coordinates and orientations) with respect to various objects.
The database feature representation unit reads an image from the database, extracts image feature points, and calculates a local descriptor.
The collation unit compares the local descriptors of the image feature points calculated by the development view feature representation unit and the database feature representation unit to identify the most similar image in the database, and obtains the position of the camera with respect to the object at the time of shooting. .
The display unit is, for example, a display device.

JP 2014-102746 A

In the AR display system of Patent Document 1, there is a problem that it is necessary to accumulate a large amount of images in a database in advance in a database. Further, the AR display system of Patent Document 1 has a problem that a developed image must be generated at high speed from a 3D model and a texture.

The main object of the present invention is to solve the above-mentioned problems, and to speed up AR display without accumulating images in a database in advance and without generating a developed image. Objective.

The data processing apparatus according to the present invention
A point cloud data acquisition unit for acquiring point cloud data composed of a plurality of points, each of which represents a three-dimensional shape of an object, each of which is set with a three-dimensional coordinate;
A point corresponding to the image feature point included in the captured image of the object is extracted from the plurality of points of the point cloud data, and the three-dimensional coordinates set for the extracted point are associated with the image feature point. And an association unit.

In the present invention, the three-dimensional coordinates of the points corresponding to the image feature points in the captured image are associated with the image feature points. For this reason, according to the present invention, the data amount can be significantly reduced by holding only the three-dimensional coordinates of the image feature points.
In the present invention, since the amount of point cloud data to be held is small, the search can be performed at high speed. Further, in the present invention, since the three-dimensional shape of the object is handled by the point cloud data, it is not necessary to generate a developed image or to store the RGB image in the database in advance, and the AR display can be speeded up.

FIG. 3 is a diagram illustrating a functional configuration example of the AR display device according to the first embodiment. FIG. 4 is a flowchart showing an operation example of the AR display device according to the first embodiment. FIG. 6 is a diagram illustrating an example of a functional configuration of an AR display device according to a second embodiment. FIG. 9 is a flowchart showing an operation example of the AR display device according to the second embodiment. FIG. 10 is a diagram illustrating a functional configuration example of an AR display device according to a third embodiment. FIG. 10 is a flowchart showing an operation example of the AR display device according to the third embodiment. FIG. 4 is a diagram showing a hardware configuration example of an AR display device according to Embodiments 1 to 3. FIG. 4 is a diagram illustrating an example of an image according to the first embodiment. FIG. 4 is a diagram illustrating an example in which an annotation image is superimposed on an image according to the first embodiment. FIG. 10 is a diagram illustrating an example of image feature points according to the second embodiment.

Embodiment 1 FIG.
*** Explanation of configuration ***
FIG. 1 shows a functional configuration example of an AR display device 1 according to the present embodiment.
FIG. 7 shows a hardware configuration example of the AR display device 1 according to the present embodiment.
The AR display device 1 is an example of a data processing device. The processing performed by the AR display device 1 corresponds to an example of a data processing method and a data processing program.
First, an outline of the AR display device 1 according to the present embodiment will be described.

The AR display device 1 according to the present embodiment acquires point cloud data and an annotation image.
The point cloud data is data representing the three-dimensional shape of an object that is a subject. The point cloud data is composed of a plurality of points. Point cloud data is usually a collection of tens of thousands of points. Three-dimensional coordinates (hereinafter also referred to as 3D coordinates) are set for each point of the point cloud data.
An annotation image is an image that is superimposed on a captured image of an object.
FIG. 9 shows an AR image obtained by virtually superimposing an underground pipe annotation image on the road image shown in FIG.
The graphics 50 (figure) indicating the shape of the pipe and the text 51 indicating the attributes of the pipe (in FIG. 9, the dimensions of the pipe) shown in FIG. 9 are annotation images.
The AR display device 1 acquires, for example, a cylindrical graphic 50 representing a manhole, and displays the cylindrical graphic 50 at the position of the manhole in the image. Moreover, the AR display device 1 acquires the text 51 representing the dimensions of the pipe, and displays the acquired text 51 at an appropriate position in the image.
As described above, when displaying the graphics 50 and the text 51 of the annotation image, the AR display device 1 selects one of the plurality of points of the point cloud data and sets the three-dimensional coordinates of the selected point. Corresponding to the graphics 50 or text 51 of the annotation image.

Next, a hardware configuration example of the AR display device 1 will be described with reference to FIG.

As shown in FIG. 7, the AR display device 1 includes a CPU (Central Processing Unit) 21, a memory 23, a GPU (Graphics Processing Unit) 25, a frame memory 26, and a RADAC (Random Access Memory Digital-to-Analog Converter) 27. Computer.
The CPU 21 executes a program that implements the annotation image editing unit 6, the world coordinate setting unit 7, and the perspective projection unit 8 shown in FIG. That is, the annotation image editing unit 6, the world coordinate setting unit 7, and the perspective projection unit 8 are realized by a program.
The GPU 25 executes a program that realizes the AR superimposing unit 9. That is, the AR superimposing unit 9 is realized by a program. The GPU 25 uses the RAMDAC 27 when the AR superimposing unit 9 performs an operation as a program.
A program for realizing the annotation image editing unit 6, the world coordinate setting unit 7, and the perspective projection unit 8 and a program for realizing the AR superimposing unit 9 are stored in the memory 23. The CPU 21 reads a program for realizing the annotation image editing unit 6, the world coordinate setting unit 7, and the perspective projection unit 8 from the memory 23 and executes this program. Further, the GPU 25 reads a program for realizing the AR superimposing unit 9 and executes this program.
The frame memory 26 stores annotation images.

The AR display device 1 is connected to a 3D sensor 22, a keyboard / mouse 29, and a monitor 28.
The 3D sensor 22 implements the image input unit 2, the RGB image generation unit 3, and the point cloud data generation unit 4 shown in FIG.
The keyboard / mouse 29 implements the annotation image input unit 5 shown in FIG.
The monitor 28 implements the display unit 10.

Next, a functional configuration example of the AR display device 1 will be described with reference to FIG.

The AR display device 1 includes an annotation image editing unit 6, a world coordinate setting unit 7, a perspective projection unit 8, and an AR superimposing unit 9.
The annotation image editing unit 6 acquires an annotation image such as text or a figure from the annotation image input unit 5 and edits the acquired annotation image.

The world coordinate setting unit 7 sets the three-dimensional coordinates of the annotation image to an arbitrary point in the point cloud data.
More specifically, the world coordinate setting unit 7 acquires point cloud data representing the three-dimensional shape of the subject.
In addition, the world coordinate setting unit 7 selects one of a plurality of points in the point cloud data, and associates the three-dimensional image set at the selected point with the annotation image. The superimposition position of the annotation image is defined by the point (point of the point cloud data) selected by the world coordinate setting unit 7. For example, by specifying the position in the RGB image (also referred to as a captured image) of the upper left vertex of the rectangle of the text 51 in FIG. 9 and the position in the RGB image of the upper right vertex, the superimposed position of the text 51 with the RGB image Is defined. In accordance with an instruction from the operator of the AR display device 1, the world coordinate setting unit 7 selects a point corresponding to the position in the RGB image of the top left vertex of the rectangle of the text 51 from the plurality of points of the point cloud data, and the text The point corresponding to the position in the RGB image of the lower right vertex of the 51 rectangle is selected.
The world coordinate setting unit 7 is an example of a point cloud data acquisition unit and an association unit. The operations performed in the world coordinate setting unit 7 are examples of point cloud data acquisition processing and association processing.

The perspective projection unit 8 projects an annotation image on 3D coordinates onto two-dimensional coordinates (hereinafter also referred to as 2D coordinates).

The AR superimposing unit 9 superimposes the annotation image projected on the 2D coordinates by the perspective projection unit 8 on the RGB image.

In FIG. 1, the image input unit 2 simultaneously measures the color and distance of the subject.
The RGB image generation unit 3 generates an RGB image from the color of the subject.
The point cloud data generation unit 4 generates point cloud data from the distance to the subject.
In the RGB image and the point cloud data, the same subject is captured from the same position and the same angle. That is, the 3D sensor 22 generates an RGB image and point cloud data in parallel for the same subject.
The annotation image input unit 5 inputs an annotation image such as text or graphics using a keyboard, a mouse, or the like.
The display unit 10 displays the superimposition result of the AR superimposing unit 9.
As described above, the image input unit 2, the RGB image generation unit 3, and the point cloud data generation unit 4 are realized by the 3D sensor 22 shown in FIG.
The annotation image input unit 5 is realized by the keyboard / mouse 29 shown in FIG.
The display unit 10 is realized by the monitor 28 shown in FIG.

*** Explanation of operation ***
Next, the operation of the AR display device 1 according to the present embodiment will be described with reference to FIG.

The image input unit 2 inputs the subject's color and distance measurement results to the RGB image generation unit 3 and the point cloud data generation unit 4.
The RGB image generation unit 3 generates an RGB image and inputs the generated RGB image to the AR superimposing unit 9.
The point cloud data generation unit 4 generates 3D coordinate point cloud data of the outline of the subject, and inputs the generated point cloud data to the world coordinate setting unit 7.
The annotation image input unit 5 generates an annotation image such as text or graphics, and inputs the generated annotation image to the annotation image editing unit 6.
The annotation image editing unit 6 edits an annotation image such as text or graphics, and inputs the edited annotation image to the world coordinate setting unit 7.
The world coordinate setting unit 7 acquires an annotation image and point cloud data. Then, the world coordinate setting unit 7 selects an arbitrary point from a plurality of points in the point cloud data, and associates the 3D coordinates set for the selected point with the annotation image to obtain an annotation image of 3D coordinates. . Further, the world coordinate setting unit 7 inputs a 3D coordinate annotation image to the perspective projection unit 8.
The perspective projection unit 8 acquires a 3D coordinate annotation image and projects the 3D coordinate annotation image onto the 2D coordinate. Further, the perspective projection unit 8 inputs the annotation image projected on the 2D coordinates to the AR superimposing unit 9.
The AR superimposing unit 9 acquires the annotation image projected on the 2D coordinates, and superimposes the annotation image projected on the 2D coordinates on the RGB image. Further, the AR superimposing unit 9 inputs the superimposition result to the display unit 10.
The display unit 10 displays the superimposed result of the AR superimposing unit 9 as an AR display for the subject.

Next, an operation example of the AR display device 1 according to the present embodiment will be described with reference to the flowchart of FIG.

In image input (step S2), the image input unit 2 captures a subject. More specifically, in the image input (step S2), the 3D sensor 22 captures the subject.

In RGB image generation (step S3), the RGB image generation unit 3 generates an RGB image. More specifically, in the RGB image generation (step S3), the subject is red, green, blue using a CCD (Charge Coupled Device) image sensor in the 3D sensor 22 or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. An RGB image having such color information is generated.

In the point cloud data generation (step S4), the point cloud data generation unit 4 generates point cloud data. More specifically, in the point cloud data generation (step S4), the 3D sensor is set to the origin based on the time when the infrared ray emitted from the infrared ray output device in the 3D sensor 22 is reflected by the subject and returns to the infrared ray receiver. Point cloud data that is a set of 3D coordinate points of the outer shape of the subject.

In the annotation image input (step S5), the annotation image input unit 5 inputs the annotation image to the annotation image editing unit 6.
More specifically, in the annotation image input (step S5), the operator of the AR display device 1 inputs the annotation image to the AR display device 1 by operating the keyboard, mouse, or the like.

In annotation image editing (step S6), the annotation image editing unit 6 edits text and graphics in the annotation image.

In the world coordinate setting (step S7), 3D coordinates of an arbitrary point of the point cloud data of the subject are given to the annotation image.
More specifically, the world coordinate setting unit 7 selects any one of a plurality of points in the point cloud data in accordance with an instruction from the operator of the AR display device 1, and uses the 3D coordinates of the selected point as an annotation. Associate with an image.

In perspective projection (step S8), the perspective projection unit 8 projects an annotation image of 3D coordinates onto 2D coordinates.
More specifically, the perspective projection unit 8 converts (X, Y, Z), which are the three-dimensional coordinates of the annotation image, to the coordinates (u, v) of the projection image, for example, by projective transformation shown in Equation 1 below. Convert to In Equation 1, [R | t] is the position of the 3D sensor 22. “R” is a rotation matrix that represents the orientation of the 3D sensor 22, and “t” is a translation vector that represents the coordinates of the 3D sensor 22. Further, “A” in Expression 1 is a fixed value of an internal parameter of the 3D sensor 22.

In AR superimposition (step S9), the AR superimposing unit 9 superimposes the projection image of the annotation image on the RGB image.

In the display (step S10), the display unit 10 displays the overlay result of the AR overlay (step S9).

*** Explanation of the effect of the embodiment ***
As described above, according to the present embodiment, the annotation image is mapped to the point cloud data that is the 3D coordinates of the subject, and the projected image of the annotation following the position of the arbitrary 3D sensor is superimposed on the RGB image. AR can be realized.

Embodiment 2. FIG.
*** Explanation of configuration ***
FIG. 3 shows a functional configuration example of the AR editing device 15 according to the present embodiment.
The AR editing device 15 according to the present embodiment is also an example of a data processing device. The processing performed by the AR editing device 15 according to the present embodiment also corresponds to an example of a data processing method and a data processing program.
Note that the hardware configuration example of the AR editing device 15 is as shown in FIG. 7, similarly to the AR display device 1 according to the first embodiment.

In the AR editing device 15 of FIG. 3, the perspective projection unit 8, the AR superimposing unit 9, and the display unit 10 are deleted from the configuration of the AR display device 1 of FIG.
On the other hand, in the AR editing device 15 of FIG. 3, an image feature point extraction unit 11, an AR data output unit 12, and an AR data 13 are added to the configuration of the AR display device 1 of FIG.
The image feature point extraction unit 11 and the AR data output unit 12 are realized by a program, which is executed by the CPU 21 in FIG.

The image feature point extraction unit 11 analyzes the RGB image and extracts image feature points of the RGB image. Image feature points exist mainly at discontinuous points in the RGB image. Each point in FIG. 10 represents an image feature point. The image feature point extraction unit 11 extracts image feature points by, for example, the Harris method, the KTK method, the Canny method, the zero crossing method, the relaxation method, the Hough transform, the dynamic contour method, the level set method, and the like.
The AR data 13 is data in which 3D coordinates in the world coordinate system of image feature points are recorded.
The AR data output unit 12 outputs the AR data 13 to the outside of the AR editing device 15.
In FIG. 3, the image input unit 2, the RGB image generation unit 3, the point cloud data generation unit 4, the annotation image input unit 5, and the annotation image editing unit 6 are the same as those in the first embodiment, and thus description thereof is omitted.
In the present embodiment, as in the first embodiment, the world coordinate setting unit 7 selects any point from a plurality of points in the point cloud data, and the three-dimensional coordinates set for the selected point. Is associated with the annotation image. Further, the world coordinate setting unit 7 extracts a point corresponding to the image feature point from a plurality of points of the point cloud data, and associates the three-dimensional coordinates set to the extracted point with the image feature point.

Hereinafter, differences from the first embodiment will be mainly described. Matters not described below are the same as those in the first embodiment.

*** Explanation of operation ***
Next, the operation of the AR editing device 15 according to the present embodiment will be described based on FIG.
The operations of the image input unit 2, RGB image generation unit 3, point cloud data generation unit 4, annotation image input unit 5, and annotation image editing unit 6 in FIG. 3 are the same as those in FIG.

The image feature point extraction unit 11 extracts image feature points of the RGB image, and inputs the extracted image feature points to the world coordinate setting unit 7.

The world coordinate setting unit 7 acquires an annotation image from the annotation image editing unit 6 and acquires point cloud data from the point cloud data generation unit 4 as in the first embodiment. Then, as in the first embodiment, the world coordinate setting unit 7 selects one of a plurality of points in the point cloud data, and uses the three-dimensional coordinates set for the selected point as an annotation image. Associate. Hereinafter, the three-dimensional coordinates associated with the annotation image are referred to as first three-dimensional coordinates. Further, the world coordinate setting unit 7 acquires image feature points from the image feature point extraction unit 11, extracts points corresponding to the acquired image feature points from a plurality of points of the point cloud data, and extracts the extracted points. 3D coordinates set in are associated with image feature points. Hereinafter, the three-dimensional coordinates associated with the image feature points are referred to as second three-dimensional coordinates. The world coordinate setting unit 7 inputs the first three-dimensional coordinates and the second three-dimensional coordinates as AR data 13 to the AR data output unit 12.

The AR data output unit 12 outputs the AR data 13 to the outside of the AR editing device 15.

Next, an operation example of the AR editing device 15 according to the present embodiment will be described with reference to the flowchart of FIG.

The image input (step S2), RGB image generation (step S3), point cloud data generation (step S4), annotation image input (step S5), and annotation image editing (step S6) in FIG. 4 are those shown in FIG. Since this is the same as the above, description thereof is omitted.

In image feature point extraction (step S11), the image feature point extraction unit 11 extracts image feature points from the RGB image. The image feature amount is described by the gradient of the brightness (brightness) of the peripheral pixels of each image feature point.

In the world coordinate setting (step S7), the world coordinate setting unit 7 records the 3D coordinates (first three-dimensional coordinates and second three-dimensional coordinates) of the annotation image and the world coordinate system of the image feature points. 13 is generated.

In the AR data output (step S12), the AR data output unit 12 outputs the AR data to the outside of the AR editing device 15.

*** Explanation of the effect of the embodiment ***
As described above, according to the present embodiment, the AR data obtained by mapping the image feature points extracted from the RGB image of the subject to the point cloud data that is the 3D coordinates can be stored in advance in the database. Further, it can be generated at high speed without generating a developed image.

Embodiment 3 FIG.
FIG. 5 shows a functional configuration example of the AR display device 100 according to the present embodiment.
The AR display device 100 according to the present embodiment is also an example of a data processing device. The processing performed by the AR display device 100 according to the present embodiment also corresponds to examples of the data processing method and the data processing program.
Note that the hardware configuration example of the AR display device 100 according to the present embodiment is as shown in FIG. 7, similarly to the AR display device 1 according to the first embodiment.

In the AR display device 100 of FIG. 5, the point cloud data generation unit 4, the annotation image input unit 5, the annotation image editing unit 6, and the world coordinate setting unit 7 are deleted from the configuration of the AR display device 1 of FIG.
On the other hand, in the AR display device 100 of FIG. 5, an image feature point extraction unit 11, a position estimation unit 14, and an AR data input unit 16 are added to the configuration of the AR display device 1 of FIG.
The image feature point extraction unit 11 and the position estimation unit 14 are realized by a program, and this program is executed by the CPU 21 of FIG.
The AR data input unit 16 is realized by the keyboard / mouse 29 of FIG.

The image feature point extraction unit 11 is the same as that shown in FIG. 3, analyzes the RGB image, and extracts image feature points of the RGB image. The operation performed by the image feature point extraction unit 11 is an example of image feature point extraction processing.
The AR data input unit 16 acquires the AR data 13. The AR data 13 is the same as that described in the second embodiment.
The position estimation unit 14 is a 3D imaging device based on 3D coordinates of image feature points in the world coordinate system and 2D coordinates in RGB images (2D coordinates of image feature points obtained by projective transformation of 3D coordinates of image feature points). The position of the sensor 22 is estimated. That is, the position estimation unit 14 estimates the position when the 3D sensor 22 captures an RGB image based on the 3D coordinates of the image feature points and the 2D coordinates of the image feature points in the RGB image. The operation performed by the position estimation unit 14 is an example of position estimation processing.

*** Explanation of operation ***
Next, the operation of the AR display device 100 according to the present embodiment will be described with reference to FIG.
The operations of the image input unit 2, the RGB image generation unit 3, the perspective projection unit 8, the AR superimposing unit 9, and the display unit 10 in FIG. 5 are the same as those in FIG. The operation of the image feature point extraction unit 11 is the same as that in FIG.

The AR data input unit 16 inputs the AR data 13 to the perspective projection unit 8 and the position estimation unit 14.
The position estimation unit 14 estimates the position of the 3D sensor 22 from the 3D coordinates of the image feature points in the world coordinate system and the 2D coordinates in the RGB image, and inputs the estimated 3D sensor 22 position to the perspective projection unit 8.

The image input (step S2), RGB image generation (step S3), perspective projection (step S8), AR superimposition (step S9), and display (step S10) in FIG. 6 are the same as those shown in FIG. The description is omitted.
Further, the processing of image feature point extraction (step S11) is the same as that in FIG.

In the AR data input (step S16), the AR data input unit 16 inputs the AR data 13 to the perspective projection unit 8.

In position estimation (step S14), the position estimation unit 14 estimates the position of the 3D sensor 22 in the RGB image.
Specifically, the position estimation unit 14 detects the coordinates x on the RGB image corresponding to the image feature points of the three-dimensional coordinates (X, Y, Z) by matching the image feature amounts. If the coordinates obtained by reprojecting the three-dimensional coordinates (X, Y, Z) of the image feature points onto the RGB image by Equation 1 are x ^, the reprojection error E is the Euclidean distance d (x, x ^) between x and x ^. (Note that the notation with “^” diagonally above and to the right of x is the same as the notation with “^” immediately above x in Equation 2). The reprojection error E can be obtained using Equation 2. The position estimation unit 14 estimates the position of the 3D sensor 22 that minimizes the error E with i image feature points, that is, [R | t] of Equation 1, and the estimated value of [R | t] The position of the 3D sensor 22.

Further, the position estimation unit 14 inputs the estimated position of the 3D sensor 22 to the perspective projection unit 8.

*** Explanation of the effect of the embodiment ***
As described above, according to the present embodiment, the AR data in which the image feature points extracted from the RGB image of the subject are mapped to the point cloud data that is 3D coordinate data is used for estimating the position of the 3D sensor. Since it is not necessary to match the developed image of the 3D model with the RGB image at each position of the 3D sensor stored in the database in advance for estimation of the position of the 3D sensor, both images are unnecessary.

As mentioned above, although embodiment of this invention was described, you may implement in combination of 2 or more among these embodiment.
Alternatively, one of these embodiments may be partially implemented.
Alternatively, two or more of these embodiments may be partially combined.
In addition, this invention is not limited to these embodiment, A various change is possible as needed.
*** Explanation of hardware configuration ***
Finally, a supplementary explanation of the hardware configuration will be given.
The CPU 21 and the GPU 25 illustrated in FIG. 7 are ICs (Integrated Circuits) that perform processing.
The memory 23 and the frame memory 26 illustrated in FIG. 7 are a RAM (Random Access Memory), a flash memory, an HDD (Hard Disk Drive), and the like.
The memory 23 also stores an OS (Operating System).
At least a part of the OS is executed by the CPU 21.
The CPU 21 executes functions of the annotation image editing unit 6, the world coordinate setting unit 7, the perspective projection unit 8, the image feature point extraction unit 11, the AR data output unit 12, and the position estimation unit 14 while executing at least a part of the OS. Execute the program to be realized.
When the CPU 21 executes the OS, task management, memory management, file management, communication control, and the like are performed.
Information, data, signal values, and the like indicating the processing results of the annotation image editing unit 6, the world coordinate setting unit 7, the perspective projection unit 8, the image feature point extraction unit 11, the AR data output unit 12, and the position estimation unit 14. The variable value is stored in the memory 23 or a register or cache memory in the CPU 21.
A program for realizing the functions of the annotation image editing unit 6, the world coordinate setting unit 7, the perspective projection unit 8, the image feature point extraction unit 11, the AR data output unit 12, the position estimation unit 14, and the AR superimposition unit 9 is You may memorize | store in portable storage media, such as a magnetic disk, a flexible disk, an optical disk, a compact disk, a Blu-ray (trademark) disk, and DVD.

Further, the “parts” of the annotation image editing unit 6, the world coordinate setting unit 7, the perspective projection unit 8, the image feature point extraction unit 11, the AR data output unit 12, the position estimation unit 14, and the AR superimposition unit 9 are referred to as “circuit”. ”Or“ step ”or“ procedure ”or“ processing ”.
The AR display device 1, the AR editing device 15, and the AR display device 100 are respectively a logic IC (Integrated Circuit), a GA (Gate Array), an ASIC (Application Specific Integrated Circuit), and an FPGA (Field-Programmable Grating). It may be realized by an electronic circuit.
The processor and the electronic circuit are also collectively referred to as a processing circuit.

1 AR display device, 2 image input unit, 3 RGB image generation unit, 4 point cloud data generation unit, 5 annotation image input unit, 6 annotation image editing unit, 7 world coordinate setting unit, 8 perspective projection unit, 9 AR superimposition unit 10, display unit, 11 image feature point extraction unit, 12 AR data output unit, 13 AR data, 14 position estimation unit, 15 AR editing device, 16 AR data input unit, 21 CPU, 22 3D sensor, 23 memory 25 GPU, 26 frame memory, 27 RAMDAC, 28 monitor, 29 keyboard / mouse, 50 graphics, 51 text, 100 AR display.

Claims

A point cloud data acquisition unit for acquiring point cloud data composed of a plurality of points, each of which represents a three-dimensional shape of an object, each of which is set with a three-dimensional coordinate;
A point corresponding to the image feature point included in the captured image of the object is extracted from the plurality of points of the point cloud data, and the three-dimensional coordinates set for the extracted point are associated with the image feature point. A data processing apparatus having an association unit.
The association unit
The point cloud data is selected from any of the plurality of points, and a three-dimensional image set for the selected point is associated with an annotation image superimposed on a captured image of the object. The data processing apparatus described.
A point cloud data acquisition unit for acquiring point cloud data composed of a plurality of points, each of which represents a three-dimensional shape of an object, each of which is set with a three-dimensional coordinate;
An association unit that selects any one of the plurality of points of the point cloud data and associates a three-dimensional image set to the selected point with an annotation image to be superimposed on the captured image of the object; A data processing apparatus.
An image feature point extraction unit that analyzes a captured image of the object and extracts image feature points included in the captured image of the object;
Position estimation for estimating a position when the photographed image of the photographing device that photographed the photographed image is photographed based on the three-dimensional coordinates of the image feature point and the two-dimensional coordinates of the image feature point in the photographed image. A data processing apparatus.
The position estimation unit
Based on the three-dimensional coordinates of the image feature points and the two-dimensional coordinates in the captured image of the image feature points obtained by projective transformation of the three-dimensional coordinates of the image feature points, the captured image of the imaging device is The data processing apparatus according to claim 3, wherein a position at the time of shooting is estimated.
The computer obtains point cloud data composed of a plurality of points, each representing a three-dimensional shape of an object, each having three-dimensional coordinates,
The computer extracts a point corresponding to an image feature point included in a captured image of the object from the plurality of points of the point cloud data, and sets the three-dimensional coordinates set for the extracted point to the image A data processing method for associating with feature points.
The computer obtains point cloud data composed of a plurality of points, each representing a three-dimensional shape of an object, each having three-dimensional coordinates,
The computer selects any one of the plurality of points of the point cloud data, and associates the 3D image set to the selected point with the annotation image superimposed on the captured image of the object Data processing method.
The computer analyzes the captured image of the object and extracts image feature points included in the captured image of the object,
Based on the three-dimensional coordinates of the image feature point and the two-dimensional coordinates of the image feature point in the photographed image, the computer captures the position of the photographing device that photographed the photographed image. Data processing method to estimate.
A point cloud data acquisition process for acquiring point cloud data composed of a plurality of points each having a three-dimensional coordinate set, in which the three-dimensional shape of the object is represented;
A point corresponding to the image feature point included in the captured image of the object is extracted from the plurality of points of the point cloud data, and the three-dimensional coordinates set for the extracted point are associated with the image feature point. A data processing program for causing a computer to execute association processing.
A point cloud data acquisition process for acquiring point cloud data composed of a plurality of points each having a three-dimensional coordinate set, in which the three-dimensional shape of the object is represented;
An association process for selecting any one of the plurality of points of the point cloud data and associating a 3D image set for the selected point with an annotation image to be superimposed on the captured image of the object; A data processing program that causes a computer to execute.
Image feature point extraction processing for analyzing a captured image of an object and extracting image feature points included in the captured image of the object;
Position estimation for estimating a position when the photographed image of the photographing device that photographed the photographed image is photographed based on the three-dimensional coordinates of the image feature point and the two-dimensional coordinates of the image feature point in the photographed image. A data processing program that causes a computer to execute processing.