JP7003594B2 - 3D point cloud display device, 3D point cloud display system, 3D point cloud display method, 3D point cloud display program, recording medium - Google Patents

Description

The present invention is a 3D point cloud display device, a 3D point cloud display system, and a 3D point cloud display that semi-automatically matches and displays images taken by a camera on the display screen of the 3D point cloud display device. The method relates to a three-dimensional point cloud display program and a recording medium.

When monitoring a wide area infrastructure such as a river, the 3D point cloud data of the terrain or structure obtained by laser scanning may be displayed on the 3D point cloud display device. Then, by superimposing image data such as moving images and still images captured by the camera on the screen of the 3D point cloud display device, it is possible to check the real-time state of the image and the surrounding environment together, such as the flooding situation of the river. Can be accurately grasped. In order to realize this function, it is necessary to match the 3D point cloud data and the image data.

Conventionally, in order to match the 3D point group data with the image data, the 3D coordinate data around the approximate position of the mobile terminal acquired from the camera that captures the actual image of the surroundings and the GPS device is acquired from the server. Feature information was extracted from each of the acquired three-dimensional coordinate data and the ortho-corrected current actual image taken by the camera, and the extracted feature information was matched with each other (Patent Document 1).

Japanese Unexamined Patent Publication No. 2016-170060 (paragraph [0007])

Patent Document 1 assumes that equipment information is superimposed and displayed on an image in real space by using augmented reality on a mobile terminal, and the user of the mobile terminal is often close to the equipment. Therefore, after acquiring the current approximate position from GPS, it is configured to transmit a request for 3D map information including an area having a radius of 20 meters around the approximate position to the 3D map information transmission unit of the server. ..

However, in the case of wide area infrastructure monitoring, the object of the image taken by the camera does not always exist around the camera, and as in Patent Document 1, the surrounding three dimensions based on the GPS data of the current approximate position. Even if you tried to match the map information with the actual image, there was a possibility that they could not be matched.

The present invention has been made to solve the above-mentioned problems, and in matching the three-dimensional point cloud data and the image data, the matching between the three-dimensional point cloud data and the image data is surely performed. The purpose is to obtain a three-dimensional point cloud display device.

The three-dimensional point group display device according to the present invention includes a data acquisition unit that acquires image data obtained by photographing a monitored object with a camera, and at least a part of a range photographed by the camera among the monitored objects. A display processing unit that performs processing to display the 3D point group data obtained by measuring in step 1 on the screen, and a calculation range setting that sets the calculation range by user input for the 3D point group data displayed on the screen. A feature information extraction unit and a feature information extraction unit that extract feature information of each of the selection data selected from the three-dimensional point group data based on the settings of the unit, the calculation range setting unit, and the image data acquired by the data acquisition unit. Based on the result of extraction by, the matching unit that matches the feature information of the 3D point group data and the feature information of the image data, and the matching result of the matching section, the image is displayed on the 3D point group data in the screen. It is equipped with a superimposition display processing unit for superimposing and displaying data.

The three-dimensional point group display device according to the present invention includes a data acquisition unit that acquires image data obtained by photographing a monitored object with a camera, and at least a part of a range photographed by the camera among the monitored objects. The display processing unit that performs the processing to display the 3D point group data obtained by measuring in step 1 and the 3D point group data displayed on the screen are set in the calculation range by the attribute information included in the image data. A feature information extraction unit that extracts the feature information of the selection data selected from the three-dimensional point group data based on the setting of the calculation range setting unit and the image data acquired by the data acquisition unit. , Based on the result of extraction by the feature information extraction unit, the matching section that matches the feature information of the 3D point group data and the feature information of the image data, and the 3D point group on the screen based on the matching result of the matching section. It is provided with a superimposition display processing unit that superimposes and displays image data on the data.

Further, in the three-dimensional point group display system according to the present invention, a camera that captures a monitored object, a data acquisition unit that acquires image data acquired by the camera, and a monitored object, which is captured by the camera. A display processing unit that performs processing to display the 3D point group data obtained by measuring including at least a part of the range on the screen, and a calculation range for the 3D point group data displayed on the screen by user input. Feature information extraction that extracts the feature information of the selection data selected from the 3D point group data and the image data acquired by the data acquisition section based on the settings of the calculation range setting section and the calculation range setting section. Based on the result of extraction by the section and the feature information extraction section, the matching section that matches the feature information of the three-dimensional point group data and the feature information of the image data, and the matching section, based on the matching result of the matching section, are three-dimensional in the screen. It is provided with a superimposition display processing unit that superimposes and displays image data on the point group data.

Further, the three-dimensional point group display method according to the present invention includes a data acquisition step of acquiring image data obtained by photographing a monitored object with a camera, and at least a part of a range photographed by the camera among the monitored objects. A display processing step that performs processing to display the 3D point group data obtained by including and measuring on the screen, and a calculation range that sets the calculation range by user input for the 3D point group data displayed on the screen. A feature information extraction processing step that extracts the feature information of each of the selection data selected from the three-dimensional point group data based on the setting in the setting step and the calculation range setting step and the image data acquired by the data acquisition step, and the feature. Based on the result of extraction by the information extraction processing step, the matching processing step that matches the feature information of the three-dimensional point group data and the feature information of the image data with each other, and the matching result in the matching processing step, three dimensions are displayed on the screen. It has a superimposition display process that superimposes and displays image data on point group data.

Further, the three-dimensional point group display program according to the present invention includes a data acquisition step of acquiring image data obtained by photographing a monitored object with a camera, and at least a part of a range photographed by the camera among the monitored objects. A display processing step that performs processing to display the 3D point group data obtained by measuring including the above, and a calculation that sets the calculation range by user input for the 3D point group data displayed on the screen. A feature information extraction processing step that extracts feature information of each of the selection data selected from the three-dimensional point group data based on the settings in the range setting step and the calculation range setting step and the image data acquired by the data acquisition step. Based on the result of extraction by the feature information extraction processing step, the matching processing step that matches the feature information of the three-dimensional point group data and the feature information of the image data, and the matching result of the matching processing step, are three-dimensional in the screen. A computer is made to execute a superimposed display processing step of superimposing and displaying image data on the point group data.

Further, the recording medium according to the present invention records the above-mentioned three-dimensional point cloud display program.

Further, according to the three-dimensional point cloud display device according to the present invention, it is possible to reliably perform matching between the three-dimensional point cloud data and the image data.

Further, according to the three-dimensional point cloud display system according to the present invention, it is possible to reliably perform matching between the three-dimensional point cloud data and the image data.

Further, according to the three-dimensional point cloud display method according to the present invention, it is possible to reliably perform matching between the three-dimensional point cloud data and the image data.

Further, according to the three-dimensional point cloud display program according to the present invention, it is possible to reliably perform matching between the three-dimensional point cloud data and the image data.

Further, according to the recording medium according to the present invention, the above program is recorded, and it becomes possible to surely perform matching between the three-dimensional point cloud data and the image data.

It is a block diagram which shows the structure of the 3D point cloud display system which concerns on Embodiment 1. FIG. It is a figure which shows the hardware configuration example of the 3D point cloud display device which concerns on Embodiment 1. FIG. It is a figure which shows the example of the arrangement of the camera and the laser scanner when the river is assumed as the monitoring target which concerns on Embodiment 1. FIG. It is a flowchart which shows the operation of the 3D point cloud display device which concerns on Embodiment 1. FIG. It is a figure which shows the content of the screen in the display | display which displays the image data which concerns on Embodiment 1 and 3D point cloud data. It is a figure which shows the structure of the message input screen which functions as the calculation range setting part which concerns on Embodiment 1. FIG. It is a figure for demonstrating the contents of the feature information extraction work and the matching work which concerns on Embodiment 1. FIG. It is a figure which shows that the superimposition display processing unit which concerns on Embodiment 1 superimposes and displays image data on 3D point cloud data in a display. It is a block diagram in the case of setting a calculation range by using a pointing device as an input unit which concerns on Embodiment 2. FIG. This is a setting screen for setting a calculation range using the pointing device according to the second embodiment. It is a block diagram which shows the structure of the arithmetic part which concerns on Embodiment 3. It is a flowchart which shows the operation of the 3D point cloud display device which concerns on Embodiment 3. FIG. It is a figure which shows the modification which concerns on Embodiment 3. It is a block diagram which shows the structure of the 3D point cloud display system which concerns on Embodiment 4. It is a flowchart which shows the operation of the 3D point cloud display device which concerns on Embodiment 4. FIG. It is a figure which shows the content of the screen in the display | display which displays the 3D point cloud data which concerns on Embodiment 4. FIG. It is a flowchart which shows the operation of the modification which concerns on Embodiment 4. FIG.

Hereinafter, the three-dimensional point cloud display device and the three-dimensional point cloud display system device and the three-dimensional point cloud display method according to the embodiment of the present invention will be described in detail with reference to the drawings. The present invention is not limited to each embodiment, and each embodiment can be combined or appropriately modified. Further, the same reference numerals are given to the equivalent configurations in each embodiment, and the description thereof will be omitted.

Embodiment 1.
Embodiment 1 of the present invention will be described with reference to FIGS. 1 to 8. The outline of the first embodiment is to provide a calculation range setting unit that accepts the setting of the calculation range for performing matching work on the three-dimensional point cloud data displayed on the screen.

FIG. 1 is a block diagram showing a configuration of a three-dimensional point cloud display system 40 according to the first embodiment. The three-dimensional point cloud display system 40 includes a three-dimensional point cloud display device 1, a camera unit 20, and a storage unit 30. The storage unit is disclosed as a configuration different from that of the 3D point cloud display device 1, but if the 3D point cloud display device 1 itself has a sufficient recording / storage area, the 3D point cloud display device 1 may contain the storage unit. A storage unit 30 may be provided.

The 3D point cloud display device 1 includes a program P (described in detail later) as a 3D point cloud display program that realizes various functions of the 3D point cloud display device 1, and a network (not shown) from the camera unit 20. The data acquisition unit 2 that acquires the three-dimensional point cloud data 31 called from the image data 22 and the storage unit 30 transmitted via the data acquisition unit 2, the image data 22, the three-dimensional point cloud data 31 acquired by the data acquisition unit 2, and described later. A display processing unit 3 that takes in data related to the setting of the calculation range by the calculation range setting unit 6 from the data acquisition unit 2 and displays it on a display 4 such as a display, and an external connection device such as a mouse, a track ball, or a keyboard. It includes an input receiving unit 5 that accepts input from the input unit 7 and the like.

Further, the three-dimensional point cloud display device 1 includes a calculation unit 8 that performs various calculations. The calculation unit 8 has a viewpoint correspondence unit 9 for associating the viewpoint of the image data 22 acquired by the data acquisition unit 2 with the viewpoint of the camera 21 and the viewpoint of the three-dimensional point cloud data 31, and the image data 22 and the three-dimensional point cloud data 31. It includes a feature information extraction unit 10 that captures and extracts each feature information related to edge information and the like, and a matching unit 11 that performs matching regarding image data 22 and three-dimensional point cloud data 31 based on each extracted feature information. There is.

When the data acquisition unit 2 receives from the input reception unit 5 the input related to the setting by the input unit 7 on the data range acquisition screen (not shown) displayed on the display unit 4, the data acquisition unit 2 communicates with the storage unit 30 and the display unit 4 It is configured to narrow down the data range of the three-dimensional point cloud data 31 to be displayed in the data acquisition. For example, it is assumed that the monitoring target to be described in FIG. 3 is selected, or the acquisition time of the three-dimensional point cloud data 31 is selected. Further, the data acquisition unit 2 includes a calculation range setting unit 6 for setting a calculation range in the calculation unit 8.

Further, the display processing unit 3 has a superimposed display processing unit 12 that aligns and displays the image data 22 on the three-dimensional point cloud data 31 based on the matching result by the matching unit 11.

The timing of each processing of the feature extraction unit 10, the matching unit 11, and the superimposed display processing unit 12 will be described. At least the three-dimensional point cloud display system 40 is used for the feature extraction work performed by the feature extraction unit 10 using the image data 22 at a certain point in time and the matching work performed by the matching unit 11 using the extracted feature information. When building, it is enough to do it as an initial setting. At the stage of operating the three-dimensional point cloud display system 40, the superimposed display processing unit 12 uses the matching result of the matching unit 11 performed as the initial setting to obtain 3 image data 22 acquired from the data acquisition unit 2 each time. It may be superimposed and displayed at the position corresponding to each point of the dimension point cloud display data 31.

The camera unit 20 transmits data to the data acquisition unit 2 of the three-dimensional point cloud display device 1 for the camera 21 that captures a monitored object such as an environment such as a river or equipment such as a factory, and the image data 22 captured by the camera 21. The transmission unit 23 and the like are included.

The image data 22 captured by the camera 21 may be a still image or a moving image. However, when the image data 22 is a moving image, the feature extraction work by the feature extraction unit 10 and the matching work by the matching unit 11 using the extracted feature information are images taken in a predetermined period. It is preferable to use the average value of the moving image data as the data 22, or to use one image at an arbitrary time point in a predetermined period. That is, when trying to perform matching based on a moving image that changes in real time, when attempting to perform feature extraction work on the feature extraction unit 10 and matching work by the matching unit 11 using the extracted feature information, the amount of calculation is calculated. This is because can be enormous.

The storage unit 30 stores a plurality of points acquired by scanning the monitoring target and its surroundings in advance with a laser scanner 24 (see FIG. 3) or the like as three-dimensional point cloud data 31. The three-dimensional point cloud data 31 is, for example, a point in the space of the world coordinate system composed of the X axis, the Y axis, and the Z axis. In the three-dimensional point cloud data 31, each point may have absolute coordinates by using GPS data (not shown) or the like.

FIG. 2 is a diagram showing a hardware configuration example of the three-dimensional point cloud display device 1. As shown in FIG. 2, the three-dimensional point cloud display device 1 includes a processor 100, a memory 101, and an interface circuit 102. The processor 100, the memory 101, and the interface circuit 102 are connected to each other so as to be able to communicate with each other by the system bus 103.

By calling the program P stored in the memory 101 as a recording medium, the processor 100 realizes the function of the display processing unit 3 in the three-dimensional point cloud display device 1a as a computer, or the function of the superimposed display processing unit 12. , The function of the feature information extraction unit 10 in the calculation unit 8, the function of the matching unit 11, and other data processing functions are realized.

Further, when the user inputs an input in the input unit 7 connected via the interface circuit 102, the processor 100 receives the input in the input receiving unit 5 in the memory 101, so that the three-dimensional point group data 31 of the storage unit 30 is received. Data can be processed (extracted, processed, converted, etc.).

The processor 100 is an example of a processing circuit, and includes one or more of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a system LSI (Large Scale Integration). The memory 101 includes a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, and an EPROM (Enable Program Read Only Memory), a magnetic disk, a flexible memory, an optical disk, and the like. Includes one or more of compact discs and DVDs (Digital Versailles Disc). Further, although the configuration of the present hardware has been described with respect to the first embodiment, the same hardware configuration can be adopted for each embodiment of the present invention.

FIG. 3 is a diagram showing an example of the arrangement of the camera 21 and the laser scanner 24 when the river 25 is assumed as the monitoring target. The camera 21 and the laser scanner 24 are installed on a hill 26 overlooking the river 25. The hill 26 is located at a position farther from the river 25 than the river bank 125.

By photographing the river 25 with the camera 21, it is possible to monitor changes in the water level and the possibility of river flooding in real time. The camera 21 may take a picture of the surrounding environment and structures in addition to the river 25 to be monitored. For example, by setting the shooting direction of the camera 21 to the shooting direction 121a and setting the shooting frame 29 to accommodate the warehouse 27 and the house 28 as structures existing in the vicinity of the river 25, the situation of the river 25 can be grasped in an easy-to-understand manner. can.

On the other hand, the laser scanner 24 has a movable scanning unit, and normally scans an area wider than the camera 21 and stores it in the storage unit 30 as three-dimensional point cloud data 31. However, the present invention is not limited to such a configuration, and the laser scanner 24 may be provided so as to obtain the three-dimensional point cloud data 31 by measuring including at least a part of the range captured by the camera 21 among the monitoring targets. .. That is, it is sufficient that the monitoring target has a portion where the data ranges of the image data 22 and the three-dimensional point cloud data 31 overlap.

The shooting direction 121a indicates, for example, a direction connecting the center of the shooting frame 29 and the camera 21. The shooting direction 121a of the camera 21 is acquired by an orientation sensor or the like (not shown), and the shooting position 121b of the camera 21 is acquired by GPS data or the like (not shown). Information about the shooting direction 121a and the shooting position 121b is embedded in, for example, the image data 22 and recorded.

When matching is performed by the matching unit 11, the difference between the image data 22 and the three-dimensional point cloud data is smaller when the shooting time by the camera 21 and the scanning time by the laser scanner 24 are the same or close to each other. It is suitable for various calculations by the calculation unit 8. However, it is possible for the calculation unit 8 to perform various calculations even if the timings of the two data are different.

FIG. 4 is a flowchart showing the operation of the three-dimensional point cloud display device 1. A description will be given with reference to FIGS. 1 and 5 to 8 as appropriate. First, as shown in FIG. 1, the data acquisition unit 2 acquires the image data 22 and the three-dimensional point cloud data 31 (data acquisition step ST1 in FIG. 4). It should be noted that in the following description of each step, in order to simplify the description, the steps ST1, step ST2, etc. may be simply described with the step names omitted.

As shown in FIG. 5, the display processing unit 3 processes the image data 22 and the three-dimensional point cloud data 31 received from the data acquisition unit 2, and displays the image data 22 and the three-dimensional point cloud data 31 in the display unit 4. (Display processing step ST2). The three-dimensional point cloud data 31, which is a world coordinate system, is displayed on the display 4 so that the viewpoint can be switched as a screen coordinate system. Next, the user inputs using the input unit 7, and the coordinate (position) of the camera 21 is set by the viewpoint corresponding unit 9 via the input receiving unit 5 as shown in the screen 4a (FIG. 5) of the display unit 4. (Viewpoint correspondence processing step ST3).

After that, the screen 4b (FIG. 6) in which the viewpoint of the camera 21 and the viewpoint of the three-dimensional point cloud data 31 match is displayed (viewpoint corresponding screen display processing step ST4). The contents of the viewpoint correspondence processing step ST3 and the viewpoint correspondence screen display processing step ST4 will be described in detail with reference to FIGS. 5 and 6.

FIG. 5 is a diagram showing the contents of the screen 4a on the display 4 for displaying the image data 22 and the three-dimensional point cloud data 31. The relationship between the river 25 to be monitored and the structures around it and the three-dimensional point cloud data 31 will be described. On the screen 4a, the river 25 is a point cloud river 25a, the riverbank 125 is a point cloud riverbank 125a, the boundary between the river 25 and the riverbank 125 is the first point cloud boundary 25a1, the second point cloud boundary 25a2, and the warehouse 27 is the point cloud warehouse. 27a, the house 28 is displayed as a point cloud house 28a, and the camera 21 is displayed as a point cloud camera 21a by the three-dimensional point cloud data 31.

The relationship between the river 25 to be monitored and the structures around it and the image data 22 will be described. On the screen 4a, the display 4 creates an image data display area 41 (image frame) in which the content of the image data 22 is displayed in the corner display area, and confirms both the three-dimensional point cloud data 31 and the image data 22 together. can. This has the effect of efficiently performing the work of setting the calculation range by the calculation range setting unit 6, which will be described in detail later. The image data 22 displayed in the image data display area 41 may be projected in real time or may be captured at a certain point in time.

In the image data display area 41, the river 25 is displayed as an image river 25b, the color undulations of the water of the river 25 are displayed as image river water 25b1, the warehouse 27 is displayed as an image warehouse 27b, and the house 28 is displayed as an image house 28b by the image data 22. ..

By arranging the image data display area 41 at the corner of the screen 4a (for example, the diagonal end area of the screen 4a), the visibility of the three-dimensional point cloud data 31 on the screen 4a is reduced in the display area. It is possible to efficiently grasp the states of both the image data 22 and the three-dimensional point cloud data 31. However, the image data display area 41 may not be provided, and only the three-dimensional point cloud data 31 may be displayed on the screen 4a, and the image data 22 may be displayed on another screen (not shown).

By the way, in performing the matching work by the matching unit 11 of the three-dimensional point cloud data 31 and the image data 22, it is preferable to extract the feature information by the feature information extraction unit 10 in a state where the viewpoints of each other match. That is, if the viewpoints are deviated from each other, the feature information of the three-dimensional point cloud data 31 and the feature information of the image data 22 extracted by the feature information extraction unit 10 are extracted in a state where the aspect ratios are different. This is because the matching work by the matching unit 11 may not be executed accurately. However, it is possible to perform matching work to some extent without necessarily matching the viewpoints.

Therefore, in order to align the viewpoints of the three-dimensional point cloud data 31 and the image data 22, for example, when the cursor 29 by the mouse as the input unit 7 is aligned with the position of the point cloud camera 21a on the screen 4a, the point cloud camera 21a The coordinates of are displayed (viewpoint correspondence processing step ST3). Then, a message 42 asking whether or not to match the viewpoint of the viewer with the viewpoint of the camera is displayed. Then, "Yes" is selected in the Yes check box (viewpoint correspondence processing step ST3). After that, as shown in FIG. 6, the display 4 displays the screen 4b so that the viewpoints of the three-dimensional point cloud data 31 and the image data 22 match (viewpoint corresponding screen display processing step ST4 in FIG. 4).

The point cloud camera 21a may not exist on the screen 4a. In that case, for example, a facility management database (not shown) that records and manages the absolute coordinates of the camera installation position when the camera 21 is installed is referred to. Then, the viewpoint correspondence processing step ST3 may be executed by inputting the referenced absolute coordinate data in the input unit 7 and making it correspond to the absolute coordinates of the point cloud data 31.

FIG. 6 is a diagram showing a configuration of a message input screen 44 that functions as a calculation range setting unit 6. The calculation range setting unit 6 sets the calculation range for performing the calculation in the three-dimensional point cloud data 31 (calculation range setting step ST5 in FIG. 4). That is, the calculation unit 8 performs a calculation on the selection data 31a selected from the three-dimensional point cloud data 31 based on the setting of the calculation range by the calculation range setting unit 6.

In addition, only the range of the matching work by the matching unit 11 may be set by the setting of the calculation range setting unit 6, and the feature information may be extracted by the feature information extraction unit 10 in the entire area of the three-dimensional point group data 31. The range of both the feature information extraction by the information extraction unit 10 and the matching work by the matching unit 11 may be set. That is, at least the range of the matching work of the matching unit 11 in the three-dimensional point cloud data 31 may be set.

A specific input method for the message input screen 44 will be described. In order to match the image data 22 with the three-dimensional point cloud data 31, it is preferable to set an area of the three-dimensional point cloud data 31 including the entire image data 22. However, if a wide area is set as the calculation range in the three-dimensional point cloud data 31, the amount of calculation by the calculation unit 8 will increase. Therefore, it is preferable to set the range around the structure as a mark, which is commonly displayed on the screen frame 41 and the screen 4b, as the calculation range to be calculated.

For example, when the warehouse 27 (point cloud warehouse 27a, image warehouse 27b) is used as a common marker, the coordinates (X1, Y1, Z1) of the point cloud warehouse 27a are displayed by moving the cursor 29 to the point cloud warehouse 27a. Then, the calculation range (X1-α1 ≦ X1 ≦ X1 + α2, Y1-β1 ≦ X1 ≦ X1 + β2, Z1-γ1 ≦ X1 ≦ X1 + γ2) is set on the message input screen 44 based on the coordinate information. By setting the calculation range using the coordinates of the structure commonly displayed in the three-dimensional point cloud data 31 and the image data 22 as a mark in this way, it is possible to reliably execute the matching work while suppressing the calculation amount. It will be possible.

In the calculation range setting step ST5, the calculation range area 44a, which is the calculation range, may be displayed on the screen 4b based on the calculation range input on the message input screen 44. By doing so, the user can know which area is set as the calculation range, and if the set calculation range is inappropriate, it is possible to easily correct it.

As an example of the calculation range setting unit 6, a method of setting a calculation range by user input on the message input screen 44 has been described, but the present invention is not limited to this. For example, the calculation range may be set by using the attribute information included in the image data 22. As the attribute information, for example, information regarding the shooting direction 121a and the shooting position 121b regarding the camera 21 and information such as a lens regarding the camera 21 are assumed.

Specifically, since the approximate range of shooting by the image data 22 can be known from the shooting direction 121a, the shooting position 121b, the focal length of the lens of the camera 21, and the like, a range in which the approximate range has a predetermined margin. May be set as the calculation range. Alternatively, the calculation range may be temporarily set automatically, and the user may adjust the calculation range based on the temporary setting to set the formal calculation range.

FIG. 7 shows a feature information extraction operation (feature information extraction process of FIG. 4) in which each feature information is extracted from the image data 22 and the three-dimensional point group data 31 acquired by the feature information extraction unit 10 via the data acquisition unit 2. Step ST6) and the matching operation (matching process step ST7 in FIG. 4) for extracting the position and size of the image data 22 in which the feature information of each within the set calculation range best matches by the matching unit 11. It is a figure for demonstrating. For example, edge information is assumed as feature information.

The edge information is boundary line information obtained by extracting the feature information of the image data 22 and the three-dimensional point cloud data 31. For example, in the case of the image data 22, when the monitoring target is a river, the boundary line can be extracted because the colors differ greatly between the water surface and the land area. When the monitoring target is a road, the boundary line can be extracted because the color of the road shoulder is different from that of the paved road.

On the other hand, in the case of the three-dimensional point cloud data 31, the point cloud is a set of x-coordinate, y-coordinate, and z-coordinate data. For example, when the monitoring target is a river, the water surface is almost parallel to the xy plane, whereas the water surface. Since the land area in the vicinity is inclined, the boundary line between the water surface and the land area can be taken.

FIGS. 7A and 7B are diagrams for explaining the extraction of edge information regarding the three-dimensional point cloud data 31. FIG. 7A is an initial state, and FIG. 7B is an extraction result of edge information. In order to make the explanation easy to understand, the display of the image data display area 41 is omitted on the screen 4b.

On the other hand, FIGS. 7 (C) and 7 (D) are diagrams for explaining the extraction of edge information regarding the image data 22. FIG. 7C is an initial state, and FIG. 7D is an extraction result of edge information. The image data display areas 41 of FIGS. 7 (C) and 7 are shown in an enlarged manner so that the images (A) and (B) of FIG. 7 can be easily compared. FIG. 7E is a diagram showing the results of matching work based on the extraction results of FIGS. 7B and 7D.

Hereinafter, the feature information extraction work and the matching work will be described in detail. In FIG. 7B, the edge information is extracted with the first point group boundary 25a1 as the first point group boundary edge 45a1 and the edge information with the second point group boundary 25a2 as the second point group boundary edge 45a2. It is displayed that the point cloud warehouse 27a is extracted as the point cloud warehouse edge 46a and the edge information is extracted as the point cloud housing 28a as the point cloud housing edge 47a.

On the other hand, in FIG. 7D, the image river 25b has edge information extracted as an image river edge 45b, the image warehouse 27b has edge information extracted as an image warehouse edge 46b, and the image house 28b has edge information as an image house edge 47b. Is extracted. Although the image river water 25b1 was displayed in FIG. 7 (C), it is not extracted as edge information, so that there is no corresponding one in FIG. 7 (D).

In addition, in FIG. 7B or D, in order to make the edge processing more appropriate while grasping the state of the edge processing, it is possible to adjust the parameters related to the edge extraction by moving the bar, for example. May be good. By doing so, the accuracy of the matching work in FIG. 7 (E) can be expected to be improved.

In FIG. 7 (E), the display 4 displays the screen 4c. On the screen 4c, the edge information acquired in the feature information extraction processing step ST6 is used, and in the matching work in the matching processing step ST7, the edge information of the three-dimensional point cloud data 31 and the edge information of the image data 22 are superimposed. The result is displayed.

Specifically, the screen 4c extracts the position on the 3D point cloud data and the size of the image data 22 where the edge information of the image data 22 overlaps with the edge information of the 3D point cloud data 31 most, and the extraction result. The display will be adjusted based on the position and size. In the result of this matching work, the image warehouse edge 46b roughly overlaps with the point cloud warehouse edge 46a, the image housing 47b roughly overlaps with the point cloud housing 47a, and the image river edge 45b overlaps with the first point cloud boundary edge 45a1. It can be read that are roughly overlapping.

FIG. 8 is a diagram showing that the superimposed display processing unit 12 superimposes and displays the image data 22 on the three-dimensional point cloud data 31 on the display 4 based on the matching result (superimposed display processing step ST8 in FIG. 4). For example, after the display of the screen 4c in FIG. 7 (E), after a predetermined time such as several seconds has elapsed, or when the screen 4c is clicked with the cursor 29, the display is as shown in FIG.

Specifically, the image data 22 is output in three-dimensional coordinates in the space where the point cloud data 31 exists, using the matching result and the information regarding the shooting direction 121a and the shooting position 121b. Then, the image data 22 is enlarged or reduced based on the matching result without changing the aspect ratio (aspect ratio), and the display processing unit 3 superimposes and displays the image data 22 on the three-dimensional point cloud data 31. I do. That is, the image river 25b is superimposed and displayed on the point cloud river 25a, the image warehouse 27b is superimposed and displayed on the point cloud warehouse 27a, and the image housing 28b is superimposed and displayed on the point cloud housing 28a.

When the image data 22 used by the matching unit 11 is image data at a certain point in time, the superimposed display processing unit 12 uses the matching result to three-dimensionally display the image data 22 acquired by the camera 21 in the future. It may be superimposed and displayed at an appropriate position with respect to the point cloud data 31.

In this way, on the screen 4c, the user perceives the real-time information to be monitored by the content displayed in the image data display area 41a on which the image data 22 is displayed, and the situation around the image data display area 41a is 3 It can be grasped from the contents of the dimension point cloud data 31, and it is possible to monitor the comprehensive monitoring target.

Although it has been explained that the three-dimensional point cloud display device 1 includes the arithmetic unit 8 in FIG. 1, for example, in a server or workstation (not shown) having a higher computing power than the three-dimensional point cloud display device 1. The arithmetic unit 8 may be stored. By doing so, it becomes possible to perform feature information extraction work and matching work in a shorter time. In addition, the storage unit 30 may also be stored in a server (not shown).

Further, although it has been described that the feature information extraction work and the matching work are displayed on the screen 4b or the screen 4c in FIG. 7, the feature information extraction work and the matching work may be executed as internal processing without being displayed on the screen 4b or the screen 4c. By reducing the load of the display processing unit 3, the processing capacity of the three-dimensional point cloud display device 1 is increased, and the processing speeds of the feature information extraction work and the matching work can be improved as compared with the case where the internal processing is not performed.

According to the three-dimensional point group display device 1 according to the first embodiment of the present invention, the data acquisition unit 2 that acquires the image data 22 obtained by photographing the monitored object with the camera 21 and the camera among the monitored objects. The display processing unit 3 that performs processing to display the three-dimensional point group data 31 obtained by measuring including at least a part of the range photographed by 21 on the screen 4a, and the three-dimensional point group data displayed on the screen 4b. Acquired by the calculation range setting unit 6 that sets the calculation range by user input with respect to 31, and the selection data 31a and the data acquisition unit 2 selected from the three-dimensional point group data 31 based on the settings of the calculation range setting unit 6. Based on the result of extraction by the feature information extraction unit 10 that extracts each feature information of the image data 22 and the feature information extraction unit 10, the feature information of the three-dimensional point group data 31 and the feature information of the image data 22 are exchanged with each other. Since the matching unit 11 for matching and the superimposed display processing unit 12 for superimposing and displaying the image data on the three-dimensional point group data 31 in the screen 4c based on the matching result of the matching unit 11, GPS data is provided. Compared with the configuration in which the surrounding three-dimensional map information and the actual image are matched with reference to the above, it is possible to reliably perform matching between the three-dimensional point group data 31 and the image data 22.

Further, in the three-dimensional point cloud display device 1 according to the first embodiment of the present invention, when the display processing unit 3 makes settings in the calculation range setting unit 6, the image data 22 is displayed inside the screen 4a. Since the image data display area 41 is created, the work of setting the calculation range by the calculation range setting unit 6 can be efficiently performed.

Further, according to the three-dimensional point cloud display device 1 according to the first embodiment of the present invention, the image data display area 41 is arranged at the corner of the screen 4a when the setting is performed by the calculation range setting unit 6. Therefore, it is possible to suppress a decrease in visibility in the display area of the three-dimensional point cloud data 31 on the screen 4a, and it is possible to efficiently grasp the states of both the image data 22 and the three-dimensional point cloud data 31. ..

Further, according to the three-dimensional point cloud display device 1 according to the first embodiment of the present invention, the calculation range setting unit 6 displays the calculation range area 44a corresponding to the calculation range set by the user input on the screen 4b. Therefore, the user can know which area is set as the calculation range, and if the set calculation range is inappropriate, it can be easily corrected.

According to the three-dimensional point cloud display device 1 according to the first embodiment of the present invention, the data acquisition unit 2 that acquires the image data 22 obtained by photographing the monitored object with the camera 21 and the camera among the monitored objects. The display processing unit 3 that performs processing to display the three-dimensional point cloud data 31 obtained by measuring including at least a part of the range photographed by 21 on the screen 4a, and the three-dimensional point cloud data displayed on the screen 4b. Three-dimensional based on the settings of the calculation range setting unit 6 for setting the calculation range based on the attribute information (for example, information on the shooting direction 121a and the shooting position 121b) included in the image data 22 for 31 and the calculation range setting unit 6. Based on the feature information extraction unit 10 that extracts the feature information of each of the selection data 31a selected from the point cloud data 31 and the image data 22 acquired by the data acquisition unit 2, and the extraction result by the feature information extraction unit 10. Image data on the 3D point cloud data 31 in the screen 4c based on the matching result of the matching unit 11 that matches the feature information of the 3D point cloud data 31 and the feature information of the image data 22 with each other. Since the superimposed display processing unit 12 for superimposing and displaying the above is provided, the three-dimensional point cloud data 31 and the image data 22 are compared with the configuration for matching the surrounding three-dimensional map information with the actual image based on the GPS data. It is possible to make sure that the matching is performed.

Further, according to the three-dimensional point group display system 40 according to the first embodiment of the present invention, the camera 21 that captures the monitored object and the data acquisition unit 2 that acquires the image data 22 captured by the camera 21. The display processing unit 3 that performs processing to display the three-dimensional point group data 31 obtained by measuring including at least a part of the range photographed by the camera 21 among the monitoring targets on the screen 4a, and the screen 4b. Selection data selected from the 3D point group data 31 based on the settings of the calculation range setting unit 6 that sets the calculation range by user input for the displayed 3D point group data 31 and the calculation range setting unit 6. The feature information extraction unit 10 that extracts the feature information of each of the image data 22 acquired by the 31a and the data acquisition unit 2, and the feature information of the three-dimensional point group data 31 based on the extraction result by the feature information extraction unit 10. A matching unit 11 that matches the feature information of the image data 22 with each other, and a superimposed display processing unit 12 that superimposes and displays the image data on the three-dimensional point group data 31 in the screen 4c based on the matching result of the matching unit 11. , It is possible to reliably match the 3D point group data 31 and the image data 22 as compared with the configuration of matching the surrounding 3D map information with the actual image based on the GPS data. become.

Further, according to the three-dimensional point cloud display system 40 according to the first embodiment of the present invention, when the display processing unit 3 makes settings in the calculation range setting unit 6, the image data 22 is displayed on the screen 4a. Since the image data display area 41 is created, the work of setting the calculation range by the calculation range setting unit 6 can be efficiently performed.

Further, according to the three-dimensional point group display method according to the first embodiment of the present invention, the data acquisition step ST1 for acquiring the image data 22 obtained by photographing the monitored object with the camera 21 and the monitored object among the monitored objects. Display processing step ST2 that performs processing to display the three-dimensional point group data 31 obtained by measuring including at least a part of the range captured by the camera 21 on the screen 4a, and the three-dimensional point group displayed on the screen 4b. The superimposed display processing step that sets the calculation range for the data by user input, the selection data 31a selected from the three-dimensional point group data based on the setting in the calculation range setting step ST5, and the image acquired by the data acquisition unit. Based on the feature information extraction process step ST6 that extracts each feature information of the data and the result of extraction by the feature information extraction unit, the superimposed display process that matches the feature information of the three-dimensional point group data 31 with the feature information of the image data. Since it has step ST7 and superimposed display processing step ST8 for superimposing and displaying the image data 22 on the three-dimensional point group data 31 in the screen 4c based on the result of the matching processing step ST7, the GPS data is used as a reference. Compared with the configuration in which the surrounding 3D map information and the actual image are matched, it becomes possible to reliably perform the matching between the 3D point group data and the image data.

Further, according to the three-dimensional point cloud display method according to the first embodiment of the present invention, the display processing step ST2 displays the image data 22 inside the screen 4b when the setting is performed in the calculation range setting step ST5. Since the image data display area 41 to be generated is created, the work of setting the calculation range in the calculation range setting step ST5 can be efficiently performed.

Further, according to the three-dimensional point group display program according to the first embodiment of the present invention, the data acquisition step ST1 for acquiring the image data 22 obtained by photographing the monitored object with the camera 21 and the monitored object among the monitored objects. Display processing step ST2 that performs processing to display the three-dimensional point group data 31 obtained by measuring including at least a part of the range captured by the camera 21 on the screen 4a, and the three-dimensional point group displayed on the screen 4b. The superimposed display processing step that sets the calculation range for the data by user input, the selection data 31a selected from the three-dimensional point group data based on the setting in the calculation range setting step ST5, and the image acquired by the data acquisition unit. Based on the feature information extraction process step ST6 that extracts each feature information of the data and the result of extraction by the feature information extraction unit, the superimposed display process that matches the feature information of the three-dimensional point group data 31 with the feature information of the image data. Based on the results of step ST7 and matching processing step ST7, the computer is made to execute the superimposed display processing step ST8 for superimposing and displaying the image data 22 on the three-dimensional point group data 31 in the screen 4c, so that the GPS data can be stored. Compared with the configuration in which the surrounding 3D map information and the actual image are matched as a reference, it is possible to reliably perform matching between the 3D point group data and the image data.

Further, according to the three-dimensional point cloud display program according to the first embodiment of the present invention, the display processing step ST2 displays the image data 22 inside the screen 4b when the setting is performed in the calculation range setting step ST5. Since the image data display area 41 to be generated is created, the work of setting the calculation range in the calculation range setting step ST5 can be efficiently performed.

Further, according to the recording medium according to the first embodiment of the present invention, since the program P is recorded, various functions by the program P can be executed.

Embodiment 2.
Embodiment 2 of the present invention will be described with reference to FIGS. 9 and 10. In the first embodiment, the calculation range is set by the calculation range setting unit 6 by inputting the coordinates of each axis of the X-axis, the Y-axis, and the Z-axis by the user input, but in the present embodiment, the mouse as the input unit 7 is used. It is a different part to set the calculation range by using the cursor movement by such as.

FIG. 9 is a block diagram when the calculation range is set by using the pointing device as the input unit 7. As the pointing device, for example, a mouse 7a, a joystick 7b, a trackball (not shown), or the like is assumed. The input receiving unit 5 receives the input from these pointing devices, and the calculation range setting unit 6a in the data acquisition unit 2a executes the setting of the calculation range by moving the cursor 29.

FIG. 10 is a setting screen for setting a calculation range using a pointing device. Consider the area that the user wants to be the calculation range for the display in the screen coordinate system displayed in the screen 4b. For example, the cursor 29 is moved by the operation of the mouse 7a to determine the calculation range in the X-axis direction and the Z-axis direction. Then, when determining the calculation range in the Y-axis direction, which is the depth, the viewpoint movement tool 51 is used to shift the viewpoint in the Y-axis direction, and the same operation as the movement operation of the cursor 29 on the X-axis or the Z-axis is performed. Just do it.

In the three-dimensional point cloud display device 1 according to the second embodiment of the present invention, the calculation range setting unit 6 sets the calculation range by using the pointing devices 7a and 7b, so that the user can visually calculate the calculation range. It becomes possible to do a range. That is, it is easier to set which range is set as the calculation range as compared with the case of inputting a numerical value as a coordinate.

Embodiment 3.
Embodiment 3 of the present invention will be described with reference to FIGS. 11 and 12. In the first embodiment, the matching work is a work of extracting the position and size of the image data in which the feature information of each of the image data and the three-dimensional point cloud data best matches within the set calculation range. The difference is that it has a function to prompt the resetting of the calculation range when the degree of is not good.

FIG. 11 is a block diagram showing the configuration of the calculation unit 8a. The calculation unit 8a includes a viewpoint correspondence unit 9, a feature information extraction unit 10, a matching unit 11, and a matching degree determination unit 11a for determining the degree of matching. The matching degree determination unit 11a functions to grasp how much the result of the matching work in the matching unit 11 is with respect to the predetermined matching degree. As a method for obtaining the degree of matching, for example, in the calculation range set in the 3D point group data 31, the number of pixels that the image data 22 matches with the 3D point group data 31 and the number of pixels that do not match are counted. However, it may be obtained from the ratio of the number of matching pixels to the total number of pixels.

FIG. 12 is a flowchart showing the operation of the three-dimensional point cloud display device 1. Steps ST1 to ST8 have the same configuration as that of the first embodiment, and steps ST2 to ST4 are omitted. The matching degree confirmation step ST17 and the warning display step ST18 for prompting the resetting of the calculation range are different steps from the first embodiment.

For example, if the result of the matching work in the matching unit 11 has not reached the matching degree predetermined in the matching degree confirmation step ST17, the flow branches to the warning display step ST18, and the display processing unit 3 “again”. Please set the calculation range and try the calculation again. " After that, the user shifts to the calculation range setting step ST5 again based on the display of the warning 144, and sets the calculation range with the cursor 29 or the like. It should be noted that the warning 144 may be anything that can be transmitted to the user, such as voice, other than characters.

The three-dimensional point cloud display device 1 according to the third embodiment of the present invention has a matching degree determination unit 11a for determining a matching result by the matching unit 11, and resets the calculation range according to the determination result. Since the urging warning 144 is notified, it is effective in preventing the superimposed display of the image data 22 on the three-dimensional point cloud data 31 from being displayed with poor accuracy.

Modification 3a
Modification 3a will be described with reference to FIG. When the user receives the warning display step ST18 and sets the calculation range again in the calculation range setting step ST5, even if the user knows the previous calculation range and newly sets the calculation range, the matching work is performed again. There is a risk that the degree of matching will not be improved and the degree of matching will not reach a predetermined level.

Therefore, it is possible to display the area of the previous calculation range as the previous calculation range area 51 and set the calculation range as the latest calculation range area 52 based on the display. That is, it is possible to grasp the area lacking in the matching work in the matching unit 11 in the previously set calculation range, and to set the latest calculation range area 52 more efficiently.

In the three-dimensional point cloud display device 1 according to the modified example 3a of the third embodiment of the present invention, when the warning 144 is notified and the calculation range is set again, the previous calculation range area 51, which is the warning target, is displayed on the screen. Since it is displayed, it has the effect of being able to efficiently reset the calculation range.

Embodiment 4.
Embodiment 4 of the present invention will be described with reference to FIGS. 14 to 16. In the first embodiment, it was mainly assumed that the camera is a fixed camera in which the field of view does not change, but in the fourth embodiment, it is assumed that the camera is a movable camera capable of shooting by changing the angle to the left or right or up and down. Assuming, the matching work is performed using the image data obtained by photographing a wider area than that of the first embodiment.

FIG. 14 is a block diagram showing the configuration of the three-dimensional point cloud display system 40a. A configuration different from the three-dimensional point cloud display system 40 in FIG. 1 of the first embodiment will be described. The camera unit 20a includes a movable camera 21a and a data receiving unit 123 that receives a movable camera control signal 13a that controls the movable camera 21a sent from the three-dimensional point cloud display device 1a.

The three-dimensional point group display device 1a includes a camera control unit 13 that sends a movable camera control signal 13a, a movable camera 21a, a program Pa as a three-dimensional point group display program that generates a movable camera control signal 13a, and a movable camera 21a. It includes an image synthesizing unit 14 for synthesizing a plurality of image data 22 captured by the camera to create a composite image data 22a. Further, the storage unit 30 has a function of receiving and storing a plurality of image data 22 synthesized by the plurality of image synthesizing units 14 from the data acquisition unit 2.

When the input receiving unit 5 receives an instruction to control the movable camera 21a by the input from the input unit 7, the instruction is transmitted to the camera control unit 13. The camera control unit 13 receives an instruction and transmits a movable camera control signal 13a to the data reception unit 123 in the camera unit 20a. The movable camera 21a is configured to receive the movable camera control signal 13a and perform various operations.

FIG. 15 is a flowchart showing the operation of the three-dimensional point cloud display system 40a when the movable camera control signal 13a is transmitted from the camera control unit 13 and the movable camera 21a receives the movable camera control signal 13a. The operation of the three-dimensional point cloud display system 40a will be described with reference to FIGS. 14 and 16 as appropriate.

First, the movable camera 21a shoots in a certain direction such as an initial position. Then, for example, continuous shooting is performed in the horizontal direction so that the boundary portions of the image data 22 overlap each other at a plurality of predetermined angles (multiple shooting step STA). Then, the captured image data 22 is sequentially stored in the storage unit 30 via the data transmission unit 23 and the data acquisition unit 2 (image storage step STB). That is, the movable camera 21a shoots in a plurality of shooting directions.

After that, when the shooting by the movable camera 21a is completed at all the plurality of predetermined angles, the data acquisition unit 2 communicates with the storage unit 30 and sends the stored image data 22 to the image synthesis unit 14. The image synthesizing unit 14 synthesizes a plurality of received image data 22 to create a composite image data 22a (image data synthesizing step STC). After the composite image data 22a is created, steps ST1 to ST8 are executed as in the first embodiment.

FIG. 16 is a diagram showing the contents of the screen 4c on the display 4 for displaying the three-dimensional point cloud data 31. The composite image data 22a is displayed in the image data display area 41a. Of the image data display areas 41a, the image data 22 displayed in the first image data display area 41a1 is taken by the shooting range as the initial position of the movable camera 21a, and is displayed in the second image data display area 41a2. The image data 22 to be imaged is taken according to the shooting range when the shooting direction of the movable camera 21a is changed.

The boundary between the shooting range in a certain shooting direction (for example, the initial position) and the shooting range in another shooting direction when the shooting direction is changed is displayed as the boundary line 42. If an abnormality occurs in the image such as an unusual noise reflected inside the composite image data 22a, the boundary line 42 is displayed to display the first image data display area 41a1 or the second image data display area. Since it is possible to easily grasp which of 41a2 is the cause, it is efficient when the composite image data 22a is regenerated.

It is described that the composite image data 22a is created from the image data 22 obtained by changing the shooting direction and the continuous shooting range is acquired, and then the feature information is extracted by the feature extraction unit 10 and the matching is performed by the matching unit 11. However, it does not necessarily have to be the image data 22 obtained by acquiring a continuous shooting range. That is, the image data of the first image data (not shown) taken in a certain shooting direction and the second image data (not shown) taken in a variable shooting direction may be discrete, and in this case, the image data may be discrete. As the relationship between the first image data and the second image data, it suffices if the relationship between the image data such as the captured angle information can be grasped.

In this way, by using the plurality of image data 22 shot in the plurality of shooting directions, it is possible to increase the extraction of the feature information by the feature extraction unit 10 as compared with the case of the first embodiment. The matching result by the matching unit 11 can be matched in a wider area. As a result, the image data 22 can be superimposed and displayed on the three-dimensional point cloud data 31 on the screen 4 more accurately than in the first embodiment.

In the three-dimensional point cloud display device 1 according to the fourth embodiment of the present invention, the data acquisition unit 2 has a plurality of images taken by the movable camera 21a in a plurality of directions when the camera 21 is a movable camera 21a whose shooting direction is variable. The image data 22 of the above is acquired, and the feature information extraction unit 10 extracts the feature information using the plurality of image data 22. Therefore, more accurately, on the three-dimensional point cloud data 31 in the screen 4c. The image data 22 can be superimposed and displayed.

Modification 4a
This modification uses the image taken by the movable camera and the 3D point cloud data to obtain the specifications of the movable camera whose unit movement amount in the shooting direction for each command is unknown, and is useful for monitoring work. This will be described with reference to FIG.

FIG. 17 is a flowchart showing the operation of this modification. Step STA to step ST7 are the same as those in the fourth embodiment. Step ST9 is the content added in this modification.

In step STA, the user transmits a command to change the shooting direction to the movable camera 21 via the camera control unit 13 shown in FIG. The unit movement amount of the movable camera 21a in this modification is unknown, for example, the user sensuously sustains a movement command step (not shown) for moving the movable camera 21a to the left for 1 second. board.

Therefore, when the shooting direction is changed in step STA, the information about the movement command step is recorded, and the information about the movement command step and the result in the matching processing step ST7 are used to use the unit movement instruction step in the unit movement amount calculation step ST9. The unit movement amount (unit movement amount in the shooting direction) of the movable camera 21a for each is obtained.

By having the unit movement amount calculation step ST9, the unit movement amount of the movable camera 21a is known for each unit movement command step, so that more precise operation can be performed in the monitoring work.

1 3D point group display device, 2 data acquisition unit, 3 display processing unit, 4 display unit, 4a, 4b, 4c screen, 5 input reception unit, 6 calculation range setting unit, 7 input unit, 7a mouse (pointing device) , 7b Joystick (pointing device), 8 calculation unit, 9 viewpoint correspondence unit, 10 feature information extraction unit, 11 matching unit, 11a matching degree determination unit, 12 superimposition display processing unit, 20 camera unit, 21 camera, 22 image data, 23 Data transmission unit, 30 Storage unit, 31 3D point group data, 31a Selection data, 40 3D point group display system, 41 Image data display area, 44a Calculation range area (calculation range), 45a1 First point group Boundary edge (edge information, feature information), 45a2 Second point group boundary edge (edge information, feature information), 45b Image river edge (edge information, feature information), 46a Point group warehouse edge (edge information, feature information) , 46b Image warehouse edge (edge information, feature information), 47a Point group housing edge (edge information, feature information), 47b Image housing edge (edge information, feature information), 51 Previous calculation range area, 101 Memory (recording medium) , 121a Shooting direction (attribute information), 121b Shooting position (attribute information), 144 Warning P, Pa program (3D point group display program), ST1 data acquisition step, ST2 display processing step, ST5 calculation range setting step, ST6 features Information extraction processing step, ST7 matching processing step, ST8 superimposed display processing step, ST9 unit movement amount calculation step, STA multiple shooting step, STB image storage step, STC image data synthesis step

Claims (16)

A data acquisition unit that acquires image data obtained by shooting a monitored object with a camera,
A display processing unit that performs a process of displaying the three-dimensional point cloud data obtained by measuring including at least a part of the range photographed by the camera among the monitored objects on the screen.
A calculation range setting unit that sets a calculation range by user input for the three-dimensional point cloud data displayed on the screen, and a calculation range setting unit.
A feature information extraction unit that extracts feature information of each of the selection data selected from the three-dimensional point cloud data based on the setting of the calculation range setting unit and the image data acquired by the data acquisition unit.
A matching unit that matches the feature information of the three-dimensional point cloud data with the feature information of the image data based on the result of the extraction by the feature information extraction unit.
A superimposed display processing unit that superimposes and displays the image data on the three-dimensional point cloud data in the screen based on the matching result of the matching unit.
A three-dimensional point cloud display device equipped with. The three-dimensional point cloud display device according to claim 1, wherein the display processing unit creates an image data display area for displaying the image data inside the screen when the setting is performed in the calculation range setting unit. .. The three-dimensional point cloud display device according to claim 2, wherein the image data display area is arranged at a corner of the screen. The three-dimensional point cloud display device according to any one of claims 1 to 3, wherein the calculation range setting unit displays a calculation range area corresponding to the calculation range on the screen. The three-dimensional point cloud display device according to any one of claims 1 to 4, wherein the calculation range setting unit sets the calculation range by using a pointing device. It has a matching degree determination unit that determines the result of the matching by the matching unit.
The three-dimensional point cloud display device according to any one of claims 1 to 5, which notifies a warning prompting the resetting of the calculation range according to the result of the determination. The three-dimensional point according to claim 6, wherein when the warning is notified and the calculation range is set again, the previous calculation range area, which is the area of the previous calculation range that was the target of the warning, is displayed on the screen. Group display device. When the camera is a movable camera whose shooting direction is variable, the data acquisition unit acquires a plurality of the image data in a plurality of directions shot by the movable camera.
The three-dimensional point cloud display device according to any one of claims 1 to 7, wherein the feature information extraction unit extracts feature information using a plurality of the image data. A data acquisition unit that acquires image data obtained by shooting a monitored object with a camera,
A display processing unit that performs a process of displaying the three-dimensional point cloud data obtained by measuring including at least a part of the range photographed by the camera among the monitored objects on the screen.
A calculation range setting unit that sets a calculation range based on the attribute information included in the image data for the three-dimensional point cloud data displayed on the screen.
A feature information extraction unit that extracts feature information of each of the selection data selected from the three-dimensional point cloud data based on the setting of the calculation range setting unit and the image data acquired by the data acquisition unit.
A matching unit that matches the feature information of the three-dimensional point cloud data with the feature information of the image data based on the result of the extraction by the feature information extraction unit.
A superimposed display processing unit that superimposes and displays the image data on the three-dimensional point cloud data in the screen based on the matching result of the matching unit.
A three-dimensional point cloud display device equipped with. A camera that shoots the object to be monitored, and
A data acquisition unit that acquires image data obtained by shooting with the camera, and
A display processing unit that performs a process of displaying the three-dimensional point cloud data obtained by measuring including at least a part of the range photographed by the camera among the monitored objects on the screen.
A calculation range setting unit that sets a calculation range by user input for the three-dimensional point cloud data displayed on the screen, and a calculation range setting unit.
A feature information extraction unit that extracts feature information of each of the selection data selected from the three-dimensional point cloud data based on the setting of the calculation range setting unit and the image data acquired by the data acquisition unit.
A matching unit that matches the feature information of the three-dimensional point cloud data with the feature information of the image data based on the result of the extraction by the feature information extraction unit.
A superimposed display processing unit that superimposes and displays the image data on the three-dimensional point cloud data in the screen based on the matching result of the matching unit.
3D point cloud display system equipped with. The three-dimensional point cloud display system according to claim 10, wherein the display processing unit creates an image data display area for displaying the image data inside the screen when the setting is performed in the calculation range setting unit. .. A data acquisition step to acquire the image data obtained by shooting the monitored object with a camera, and
A display processing step for displaying on the screen the three-dimensional point cloud data obtained by measuring including at least a part of the range photographed by the camera among the monitoring targets.
A calculation range setting step for setting a calculation range by user input for the three-dimensional point cloud data displayed on the screen, and a calculation range setting step.
A feature information extraction processing step for extracting feature information of each of the selection data selected from the three-dimensional point cloud data based on the setting in the calculation range setting step and the image data acquired by the data acquisition step.
A matching processing step for matching the feature information of the three-dimensional point cloud data with the feature information of the image data based on the result of the extraction by the feature information extraction processing step.
Based on the matching result in the matching processing step, the superimposed display processing step of superimposing and displaying the image data on the three-dimensional point cloud data in the screen, and
A three-dimensional point cloud display method having. The three-dimensional point cloud display method according to claim 12, wherein the display processing step creates an image data display area for displaying the image data inside the screen when the setting is performed in the calculation range setting step. .. A data acquisition step to acquire the image data obtained by shooting the monitored object with a camera, and
A display processing step for displaying on the screen the three-dimensional point cloud data obtained by measuring including at least a part of the range photographed by the camera among the monitoring targets.
A calculation range setting step for setting a calculation range by user input for the three-dimensional point cloud data displayed on the screen, and a calculation range setting step.
A feature information extraction processing step for extracting feature information of each of the selection data selected from the three-dimensional point cloud data based on the setting in the calculation range setting step and the image data acquired by the data acquisition step.
A matching processing step for matching the feature information of the three-dimensional point cloud data with the feature information of the image data based on the result of the extraction by the feature information extraction processing step.
Based on the matching result of the matching processing step, the superimposed display processing step of superimposing and displaying the image data on the three-dimensional point cloud data in the screen,
A 3D point cloud display program for making a computer execute. The three-dimensional point cloud display program according to claim 14, wherein the display processing step creates an image data display area for displaying the image data inside the screen when the setting is performed in the calculation range setting step. .. A recording medium on which the three-dimensional point cloud display program according to claim 14 or 15 is recorded.

Images