US10354447B2 - Image processing device, image processing method, and image processing program - Google Patents

Image processing device, image processing method, and image processing program Download PDF

Info

Publication number
US10354447B2
US10354447B2 US15/704,215 US201715704215A US10354447B2 US 10354447 B2 US10354447 B2 US 10354447B2 US 201715704215 A US201715704215 A US 201715704215A US 10354447 B2 US10354447 B2 US 10354447B2
Authority
US
United States
Prior art keywords
point cloud
cloud data
image
cut
plane
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US15/704,215
Other versions
US20180082472A1 (en
Inventor
Naoki Morikawa
Tadayuki Ito
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Topcon Corp
Original Assignee
Topcon Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Topcon Corp filed Critical Topcon Corp
Assigned to TOPCON CORPORATION reassignment TOPCON CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: ITO, TADAYUKI, MORIKAWA, NAOKI
Publication of US20180082472A1 publication Critical patent/US20180082472A1/en
Application granted granted Critical
Publication of US10354447B2 publication Critical patent/US10354447B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/003—Navigation within 3D models or images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/005—General purpose rendering architectures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/285—Analysis of motion using a sequence of stereo image pairs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images

Definitions

  • the present invention relates to an image processing technique for point cloud data.
  • Point cloud data that are obtained by using a laser scanner or by stereophotogrammetry are publicly known (for example, refer to WO 2011/070927 and Japanese Unexamined Patent Application Laid-Open No. 2012-230594).
  • Point cloud data of an object can generate a 3-D image of the object when three-dimensionally displayed on a screen of a PC or other device.
  • the point cloud data is typically used for generating a three-dimensional model therefrom.
  • the three-dimensional model is data showing an outline of an object and has a high affinity with three-dimensional data that can be processed by CAD software.
  • a 3-D image of an object is displayed by a set of dots.
  • the point cloud data is raw data that is not a three-dimensional model data
  • the outline of the object can be indistinct, and it would be difficult to instantly recognize which part is a periphery or an edge of the object (for example, a building).
  • the 3-D image of the point cloud may be modified, for example, by changing the colors of the dots in accordance with difference in directions of planes, to enable easy understanding of its stereoscopic structure.
  • this 3-D image of the point cloud can still cause the above problem. In the case of using raw data that is not processed, the above problem tends to occur.
  • FIG. 2A is a perspective view of an example of a 3-D image of point cloud data of a cubic object. As shown in FIG. 2A , when the image is displayed by simple dots, the perspective is difficult to recognize, and it is difficult to distinguish which part is a periphery of the cubic object.
  • FIG. 2B shows an example of an image of point cloud data that is superimposed with peripheries of a cubic object.
  • the peripheries of the cubic object are extracted from a three-dimensional model data of the point cloud data.
  • the perspective is easily recognized.
  • the image shown in FIG. 2B is obtained after the point cloud data is processed into the three-dimensional model data. That is, the image shown in FIG. 2B cannot be displayed by using the point cloud data before the point cloud data is processed into the three-dimensional model data.
  • an object of the present invention is to provide a technique for facilitating understanding of a stereoscopic structure of an object in a 3-D image of point cloud data.
  • a first aspect of the present invention provides an image processing device including a first point cloud data display controlling unit, a marker display controlling unit, and a second point cloud data display controlling unit.
  • the first point cloud data display controlling unit displays a 3-D image of point cloud data on a screen.
  • the marker display controlling unit displays a marker to specify a cut section of the 3-D image of the point cloud data.
  • the second point cloud data display controlling unit displays a point cloud that constitutes the cut section among the point cloud data, as another image.
  • the displayed marker may be movable in a direction perpendicular to the cut section to cut another section, and a point cloud constituting the another cut section may be extracted from among the point cloud data.
  • a point cloud in a predetermined area in the direction perpendicular to the cut section may be extracted as the point cloud constituting the cut section.
  • an image of the point cloud data that constitutes the cut section may be colored differently from an image of the rest of the point cloud data.
  • the marker in any one of the first to the fourth aspects of the present invention, may indicate a range of the cut section, and one or both of the position and the size of the marker may be changeable.
  • the direction of the cut section may be set so as to align with the direction of a plane of the 3-D image of the point cloud data.
  • a seventh aspect of the present invention provides an image processing method including displaying a 3-D image of point cloud data on a screen, displaying a marker to specify a cut section of the 3-D image of the point cloud data, and displaying a point cloud that constitutes the cut section among the point cloud data, as another image.
  • An eighth aspect of the present invention provides a non-transitory computer recording medium storing computer executable instructions that, when executed by a computer processor, cause the computer processor to display a 3-D image of point cloud data on a screen, display a marker to specify a cut section of the 3-D image of the point cloud data, and display a point cloud that constitutes the cut section among the point cloud data, as another image.
  • the present invention provides a technique for facilitating understanding of a stereoscopic structure of an object in a 3-D image of point cloud data.
  • FIG. 1 is a block diagram of an embodiment.
  • FIGS. 2A and 2B are conceptual diagrams each showing an example of point cloud data that is displayed on a screen.
  • FIG. 3 is a conceptual diagram showing an example of point cloud data that is displayed on a screen.
  • FIGS. 4A and 4B are conceptual diagrams each showing an example of point cloud data that is displayed on a screen.
  • FIG. 5 shows a 3-D image of point cloud data that is processed.
  • FIG. 6 shows the 3-D image (intensity image) of the point cloud data shown in FIG. 5 and a sectional image of a part of the 3-D image.
  • FIG. 7 shows a 3-D image of point cloud data.
  • FIG. 8 shows a 3-D image of point cloud data.
  • FIG. 9 shows a 3-D image of point cloud data.
  • FIG. 10 shows a 3-D image of point cloud data.
  • FIG. 11 shows a 3-D image of point cloud data.
  • FIG. 12 shows a 3-D image of point cloud data.
  • FIG. 1 shows an image processing device 100 that is configured to process point cloud data.
  • the image processing device 100 is constructed not by dedicated hardware but by application software using a personal computer (PC).
  • the application software may be installed in the PC and is operated to execute the function of the image processing device 100 .
  • each functional unit shown in FIG. 1 is constructed by software.
  • Each of the functional units shown in FIG. 1 may be composed of a dedicated arithmetic circuit.
  • a functioning unit constructed of software and a functioning unit composed of a dedicated arithmetic circuit may be used together.
  • each of the functional units shown in the drawing may be constructed of an electronic circuit such as a central processing unit (CPU), an application specific integrated circuit (ASIC), or a programmable logic device (PLD) such as a field programmable gate array (FPGA).
  • CPU central processing unit
  • ASIC application specific integrated circuit
  • PLD programmable logic device
  • FPGA field programmable gate array
  • each of the functional units is to be constructed of dedicated hardware or is to be constructed of software so that programs are executed by a CPU is selected in consideration of necessary operating speed, cost, amount of electricity consumed, and other factors. For example, if a specific functional unit is composed of an FPGA, the operating speed is superior, but the production cost is high. On the other hand, if a specific functional unit is configured so that programs are executed by a CPU, the production cost is reduced because hardware resources are conserved. However, when the functional unit is constructed using a CPU, its operating speed is inferior to that of dedicated hardware. Constructing the functional unit by dedicated hardware and constructing the functional unit by software differ from each other, as described above, but are equivalent to each other from the viewpoint of obtaining a specific function.
  • the image processing device 100 includes a point cloud data obtaining unit 101 , a point cloud data display controlling unit 102 , a marker display controlling unit 103 , a specified point cloud extracting unit 104 , and a specified point cloud display controlling unit 105 .
  • the point cloud data obtaining unit 101 obtains point cloud data that is measured by a three-dimensional laser scanner, therefrom, through an appropriate communication means.
  • the point cloud data may also be obtained by a method of calculating three-dimensional coordinates of numerous feature points that are extracted from photographed images, based on the principle of stereophotogrammetry.
  • the point cloud data display controlling unit 102 displays the point cloud data on an appropriate image displaying unit, such as a screen of a PC.
  • the point cloud data may be displayed by dots as they are or by changing the colors of the dots, or may be displayed after being subjected to various kinds of image processing, such as removal of noise components.
  • the marker display controlling unit 103 controls display of a marker to cut a 3-D image of the point cloud data at a freely selected flat plane.
  • FIG. 3 shows point cloud data 150 of a cubic object, which represents a building, and shows a marker 200 as an example of the displayed marker.
  • the marker 200 is movable in an X direction or a specific horizontal direction by operation of an operator, which may be performed by using a mouse.
  • the marker 200 is moved to specify a position (position on the X axis) in the 3-D image of the point cloud data to be cut along a Y-Z plane.
  • the Z direction is a vertical direction
  • the Y direction is a horizontal direction orthogonally crossing the X direction.
  • the azimuths shown in FIG. 3 may be set such that the X direction is set in the north-south direction (or the east-west direction), the Y direction is set in the east-west direction (or the north-south direction), and the Z direction is set in the vertical direction.
  • This setting facilitates understanding of a relationship between a cut section and the azimuths.
  • the directions of the X axis, Y axis, and Z axis may be set in accordance with extending directions of walls of the building.
  • FIG. 3 shows an example of this case.
  • FIGS. 4A and 4B each show an example of a sub-image that is displayed separately from the image shown in FIG. 3 on the same screen at the same time.
  • the sub-image may be displayed on a screen that is different from the screen showing a main image or may be displayed on the same screen by switching from the main image.
  • FIG. 4A shows point cloud data 151 of a plane shape that is obtained by cutting a periphery on the X axis of the point cloud data 150 of the cubic object shown in FIG. 3 along the Y-Z plane.
  • a front end plane on the X axis of the point cloud data 150 is cut along the Y-Z plane, and the cut part of the plane is displayed.
  • the processing step of cutting the 3-D image of the point cloud data 150 shown in FIG. 3 along the Y-Z plane and extracting the point cloud data 151 and 152 as shown in FIGS. 4A and 4B is performed as described below.
  • threshold values are set, and the points in proximity to the Y-Z plane are extracted in an area within the threshold values.
  • the 3-D image of the point cloud data is cut by using the marker 200 .
  • the point cloud constituting the cut section is displayed as another image separately from the 3-D image.
  • This facilitates understanding of the stereoscopic structure of the point cloud data.
  • peripheries of the point cloud data 150 which are respectively cut along the Y-Z plane, are successively displayed. Tracing the display history facilitates understanding of the stereoscopic shape (peripheries) of the object, which is the base for the point cloud data 150 .
  • the moving direction of the marker 200 is set by an operator as desired.
  • the marker 200 may be moved automatically, and point clouds constituting cut sections, which are successively obtained, may be displayed by thumbnail images aligning on a time axis.
  • Such a UI display also facilitates understanding of the stereoscopically distributed condition of the point cloud data.
  • the Y-Z plane to be used for the cutting has a thin thickness, but it may have a large thickness. In such case, a point cloud that is distributed also in the X axis direction is extracted.
  • the specified point cloud extracting unit 104 extracts the point cloud data of a cut section that is cut along a specified plane of the 3-D image of the point cloud data. For example, the specified point cloud extracting unit 104 cuts the 3-D image of the point cloud data 150 , as shown in FIG. 3 , and extracts the point cloud data 151 and 152 , as shown in FIG. 4 , from the point cloud data 150 .
  • the specified point cloud display controlling unit 105 controls display of the point cloud data that is extracted by the specified point cloud extracting unit 104 .
  • the specified point cloud display controlling unit 105 controls display of the image as shown in FIG. 4 .
  • the image processing device 100 includes the point cloud data display controlling unit 102 , the marker display controlling unit 103 , and the specified point cloud display controlling unit 105 .
  • the point cloud data display controlling unit 102 displays a 3-D image of point cloud data.
  • the marker display controlling unit 103 displays the marker to specify a cut section of the 3-D image of the point cloud data.
  • the specified point cloud display controlling unit 105 displays a point cloud that constitutes the cut section among the point cloud data, as another image.
  • This structure is configured to extract a periphery of an object and displays the periphery, thereby facilitating understanding of the stereoscopically distributed condition of the point cloud data. For example, in a case of using point cloud data based on a building, a stereoscopic shape of the building is easily understood without generating a three-dimensional model.
  • FIG. 5 shows a displayed 3-D image of point cloud data of an object after the point cloud data is processed.
  • FIG. 5 shows a gray scale image, this image can be an easily understandable point cloud image in which each point is colored, in actual use.
  • the coloring processing is performed by superimposing the point cloud data on a photographic image and coloring the points in areas corresponding to the areas in the photographic image. The photographic image is photographed by a camera at the same time when the point cloud data is obtained.
  • FIG. 6 shows a point cloud image that is displayed on the left side.
  • This point cloud image is obtained by shading the point clouds of the point cloud data in FIG. 5 , in accordance with intensities of reflection light.
  • FIG. 6 shows an image of a point cloud, as seen from a direction perpendicular to a cut section, on the right side.
  • This point cloud constitutes the cut section that is cut along a strip-shaped line shown in the left image.
  • a part of the 3-D image of the point cloud data is cut (refer to the left image), and the point cloud constituting the cut section is displayed as seen from the direction perpendicular to the cut section (refer to the right image). This facilitates recognizing of the outline of the object.
  • the line for defining the cut section as shown in the point cloud image on the left side in FIG. 6 can have any width.
  • the line may have a wide width to extract points in an area having a wide width in a direction perpendicular to the cut section.
  • a marker having a function different from the function of the marker 200 may also be displayed.
  • This marker is a cut-area indication.
  • the cut-area indication marker indicates an area of a part of the 3-D image of the point cloud data to be cut.
  • the display of the cut-area indication marker is also controlled by the marker display controlling unit 103 .
  • FIG. 7 shows an example of displaying a cut-area indication marker on a displayed point cloud image. This marker has a small thickness and has a flat rectangular shape. Either one of the marker 200 or the cut-area indication marker may be displayed, or they may be displayed at the same time.
  • the 3-D image shown in FIG. 7 is an easy-to-see point cloud image of which points are shaded.
  • the shading processing is performed by superimposing the point cloud data on a photographic image and shading the points in areas corresponding to the areas in the photographic image. The photographic image is photographed by a camera at the same time when the point cloud data is obtained.
  • the cut-area indication marker is displayed on the image showing the entirety of the point cloud data. This facilitates visual understanding of the part to be cut.
  • the position and the size of the cut-area indication marker can be specified.
  • the cut-area indication marker as shown in FIG. 7 may be left-clicked and dragged by using a mouse to move its position.
  • the cut-area indication marker may be right-clicked and dragged by using a mouse to change its size.
  • the size can be changed along respective three axis directions that orthogonally cross each other.
  • FIG. 7 also shows an example of a displayed color image of the cut part.
  • the entirety of the image is monochromatic, but the cut part is colored, thereby facilitating understanding of the stereoscopic structure of the cut part.
  • FIG. 8 shows a screen showing only a part that is cut by the cut-area indication marker shown in FIG. 7 .
  • the cut-area indication marker may be rotated. Since three-dimensional coordinates of each point of the point cloud data are preliminarily known, rotating the cut-area indication marker changes points to be cut and also changes the image of the cut part (for example, the image shown in FIG. 8 ).
  • the cut-area indication marker may be displayed so as to align with the direction of a plane of the object.
  • An example of this case is shown in FIG. 11 .
  • the cut-area indication marker is displayed by aligning the normal line of the plane of the cut-area indication marker with the normal line of a plane of the object.
  • Point cloud data is mostly handled by targeting planes of an object. In such cases, the display of the cut-area indication marker aligning with a plane of an object is convenient.
  • the processing step for aligning the direction of the plane (direction of the normal line) of the cut-area indication marker with the direction of a plane (direction of the normal line) of an object is performed as described below.
  • the aligning processing is performed by the plane direction calculator 106 .
  • a part of an object, on which the cut-area indication marker aligning with the normal line of a plane of the object is to be displayed is specified.
  • a wall surface at the right side of the building is specified. This operation is performed by using a mouse, for example.
  • points of the specified part are obtained, and moreover, multiple points around the obtained points (specified points) are obtained. For example, points in a quadrangular area of 15 ⁇ 15 points centering at each specified point are obtained.
  • the area of the points to be obtained and the method of obtaining the points can be freely set. For example, instead of successive points, points at every five points or every ten points may be obtained.
  • a plane equation that fits to point cloud composed of the specified points and the surrounding multiple points is derived.
  • a normal line of a calculated plane is calculated and obtained as a normal line of the plane of the specified part in the object.
  • the direction of the cut-area indication marker is adjusted to align with the normal line of the plane of the specified part.
  • the direction of the cut-area indication marker is adjusted so that the normal line of the cut-area indication marker will align with the normal line of the specified part, which is calculated by the above method.
  • the adjustment of the direction is performed by the marker display controlling unit 103 . Consequently, a cut-area indication marker aligning with the direction of the plane of the object is displayed as shown in FIG. 11 , for example.
  • the cut-area indication marker is displayed to obliquely cut the building and extract a point cloud constituting the obliquely cut plane.
  • the displayed image that is formed only of this extracted point cloud is shown in FIG. 10 , but this image is difficult to visually understand clearly.
  • setting the cut-area indication marker to align with the normal line of a plane of an object enables display of a point cloud image in which the object is easy to visually understand.
  • This case is exemplified by the image on the right side in FIG. 11 and is also exemplified in FIG. 12 , which is a displayed image that shows only the extracted point cloud.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Architecture (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Processing Or Creating Images (AREA)
  • Optical Radar Systems And Details Thereof (AREA)

Abstract

A technique for facilitating understanding of a stereoscopic structure of an object in a 3-D image of point cloud data is provided. An image processing device includes a point cloud data display controlling unit, a marker display controlling unit, and a specified point cloud display controlling unit. The point cloud data display controlling unit displays a 3-D image of point cloud data on a screen. The marker display controlling unit displays a marker to specify a cut section of the 3-D image of the point cloud data. The specified point cloud display controlling unit displays a point cloud that constitutes the cut section among the point cloud data, as another image.

Description

CROSS-REFERENCE TO RELATED APPLICATION
This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2016-181696 filed Sep. 16, 2016, the entire contents of which are incorporated herein by reference.
BACKGROUND OF THE INVENTION Technical Field
The present invention relates to an image processing technique for point cloud data.
Background Art
Point cloud data that are obtained by using a laser scanner or by stereophotogrammetry are publicly known (for example, refer to WO 2011/070927 and Japanese Unexamined Patent Application Laid-Open No. 2012-230594).
Point cloud data of an object can generate a 3-D image of the object when three-dimensionally displayed on a screen of a PC or other device. The point cloud data is typically used for generating a three-dimensional model therefrom. The three-dimensional model is data showing an outline of an object and has a high affinity with three-dimensional data that can be processed by CAD software.
When point cloud data is displayed before it is processed, a 3-D image of an object is displayed by a set of dots. However, since the point cloud data is raw data that is not a three-dimensional model data, the outline of the object can be indistinct, and it would be difficult to instantly recognize which part is a periphery or an edge of the object (for example, a building). The 3-D image of the point cloud may be modified, for example, by changing the colors of the dots in accordance with difference in directions of planes, to enable easy understanding of its stereoscopic structure. However, this 3-D image of the point cloud can still cause the above problem. In the case of using raw data that is not processed, the above problem tends to occur.
A specific example will be described. FIG. 2A is a perspective view of an example of a 3-D image of point cloud data of a cubic object. As shown in FIG. 2A, when the image is displayed by simple dots, the perspective is difficult to recognize, and it is difficult to distinguish which part is a periphery of the cubic object.
FIG. 2B shows an example of an image of point cloud data that is superimposed with peripheries of a cubic object. The peripheries of the cubic object are extracted from a three-dimensional model data of the point cloud data. In this case, the perspective is easily recognized. The image shown in FIG. 2B is obtained after the point cloud data is processed into the three-dimensional model data. That is, the image shown in FIG. 2B cannot be displayed by using the point cloud data before the point cloud data is processed into the three-dimensional model data.
SUMMARY OF THE INVENTION
In view of these circumstances, an object of the present invention is to provide a technique for facilitating understanding of a stereoscopic structure of an object in a 3-D image of point cloud data.
A first aspect of the present invention provides an image processing device including a first point cloud data display controlling unit, a marker display controlling unit, and a second point cloud data display controlling unit. The first point cloud data display controlling unit displays a 3-D image of point cloud data on a screen. The marker display controlling unit displays a marker to specify a cut section of the 3-D image of the point cloud data. The second point cloud data display controlling unit displays a point cloud that constitutes the cut section among the point cloud data, as another image.
According to a second aspect of the present invention, in the first aspect of the present invention, the displayed marker may be movable in a direction perpendicular to the cut section to cut another section, and a point cloud constituting the another cut section may be extracted from among the point cloud data.
According to a third aspect of the present invention, in the first or the second aspect of the present invention, a point cloud in a predetermined area in the direction perpendicular to the cut section may be extracted as the point cloud constituting the cut section.
According to a fourth aspect of the present invention, in any one of the first to the third aspects of the present invention, an image of the point cloud data that constitutes the cut section may be colored differently from an image of the rest of the point cloud data.
According to a fifth aspect of the present invention, in any one of the first to the fourth aspects of the present invention, the marker may indicate a range of the cut section, and one or both of the position and the size of the marker may be changeable.
According to a sixth aspect of the present invention, in any one of the first to the fifth aspects of the present invention, the direction of the cut section may be set so as to align with the direction of a plane of the 3-D image of the point cloud data.
A seventh aspect of the present invention provides an image processing method including displaying a 3-D image of point cloud data on a screen, displaying a marker to specify a cut section of the 3-D image of the point cloud data, and displaying a point cloud that constitutes the cut section among the point cloud data, as another image.
An eighth aspect of the present invention provides a non-transitory computer recording medium storing computer executable instructions that, when executed by a computer processor, cause the computer processor to display a 3-D image of point cloud data on a screen, display a marker to specify a cut section of the 3-D image of the point cloud data, and display a point cloud that constitutes the cut section among the point cloud data, as another image.
The present invention provides a technique for facilitating understanding of a stereoscopic structure of an object in a 3-D image of point cloud data.
BRIEF DESCRIPTION OF DRAWINGS
FIG. 1 is a block diagram of an embodiment.
FIGS. 2A and 2B are conceptual diagrams each showing an example of point cloud data that is displayed on a screen.
FIG. 3 is a conceptual diagram showing an example of point cloud data that is displayed on a screen.
FIGS. 4A and 4B are conceptual diagrams each showing an example of point cloud data that is displayed on a screen.
FIG. 5 shows a 3-D image of point cloud data that is processed.
FIG. 6 shows the 3-D image (intensity image) of the point cloud data shown in FIG. 5 and a sectional image of a part of the 3-D image.
FIG. 7 shows a 3-D image of point cloud data.
FIG. 8 shows a 3-D image of point cloud data.
FIG. 9 shows a 3-D image of point cloud data.
FIG. 10 shows a 3-D image of point cloud data.
FIG. 11 shows a 3-D image of point cloud data.
FIG. 12 shows a 3-D image of point cloud data.
 PREFERRED EMBODIMENTS OF THE INVENTION
FIG. 1 shows an image processing device 100 that is configured to process point cloud data. Typically, the image processing device 100 is constructed not by dedicated hardware but by application software using a personal computer (PC). The application software may be installed in the PC and is operated to execute the function of the image processing device 100.
When a PC is used, each functional unit shown in FIG. 1 is constructed by software. Each of the functional units shown in FIG. 1 may be composed of a dedicated arithmetic circuit. A functioning unit constructed of software and a functioning unit composed of a dedicated arithmetic circuit may be used together. For example, each of the functional units shown in the drawing may be constructed of an electronic circuit such as a central processing unit (CPU), an application specific integrated circuit (ASIC), or a programmable logic device (PLD) such as a field programmable gate array (FPGA).
Whether each of the functional units is to be constructed of dedicated hardware or is to be constructed of software so that programs are executed by a CPU is selected in consideration of necessary operating speed, cost, amount of electricity consumed, and other factors. For example, if a specific functional unit is composed of an FPGA, the operating speed is superior, but the production cost is high. On the other hand, if a specific functional unit is configured so that programs are executed by a CPU, the production cost is reduced because hardware resources are conserved. However, when the functional unit is constructed using a CPU, its operating speed is inferior to that of dedicated hardware. Constructing the functional unit by dedicated hardware and constructing the functional unit by software differ from each other, as described above, but are equivalent to each other from the viewpoint of obtaining a specific function.
The image processing device 100 includes a point cloud data obtaining unit 101, a point cloud data display controlling unit 102, a marker display controlling unit 103, a specified point cloud extracting unit 104, and a specified point cloud display controlling unit 105. The point cloud data obtaining unit 101 obtains point cloud data that is measured by a three-dimensional laser scanner, therefrom, through an appropriate communication means. The point cloud data may also be obtained by a method of calculating three-dimensional coordinates of numerous feature points that are extracted from photographed images, based on the principle of stereophotogrammetry.
The point cloud data display controlling unit 102 displays the point cloud data on an appropriate image displaying unit, such as a screen of a PC. The point cloud data may be displayed by dots as they are or by changing the colors of the dots, or may be displayed after being subjected to various kinds of image processing, such as removal of noise components.
The marker display controlling unit 103 controls display of a marker to cut a 3-D image of the point cloud data at a freely selected flat plane. FIG. 3 shows point cloud data 150 of a cubic object, which represents a building, and shows a marker 200 as an example of the displayed marker.
The marker 200 is movable in an X direction or a specific horizontal direction by operation of an operator, which may be performed by using a mouse. The marker 200 is moved to specify a position (position on the X axis) in the 3-D image of the point cloud data to be cut along a Y-Z plane. Here, the Z direction is a vertical direction, and the Y direction is a horizontal direction orthogonally crossing the X direction.
The azimuths shown in FIG. 3 may be set such that the X direction is set in the north-south direction (or the east-west direction), the Y direction is set in the east-west direction (or the north-south direction), and the Z direction is set in the vertical direction. This setting facilitates understanding of a relationship between a cut section and the azimuths. In another case, when the point cloud data 150 represents a building, the directions of the X axis, Y axis, and Z axis may be set in accordance with extending directions of walls of the building. FIG. 3 shows an example of this case.
FIGS. 4A and 4B each show an example of a sub-image that is displayed separately from the image shown in FIG. 3 on the same screen at the same time. The sub-image may be displayed on a screen that is different from the screen showing a main image or may be displayed on the same screen by switching from the main image. When the marker 200 shown in FIG. 3 is moved on the X axis until the Y-Z plane containing the marker 200 overlaps the point cloud data 150, the overlapped part (point cloud data to be cut along the Y-Z plane) is displayed as shown in FIG. 4A or 4B. FIG. 4A shows point cloud data 151 of a plane shape that is obtained by cutting a periphery on the X axis of the point cloud data 150 of the cubic object shown in FIG. 3 along the Y-Z plane. In this case, a front end plane on the X axis of the point cloud data 150 is cut along the Y-Z plane, and the cut part of the plane is displayed.
When the marker 200 shown in FIG. 3 is moved further in the X axis positive direction from the condition shown in FIG. 4A, the position of a plane in the point cloud data 150 to be cut along the Y-Z plane is changed. This case is exemplified in FIG. 4B. In this case, since point clouds for an inside of the cubic object, which is the base for the point cloud data 150, are not obtained, a frame-shaped point cloud is obtained as a point cloud of the cut plane.
The processing step of cutting the 3-D image of the point cloud data 150 shown in FIG. 3 along the Y-Z plane and extracting the point cloud data 151 and 152 as shown in FIGS. 4A and 4B is performed as described below. First, three-dimensional coordinates of each point constituting the point cloud data 150 is preliminarily obtained. Coordinates of the Y-Z plane that is specified by using the marker 200 are determined from a plane equation in a coordinate system that describes the point cloud data 150. Thus, a point cloud crossing or in proximity to the Y-Z plane, which is specified by using the marker 200, is extracted from among the point cloud data 150. To be more precise, since the point clouds are discretely distributed, threshold values are set, and the points in proximity to the Y-Z plane are extracted in an area within the threshold values.
As shown in FIGS. 4A and 4B, the 3-D image of the point cloud data is cut by using the marker 200. The point cloud constituting the cut section is displayed as another image separately from the 3-D image. This facilitates understanding of the stereoscopic structure of the point cloud data. For example, while the marker 200 is moved in the X axis direction, peripheries of the point cloud data 150, which are respectively cut along the Y-Z plane, are successively displayed. Tracing the display history facilitates understanding of the stereoscopic shape (peripheries) of the object, which is the base for the point cloud data 150.
The moving direction of the marker 200 is set by an operator as desired. In another case, for example, the marker 200 may be moved automatically, and point clouds constituting cut sections, which are successively obtained, may be displayed by thumbnail images aligning on a time axis. Such a UI display also facilitates understanding of the stereoscopically distributed condition of the point cloud data.
In the above cases, the Y-Z plane to be used for the cutting has a thin thickness, but it may have a large thickness. In such case, a point cloud that is distributed also in the X axis direction is extracted.
The specified point cloud extracting unit 104 extracts the point cloud data of a cut section that is cut along a specified plane of the 3-D image of the point cloud data. For example, the specified point cloud extracting unit 104 cuts the 3-D image of the point cloud data 150, as shown in FIG. 3, and extracts the point cloud data 151 and 152, as shown in FIG. 4, from the point cloud data 150.
The specified point cloud display controlling unit 105 controls display of the point cloud data that is extracted by the specified point cloud extracting unit 104. For example, the specified point cloud display controlling unit 105 controls display of the image as shown in FIG. 4.
As described above, the image processing device 100 includes the point cloud data display controlling unit 102, the marker display controlling unit 103, and the specified point cloud display controlling unit 105. The point cloud data display controlling unit 102 displays a 3-D image of point cloud data. The marker display controlling unit 103 displays the marker to specify a cut section of the 3-D image of the point cloud data. The specified point cloud display controlling unit 105 displays a point cloud that constitutes the cut section among the point cloud data, as another image.
This structure is configured to extract a periphery of an object and displays the periphery, thereby facilitating understanding of the stereoscopically distributed condition of the point cloud data. For example, in a case of using point cloud data based on a building, a stereoscopic shape of the building is easily understood without generating a three-dimensional model.
FIG. 5 shows a displayed 3-D image of point cloud data of an object after the point cloud data is processed. Although FIG. 5 shows a gray scale image, this image can be an easily understandable point cloud image in which each point is colored, in actual use. The coloring processing is performed by superimposing the point cloud data on a photographic image and coloring the points in areas corresponding to the areas in the photographic image. The photographic image is photographed by a camera at the same time when the point cloud data is obtained.
FIG. 6 shows a point cloud image that is displayed on the left side. This point cloud image is obtained by shading the point clouds of the point cloud data in FIG. 5, in accordance with intensities of reflection light. FIG. 6 shows an image of a point cloud, as seen from a direction perpendicular to a cut section, on the right side. This point cloud constitutes the cut section that is cut along a strip-shaped line shown in the left image. As shown in FIG. 6, a part of the 3-D image of the point cloud data is cut (refer to the left image), and the point cloud constituting the cut section is displayed as seen from the direction perpendicular to the cut section (refer to the right image). This facilitates recognizing of the outline of the object. The line for defining the cut section as shown in the point cloud image on the left side in FIG. 6 can have any width. The line may have a wide width to extract points in an area having a wide width in a direction perpendicular to the cut section.
A marker having a function different from the function of the marker 200 may also be displayed. This marker is a cut-area indication. The cut-area indication marker indicates an area of a part of the 3-D image of the point cloud data to be cut. The display of the cut-area indication marker is also controlled by the marker display controlling unit 103. FIG. 7 shows an example of displaying a cut-area indication marker on a displayed point cloud image. This marker has a small thickness and has a flat rectangular shape. Either one of the marker 200 or the cut-area indication marker may be displayed, or they may be displayed at the same time. The 3-D image shown in FIG. 7 is an easy-to-see point cloud image of which points are shaded. The shading processing is performed by superimposing the point cloud data on a photographic image and shading the points in areas corresponding to the areas in the photographic image. The photographic image is photographed by a camera at the same time when the point cloud data is obtained.
As shown in FIG. 7, the cut-area indication marker is displayed on the image showing the entirety of the point cloud data. This facilitates visual understanding of the part to be cut. The position and the size of the cut-area indication marker can be specified. For example, the cut-area indication marker as shown in FIG. 7 may be left-clicked and dragged by using a mouse to move its position. Additionally, the cut-area indication marker may be right-clicked and dragged by using a mouse to change its size. The size can be changed along respective three axis directions that orthogonally cross each other. These operations are also controlled by the marker display controlling unit 103.
FIG. 7 also shows an example of a displayed color image of the cut part. The entirety of the image is monochromatic, but the cut part is colored, thereby facilitating understanding of the stereoscopic structure of the cut part. FIG. 8 shows a screen showing only a part that is cut by the cut-area indication marker shown in FIG. 7.
The cut-area indication marker may be rotated. Since three-dimensional coordinates of each point of the point cloud data are preliminarily known, rotating the cut-area indication marker changes points to be cut and also changes the image of the cut part (for example, the image shown in FIG. 8).
The cut-area indication marker may be displayed so as to align with the direction of a plane of the object. An example of this case is shown in FIG. 11. In the case shown in FIG. 11, the cut-area indication marker is displayed by aligning the normal line of the plane of the cut-area indication marker with the normal line of a plane of the object. Point cloud data is mostly handled by targeting planes of an object. In such cases, the display of the cut-area indication marker aligning with a plane of an object is convenient.
The processing step for aligning the direction of the plane (direction of the normal line) of the cut-area indication marker with the direction of a plane (direction of the normal line) of an object is performed as described below. The aligning processing is performed by the plane direction calculator 106. In the aligning processing, first, a part of an object, on which the cut-area indication marker aligning with the normal line of a plane of the object is to be displayed, is specified. In the case shown in FIG. 9, for example, a wall surface at the right side of the building is specified. This operation is performed by using a mouse, for example.
Next, points of the specified part are obtained, and moreover, multiple points around the obtained points (specified points) are obtained. For example, points in a quadrangular area of 15×15 points centering at each specified point are obtained. The area of the points to be obtained and the method of obtaining the points can be freely set. For example, instead of successive points, points at every five points or every ten points may be obtained.
Then, a plane equation that fits to point cloud composed of the specified points and the surrounding multiple points is derived. Then, a normal line of a calculated plane is calculated and obtained as a normal line of the plane of the specified part in the object. The direction of the cut-area indication marker is adjusted to align with the normal line of the plane of the specified part. Specifically, the direction of the cut-area indication marker is adjusted so that the normal line of the cut-area indication marker will align with the normal line of the specified part, which is calculated by the above method. The adjustment of the direction is performed by the marker display controlling unit 103. Consequently, a cut-area indication marker aligning with the direction of the plane of the object is displayed as shown in FIG. 11, for example.
For example, in the case shown in FIG. 9, the cut-area indication marker is displayed to obliquely cut the building and extract a point cloud constituting the obliquely cut plane. The displayed image that is formed only of this extracted point cloud is shown in FIG. 10, but this image is difficult to visually understand clearly. In contrast, setting the cut-area indication marker to align with the normal line of a plane of an object enables display of a point cloud image in which the object is easy to visually understand. This case is exemplified by the image on the right side in FIG. 11 and is also exemplified in FIG. 12, which is a displayed image that shows only the extracted point cloud.

Claims (4)

What is claimed is:
1. An image processing device comprising a processor, the processor configured to:
display points of point cloud data on a screen, the point cloud data having known three-dimensional coordinates in an XYZ coordinate system, and the displayed points forming a 3-D image;
display a marker on the screen to specify a cut section of the 3-D image in response to designation of a position of the cut section by an operator;
extract points of the point cloud data within a threshold range to the cut section in the XYZ coordinate system in response to specification of the cut section in the XYZ coordinate system by the displayed marker;
display the extracted points on the screen;
derive a plane equation relating to a plane constituting an object from which the point cloud data is obtained and calculate a normal line of the plane on a basis of the plane equation; and
display a cut-area indication marker on the screen, the cut-area indication marker showing a flat part to be cut out from the point cloud data of the 3-D image,
wherein a direction of a normal line of the flat part coincides with a direction of the normal line of the plane.
2. An image processing method comprising:
displaying points of point cloud data on a screen, the point cloud data having known three-dimensional coordinates in an XYZ coordinate system, and the displayed points forming a 3-D image;
displaying a marker on the screen to specify a cut section of the 3-D image in response to designation of a position of the cut section by an operator;
extracting points of the point cloud data within a threshold range to the cut section in the XYZ coordinate system in response to specification of the cut section in the XYZ coordinate system by the displayed marker;
displaying the extracted points on the screen;
deriving a plane equation relating to a plane constituting an object from which the point cloud data is obtained and calculating a normal line of the plane on a basis of the plane equation; and
displaying a cut-area indication marker on the screen, the cut-area indication marker showing a flat part to be cut out from the point cloud data of the 3-D image,
wherein a direction of a normal line of the flat part coincides with a direction of the calculated normal line of the plane.
3. A non-transitory computer recording medium storing computer executable instructions that, when executed by a computer processor, cause the computer processor to:
display points of point cloud data on a screen, the point cloud data having known three-dimensional coordinates in an XYZ coordinate system, and the displayed points forming a 3-D image;
display a marker on the screen to specify a cut section of the 3-D image in response to designation of a position of the cut section by an operator;
extract points of the point cloud data within a threshold range to the cut section in the XYZ coordinate system in response to specification of the cut section in the XYZ coordinate system by the displayed marker;
display the extracted points on the screen;
derive a plane equation relating to a plane constituting an object from which the point cloud data is obtained and calculate a normal line of the plane on a basis of the plane equation; and
display a cut-area indication marker on the screen, the cut-area indication marker showing a flat part to be cut out from the point cloud data of the 3-D image,
wherein a direction of a normal line of the flat part coincides with a direction of the calculated normal line of the plane.
4. An image processing device comprising a processor, the processor configured to:
display points of point cloud data on a screen, the point cloud data having known three-dimensional coordinates in an XYZ coordinate system, and the displayed points forming a 3-D image;
derive a plane equation relating to a plane constituting an object from which the point cloud data is obtained and calculate a normal line of the plane on a basis of the plane equation; and
display a cut-area indication marker on the screen, the cut-area indication marker showing a flat part to be cut out from the point cloud data of the 3-D image,
wherein a direction of a normal line of the flat part coincides with a direction of the normal line of the plane.
US15/704,215 2016-09-16 2017-09-14 Image processing device, image processing method, and image processing program Active US10354447B2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2016181696A JP2018045587A (en) 2016-09-16 2016-09-16 Image processing device, image processing method, and image processing program
JP2016-181696 2016-09-16

Publications (2)

Publication Number Publication Date
US20180082472A1 US20180082472A1 (en) 2018-03-22
US10354447B2 true US10354447B2 (en) 2019-07-16

Family

ID=61621188

Family Applications (1)

Application Number Title Priority Date Filing Date
US15/704,215 Active US10354447B2 (en) 2016-09-16 2017-09-14 Image processing device, image processing method, and image processing program

Country Status (2)

Country Link
US (1) US10354447B2 (en)
JP (1) JP2018045587A (en)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6910820B2 (en) * 2017-03-02 2021-07-28 株式会社トプコン Point cloud data processing device, point cloud data processing method, point cloud data processing program
JP7439410B2 (en) * 2019-08-07 2024-02-28 オムロン株式会社 Image processing device, image processing method and program
CN111006645A (en) * 2019-12-23 2020-04-14 青岛黄海学院 Unmanned aerial vehicle surveying and mapping method based on motion and structure reconstruction
JP6708917B1 (en) * 2020-02-05 2020-06-10 リンクウィズ株式会社 Shape detection method, shape detection system, program
JP7468002B2 (en) * 2020-03-10 2024-04-16 日本電気株式会社 Anomaly display device, anomaly display system, anomaly display method, and anomaly display program
JP2022015633A (en) 2020-07-09 2022-01-21 株式会社トプコン Surveying device

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060116838A1 (en) * 2004-11-26 2006-06-01 Hon Hai Precision Industry Co., Ltd. Method and system for cutting point cloud automatically
WO2011070927A1 (en) 2009-12-11 2011-06-16 株式会社トプコン Point group data processing device, point group data processing method, and point group data processing program
JP2012230594A (en) 2011-04-27 2012-11-22 Topcon Corp Three-dimensional point group position data processor, three-dimensional point group position data processing method, three-dimensional point group position data processing system, and program
US20160232259A1 (en) * 2015-02-06 2016-08-11 Clearedge3D, Inc. Apparatus and method for interactively extracting shapes from a point cloud

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3937913B2 (en) * 2002-05-10 2007-06-27 キヤノン株式会社 Information processing device
DE10233668A1 (en) * 2002-07-24 2004-02-19 Siemens Ag Data record processing method for a data record for capacity describing a tubular vessel in the body and its surroundings defines a working point for a cutting area
JP2013092888A (en) * 2011-10-25 2013-05-16 Toshiba Corp Data processor
JP5948479B1 (en) * 2015-09-11 2016-07-06 大豊精機株式会社 System, method and computer software program for displaying and operating two-dimensional or three-dimensional object data

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060116838A1 (en) * 2004-11-26 2006-06-01 Hon Hai Precision Industry Co., Ltd. Method and system for cutting point cloud automatically
WO2011070927A1 (en) 2009-12-11 2011-06-16 株式会社トプコン Point group data processing device, point group data processing method, and point group data processing program
US20120256916A1 (en) 2009-12-11 2012-10-11 Kazuo Kitamura Point cloud data processing device, point cloud data processing method, and point cloud data processing program
JP2012230594A (en) 2011-04-27 2012-11-22 Topcon Corp Three-dimensional point group position data processor, three-dimensional point group position data processing method, three-dimensional point group position data processing system, and program
US20160232259A1 (en) * 2015-02-06 2016-08-11 Clearedge3D, Inc. Apparatus and method for interactively extracting shapes from a point cloud

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
"James Thomas", "Clipping and Slicing a Point Cloud", Apr. 18, 2016, MicroSurvey (Year: 2016). *
Yue Li and Matthew Hielsberg, "A Tutorial for 3D Point Cloud Editor". Apr. 9, 2010, Texas A&M University (Year: 2010). *

Also Published As

Publication number Publication date
JP2018045587A (en) 2018-03-22
US20180082472A1 (en) 2018-03-22

Similar Documents

Publication Publication Date Title
US10354447B2 (en) Image processing device, image processing method, and image processing program
JP5799273B2 (en) Dimension measuring device, dimension measuring method, dimension measuring system, program
CN111176224A (en) Industrial safety monitoring arrangement using digital twinning
KR101973917B1 (en) Three-dimensional measuring device and its supporting method of measurement
US10382747B2 (en) Image processing apparatus, image processing method, and image processing program
JP2011215956A (en) Method and program for displaying cursor
JP6910820B2 (en) Point cloud data processing device, point cloud data processing method, point cloud data processing program
JP2011185650A (en) Model generation apparatus and model generation program
JP6468757B2 (en) 3D model generation method, 3D model generation system, and 3D model generation program
JP6380685B2 (en) Dimension measuring device
CN104915994A (en) 3D view drawing method and system of three-dimensional data
US20180068492A1 (en) Image processing device, image processing method, and image processing program
JP6132246B2 (en) Dimension measurement method
JP7112908B2 (en) Survey data processing device and cross-sectional drawing creation method for creating cross-sectional view of 3D point cloud data
JP7112469B2 (en) Image processing device, image processing method, image processing program
RU2016146417A (en) PIXEL SHADING REDUCTION TECHNOLOGIES
Aydar et al. A low-cost laser scanning system design
EP3121792B1 (en) Processing device for label information for multi-viewpoint images and processing method for label information
JP6659641B2 (en) 3D model creation device
RU2522809C1 (en) Method for photogrammetric measurement of dimensions and monitoring shape of body bounded by set of interconnected surfaces
CN113227710A (en) Flight control method, device, system and computer readable medium for unmanned aerial vehicle
JP2020013390A (en) Information processing apparatus, information processing program, and information processing method
US11941852B2 (en) Three-dimensional measurement device, three-dimensional measurement method, and three-dimensional measurement program
JP2004252815A (en) Image display device, its method and program
CN116128888A (en) Three-dimensional ultrasonic image cutting method and device, ultrasonic equipment and storage medium

Legal Events

Date Code Title Description
AS Assignment

Owner name: TOPCON CORPORATION, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MORIKAWA, NAOKI;ITO, TADAYUKI;REEL/FRAME:043590/0796

Effective date: 20170823

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4