JP2017036998A - Color information determination device and image generation device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a color information determination device capable of providing an easily viewable image, and an image generation device.SOLUTION: A color information determination device is constituted of a first acquisition section and a determination section. The first acquisition section acquires point group data being measured by measuring instruments including a distance meter, including a plurality of point data representing individual positions of a plurality of points included in an object including a first portion and representing a three-dimensional shape of the object and information related to a positional relation of the distance meter to the object when the point group data are measured. The determination section determines color information for each of the plurality of point data based on arrangement information based on at least any of a position, a shape and a size of the first portion and information related to the positional relation of the distance meter.SELECTED DRAWING: Figure 1

Description

  Embodiments described herein relate generally to a color information determination device and an image generation device.

  An image can be generated using data obtained by measuring the three-dimensional shape of the object. Using such an image, for example, the shape of an arbitrary part of the object can be confirmed. In such an image, it is desired to provide an easy-to-see image.

JP 2011-223582 A

  Embodiments of the present invention provide a color information determination device and an image generation device that can provide an easy-to-see image.

  According to the embodiment of the present invention, a color information determination device including a first acquisition unit and a determination unit is provided. The first acquisition unit is measured by a measuring instrument including a distance meter, and includes a plurality of point data representing respective positions of a plurality of points included in the object including the first portion, and the three-dimensional shape of the object And information regarding the positional relationship of the distance meter with respect to the object when the point cloud data is measured. The determination unit is configured to color each of the plurality of point data based on arrangement information based on at least one of a position, a shape, and a size of the first portion and the information on the positional relationship of the distance meter. Determine information.

1 is a block diagram illustrating a color information determination device according to a first embodiment. It is a schematic diagram which illustrates the color information determination apparatus which concerns on 1st Embodiment. It is a flowchart which illustrates the process of the color information determination apparatus which concerns on 1st Embodiment. It is a schematic diagram which illustrates point cloud data. FIG. 5A to FIG. 5C are schematic views illustrating point group data to which color information is assigned. 1 is a block diagram illustrating an image generation apparatus according to a first embodiment. 3 is a flowchart illustrating processing of the image generation apparatus according to the first embodiment. FIG. 8A to FIG. 8C are schematic views illustrating processing in the display image generation unit. It is a block diagram which illustrates the color information deciding device concerning a 2nd embodiment. It is a flowchart which illustrates the process of the color information determination apparatus which concerns on 2nd Embodiment. It is a flowchart which illustrates calculation of a measurement object surface group. It is a schematic diagram which illustrates calculation of a measurement object surface group. It is a block diagram which illustrates the image generation device concerning a 2nd embodiment. 6 is a flowchart illustrating processing of an image generation apparatus according to a second embodiment.

Each embodiment will be described below with reference to the drawings.
The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between the parts, and the like are not necessarily the same as actual ones. Further, even when the same part is represented, the dimensions and ratios may be represented differently depending on the drawings.
Note that, in the present specification and each drawing, the same elements as those described above with reference to the previous drawings are denoted by the same reference numerals, and detailed description thereof is omitted as appropriate.

(First embodiment)
FIG. 1 is a block diagram illustrating a color information determination device according to the first embodiment.
As illustrated in FIG. 1, the color information determination apparatus 101 according to the present embodiment includes a first acquisition unit 1 and a determination unit 2. The determination unit 2 includes a measurement direction determination unit 21 and a color information determination unit 22. The color information determination device 101 according to the present embodiment is a device that determines color information for point cloud data representing a three-dimensional shape of an object. For example, the color information determination apparatus 101 assigns color information to point cloud data.

The point cloud data is measured using a measuring instrument having a range sensor (distance meter) such as a laser scanner or a laser range finder, for example.
The first acquisition unit 1 acquires the measured point cloud data, the position of a distance meter (measurement device) at the time of measurement, and the attitude of the distance meter (measurement device) at the time of measurement.
The determination unit 2 assigns color information to the point cloud data in accordance with the relationship between a measurement object described later and the acquired attitude of the distance meter.

  For example, shape data representing the three-dimensional shape of the object can be displayed on a display or the like based on point cloud data to which color information is assigned. Thereby, it is possible to obtain a color image that is easy to see. For example, the user selects two measurement points on the shape represented by the shape data on such an image. Based on the shape data, the distance between the two selected points can be calculated. Thereby, the user can obtain, for example, the size and length of an arbitrary portion of the object whose point cloud data is measured.

  An arithmetic device including a CPU (Central Processing Unit), a memory, and the like can be used for each block included in the devices (color information determination device and image generation device described later) according to the embodiment. An integrated circuit such as an LSI (Large Scale Integration) or an IC (Integrated Circuit) chip set can be used for part or all of the apparatus according to the embodiment. An individual circuit may be used for each block shown, or a circuit in which a part or all of the blocks are integrated may be used. Each block may be provided integrally, or a part of the blocks may be provided separately. In addition, a part of each block may be provided separately. The integration is not limited to an LSI, and a dedicated circuit or a general-purpose processor may be used. In addition, an input / output interface or an input / output terminal that communicates with the outside through a wired or wireless connection can be used as the acquisition unit.

  Each block shown in the figure is configured to be able to communicate with each other directly or indirectly via a communication network. For example, each block can transmit and receive shape data and the like. The type of communication network is arbitrary. For example, the blocks may be configured to communicate with each other via a LAN (Local Area Network) installed in a building. In addition, the blocks may be in a form that can communicate with each other via a network (cloud) such as the Internet. The block diagram illustrated in the description of the embodiment is an example of a main configuration of the apparatus (color information determination apparatus and image generation apparatus) according to the embodiment, and may not necessarily match the actual configuration of the program module. is there. Each block is a separate device and may be installed separately.

FIG. 2 is a schematic view illustrating the usage status of the color information determination device according to the first embodiment.
The target object 60 whose three-dimensional shape is measured includes a first portion P1.
As illustrated in FIG. 2, the object 60 extends in the Z-axis direction (first extending direction). For example, the shape of the object 60 includes a cylindrical portion that extends in the Z-axis direction. The first portion P1 is a member or the like that is arranged on the basis of, for example, a direction perpendicular to the Z-axis direction or a direction parallel to the Z-axis direction. A direction perpendicular to the Z-axis direction is defined as an X-axis direction, and a direction perpendicular to the Z-axis direction and the X-axis direction is defined as a Y-axis direction.

  In the three-dimensional space, the X-axis direction has a degree of freedom with respect to the rotation direction with the Z axis as the rotation axis, and may be arbitrarily determined. For example, it is desirable that the direction in which the measurement object described later appears repeatedly is the X-axis direction. Alternatively, when a specific measurement target is determined, it is desirable to set the X-axis direction with reference to a portion (for example, a portion assumed to be present) that the target has. For example, in the case of an elevator hoistway, the direction in which the sill present at the foot of the elevator door extends may be the X-axis direction.

  In the example shown in FIG. 2, the object 60 is an elevator hoistway (inside the elevator hoistway), and the first portion P1 is a bracket 61 installed in the elevator hoistway. In this example, the elevator hoistway further includes a rail 62, a wall 63, and a wall 64.

  The wall 63 and the wall 64 are inner wall surfaces of the elevator hoistway facing each other, and extend in the Z-axis direction. The rail 62 is a member that is attached to control the traveling direction of the elevator car, and extends in the Z-axis direction. The bracket 61 is a member attached to the wall surface to support the rail. For example, the plurality of brackets 61 are repeatedly arranged along the Z-axis direction in the elevator hoistway.

  For example, the user measures the distance between the brackets 61 using the measured three-dimensional point cloud data. Like the bracket 61 in this case, a portion related to a measurement target such as a dimension or a length is referred to as a “measurement target”. In other words, the first part P1 is a measurement object.

  The user may measure the distance between the wall 63 and the wall 64 or the distance between the wall 63 and the rail 62 using the three-dimensional shape data. In this case, the measurement object is one of the wall 63, the wall 64, and the rail 62. However, the measurement target is not limited to the above example, and for example, any part of the target 60 can be selected as the measurement target (first part).

  In the following description of the embodiments, for convenience of explanation, a case will be described in which the object 60 is an elevator hoistway (hereinafter, hoistway), a member provided therein, and the like.

  For example, a measuring instrument 75 having a distance meter 71 is installed in the hoistway. For example, the measuring instrument 75 is installed on an elevator car. By being installed on a movable elevator car, the measuring instrument 75 can measure the inside of the hoistway over a wide range. For example, the arrangement of the distance meter 71 in the entire measuring instrument 75 is defined in advance.

  The distance meter 71 is a three-dimensional distance measuring device (range sensor), and specifically, a laser range finder or the like is used. For example, the distance meter 71 irradiates the hoistway with laser light and measures the reflected light of the irradiated laser light. Thereby, the distance meter 71 measures the distance between the distance meter 71 and the object 60. For example, the distance meter 71 irradiates a plurality of points (regions) on the object 60 with laser light. Thereby, the distance meter 71 measures the distance between each of the plurality of points on the object 60 and the distance meter 71.

  A stereo camera may be used for the distance meter 71. When a stereo camera is used, the depth of the object is estimated at each pixel on the image based on the image. Thereby, a three-dimensional point group can be obtained.

  In this way, point cloud data representing the three-dimensional shape of the object 60 can be obtained by the distance meter 71. In addition, the acquisition method of the point cloud data in the embodiment is not limited to the method using the distance meter 71 described above.

The point cloud data includes a plurality of point data. Each of the plurality of point data is position information representing the position of each of the plurality of points on the object 60. The point cloud data includes point data related to points on the measurement object.
The color information determination apparatus 101 determines color information for each of the plurality of point data as described above. As a result, a color is assigned to each of the plurality of point data.

Next, each block of the color information determination apparatus 101 will be described in order.
FIG. 3 is a flowchart illustrating the process of the color information determination device according to the first embodiment.
The first acquisition unit 1 acquires point cloud data representing the three-dimensional shape of the hoistway from the distance meter 71 (measurement device 75) (step S101).

  Furthermore, the 1st acquisition part 1 acquires the information of the positional relationship with respect to the target object 60 of the distance meter 71 (measurement device 75) when point cloud data are measured. This positional relationship information includes information on the position of the distance meter 71 and information on the attitude of the distance meter 71. For example, the first acquisition unit 1 acquires the position data of the distance meter 71 at the start of measurement and the posture data of the distance meter 71 at the start of measurement.

  The position information is represented as, for example, three-dimensional coordinates with reference to the origin of the coordinate system in which the point cloud data acquired by the first acquisition unit 1 is defined. For example, the position of the distance meter 71 at the start of measurement may be used as the origin.

  Here, the attitude data of the distance meter 71 is information on the direction of the distance meter 71. The posture data is represented by three vectors (hereinafter, three posture vectors) in the coordinate system in which the point cloud data acquired by the first acquisition unit 1 is defined. These three vectors correspond to the physical orientation of the distance meter 71.

  Hereinafter, for convenience of explanation, the directions of the three orientation vectors are defined as an X ′ direction, a Y ′ direction, and a Z ′ direction. That is, the three posture vectors are represented as X ′, Y ′, and Z ′. Here, the X ′ direction is a direction along a normal vector of a plane including a vector indicating the upper side of the distance meter 71 and a vector indicating the front direction of the distance meter 71. The Y ′ direction is a direction along a vector indicating the upper side of the distance meter 71. The Z ′ direction is a direction along a vector indicating the forward direction of the distance meter 71. The forward direction of the distance meter 71 refers to, for example, a direction from the distance meter 71 toward the measurement region (measurement point) where the distance meter 71 measures the distance.

  Hereinafter, for the sake of simplicity, it is assumed that the Z ′ direction faces the ceiling direction of the hoistway and the X ′ direction faces the X-axis direction. That is, the case where the direction of the distance meter 71 with respect to the hoistway is determined will be described. Here, the direction of the distance meter 71 with respect to the hoistway is the relationship between the Z ′ direction (vector indicating the direction of the distance meter 71) and the aforementioned Z-axis direction (vector indicating the shape of the hoistway), and X It depends on the relationship between the 'direction and the X-axis direction. That is, in the following description, it is assumed that the Z ′ direction and the Z axis direction are parallel to each other, and the X ′ direction and the X axis direction are parallel to each other. Similarly, it is assumed that the position data is defined by a coordinate system in which the point cloud data acquired by the first acquisition unit 1 is defined. Thereby, the position with respect to the hoistway of the distance meter 71 and the direction (posture) of the distance meter 71 with respect to the hoistway are acquired.

  The first acquisition unit 1 may acquire data from an external storage that stores at least one of the above point cloud data, posture data, and position data. The external storage device is not limited to a storage medium such as a hard disk or a CD, but includes a server connected via a communication network.

  The acquisition of the posture data and the position data is not limited to the acquisition from the external storage and the acquisition from the measuring instrument 75. For example, the principal component analysis may be performed on the position coordinates of the point cloud data, and the principal axes corresponding to the eigenvalues may be set as the posture vectors X ′, Y ′, and Z ′ in descending order of the eigenvalues. Similarly, it is good also considering what averaged the position coordinate of point cloud data as a position of a measuring device.

  For example, the measurement object related to the measurement target is specified from outside the color information determination apparatus 101. For example, the user selects one of the hoistway wall surfaces (wall 63, wall 64, etc.), bracket 61, rail 62, and the like. Thereby, a measurement object is designated.

  The measurement direction determination unit 21 first acquires measurement target information corresponding to the designated measurement target (step S102). Thereafter, the measurement direction determination unit 21 determines a reference direction (reference direction vector) based on the measurement target information (step S103). The reference direction vector is a vector serving as a reference when the color information determination unit 22 determines the color information of the point cloud data.

  First, the measurement target information acquired in step S102 is arrangement information based on at least one of the position of the measurement target in the hoistway, the shape of the measurement target, and the size of the measurement target.

Specifically, for example, when the bracket 61 is designated as a measurement object, a plurality of measurement objects (brackets 61) are arranged in the hoistway. For example, the measurement objects are arranged at predetermined intervals in a specific direction. At this time, the measurement target information is a vector indicating the direction in which the measurement target is repeatedly arranged. For example, when installing the distance meter 71 at the measurement site, the user can check the direction in which the measurement objects are repeatedly arranged as viewed from the distance meter 71. The direction is recorded as measurement target information. Thereafter, when dimension measurement is performed, a reference direction vector is obtained based on the measurement target information and the posture vector.
When measuring the distance between the wall 63 and the rail 62, for example, the wall 63 is designated as the measurement object (first portion). At this time, the measurement target information is a vector indicating a direction from the wall 63 (first portion) to the rail 62 (second portion). For example, the measurement target information is a vector indicating a second direction Dx perpendicular to the first direction Dz in which the wall 63 and the rail 62 extend. The second direction Dx indicates a direction from the wall 63 (first portion) toward the rail 62 (second portion).

  For example, as described with reference to FIG. 2, the arrangement of the measurement object in the hoistway is defined according to the Z-axis direction. The direction of the distance meter 71 (Z ′ direction) is, for example, parallel to the Z-axis direction. Thereby, the direction etc. in which each measurement object is located when viewed from the distance meter 71 can be obtained. That is, measurement target information can be obtained.

The measurement target information is defined on a coordinate system determined by attitude data corresponding to the arrangement of the distance meter 71.
Although it may be difficult to give the measurement target information accurately, the measurement target information only needs to be roughly defined in consideration of actual use.

  For example, when measurement objects are arranged along the direction toward the front of the distance meter 71, a vector in the Z ′ direction may be used as the measurement object information. When the measurement objects are arranged along a direction near the middle between the direction toward the front of the distance meter 71 and the direction toward the upper side of the distance meter 71, the vector of the Y ′ direction and the vector of the Z ′ direction are The sum may be used as measurement target information.

  In the hoistway, the measurement object and the measurement object information are known in advance. Further, a distance meter 71 can be installed in the hoistway so that the posture data is reproduced to some extent. In such a case, the measurement target information can be reused. In order to reproduce the attitude data, for example, the positional relationship between a specific portion of the distance meter 71 and the door direction of the elevator is determined in advance. By determining the installation method of the distance meter 71 with respect to the hoistway in advance, the posture data can be reproduced.

An example of the procedure for calculating the reference direction vector “v_base” is shown below.
(Step S102)
A measurement object is specified by a user or the like. For example, the measurement target is specified by selecting the measurement target using a GUI (Graphical User Interface) having a selection function such as a pull-down menu. When the measurement object is designated, the measurement direction determination unit 21 selects measurement object information corresponding to the designated measurement object.

(Step S103)
A direction vector v given as measurement target information is converted from a coordinate system of posture data to a coordinate system of point cloud data.

For this, first, a rotation matrix R is obtained. The rotation matrix R is obtained, for example, as a rotation matrix from the posture represented by X, Y, Z to the posture represented by X ′, Y ′, Z ′. By using the following expression using this, it is possible to convert the coordinate system based on the posture vector acquired by the first acquisition unit 1 to the coordinate system of the point cloud data.
v_base = Rv
As described above, the reference direction vector v_base is a vector obtained by performing coordinate conversion on the measurement target information (direction vector v).

  As described above, the measurement object information and the reference based on the arrangement information of the measurement object in the hoistway and the position and orientation of the distance meter 71 with respect to the hoistway (positional relationship of the distance meter 71 with respect to the object 60). A direction vector is given.

  Thereafter, the color information determination unit 22 obtains a distance from the plane defined by the reference direction vector and the position data to the position of each point represented by the point cloud data (step S104). Thereafter, the color information determination unit 22 assigns color information to the point cloud data according to the value obtained in step S104 (step S105). Processing for each of the plurality of points represented by the point cloud data is the same as each other. Therefore, in the following, processing for a certain point (first point) will be described.

(Step S104)
A distance base_dist to the first point is calculated using a plane equation defined by the reference direction vector and the position data. The plane is, for example, a plane perpendicular to the reference direction vector.

(Step S105)
A predetermined reference distance color_base (first distance) is given as a reference distance for coloring. The value of the reference distance color_base is arbitrary. Using this reference distance color_base, the distance value is normalized to a value up to 1.0 by the following equation. Based on the normalized value, a color to be assigned to the first point is determined.
color_base '= mod (color_base, base_dist) / base_dist
Here, mod (a, b) is a function for obtaining a remainder when a is divided by b.
A color reference is set between 0.0 and 1.0. For example, a plurality of reference values and a color corresponding to each reference value are set as the color reference. The color of the first point is determined with reference to the set color standard.

  Specifically, if the calculated value (color_base ') is the same as any of the reference values, the color corresponding to the reference value is adopted as the first point color. When the calculated value is different from any of the reference values, a reference value closest to the calculated value among reference values larger than the calculated value and a reference value closest to the calculated value among values smaller than the calculated value are The color is obtained by linear interpolation using.

  For example, the reference value of the color reference is set to 0.0, 0.33, and 0.66. The color corresponding to 0.0 is (R (red), G (green), B (blue)) = (255, 0, 0), and the color corresponding to 0.33 is (R, G, B) = (0, 255, 0), and the color corresponding to 0.66 is (R, G, B) = (0, 0, 225). The color of the value between the reference values is obtained by linear interpolation based on the reference value color larger than the value and the reference value color smaller than the value.

For example, when color_base ′ = 0.2, (R, G, B) = (255, 0) depending on the distance from 0.2 to 0.0 and the distance from 0.2 to 0.33. , 0) and (R, G, B) = (0,255,0), the color is determined.
FIG. 4 is a schematic diagram illustrating point cloud data.
The plurality of points represented by the point cloud data includes, for example, first to fourth points D1 to D4.
The first to fourth points D1 to D4 are arranged linearly. For example, the first to fourth points D1 to D4 are arranged in the direction of the normal line of the plane defined in step S104.

The third point D3 is located between the first point D1 and the second point D2. The fourth point D4 is located between the first point D1 and the third point D3.
A distance La between the first point D1 and the third point D3 is a reference distance color_base. A distance Lb between the second point D2 and the third point D3 is a reference distance color_base. That is, the third point D3 is located at the midpoint of the line segment connecting the first point D1 and the second point D2.
The distance Lc between the first point D1 and the fourth point D4 is shorter than the reference distance color_base.

  The point cloud data includes first to fourth point data. The first to fourth point data correspond to the first to fourth points D1 to D4, respectively.

The color determined for the first point D1 (first point data) is the same as the color determined for the second point D2 (second point data).
The color determined for the third point D3 (third point data) is the same as the color determined for the second point D2 (second point data).
The color determined for the fourth point D4 (fourth point data) is different from the color determined for the first point D1 (first point data).
As described above, the assigned color information has periodicity. That is, the color assigned by the determination unit returns to the same color at a predetermined distance.

  As described above, a color is assigned to each of a plurality of points (a plurality of point data). Thereby, a color is allocated to each point on the object 60 corresponding to the color information. In addition, it is not necessary to provide the color reference at equal intervals, and the pitch of the color reference may be arbitrarily adjusted. The color set as the color reference is also arbitrary.

  In addition, although the case where measurement object information was one vector was demonstrated, it does not restrict to this. Two or more vectors may be evaluated simultaneously. For example, when four walls are used as the measurement object, two vectors that intersect with the walls facing each other are obtained. In this case, base_dist may be calculated for each piece of measurement target information, and the color having the smallest value or the largest value may be assigned as the final base_dist.

In the above description, the processing is performed by converting the measurement target information to the coordinate system of the point cloud data. However, the present invention is not limited to this. The coordinates p of each point of the point cloud data may be converted into the coordinate system of the attitude data. In this case, the coordinate p ′ after conversion is the position data t, the rotation matrix R, and the rotation matrix from the posture represented by X ′, Y ′, Z ′ to the posture represented by X, Y, Z. In this case, it can be obtained by the following formula.
p ′ = R (pt)
The measurement object information is directly used as a reference direction vector, and a plane for obtaining base_dist is defined by the reference direction vector and the origin.

  With the progress of aging social infrastructure in recent years, demands for maintenance and repair of these have increased. This is no exception in elevators, one of the familiar infrastructures, and demand for replacement is increasing. At the time of replacement, there are already facilities that constitute the elevator before replacement. In order to replace them with new equipment, the dimensions of the existing equipment are measured and building materials suitable for the elevator hoistway are determined. However, it is difficult to stop the operating elevator for a long time from the viewpoint of the convenience of the building. Therefore, only a limited number of skilled engineers can perform the measurement work.

  On the other hand, there is a method of acquiring three-dimensional shape data (shape data) in an elevator hoistway using equipment such as a laser range finder that can measure a three-dimensional distance from a certain point to an object. A method using such equipment is called three-dimensional measurement. This method has advantages such that the time for stopping the elevator is short, skilled techniques are not required, and the shape in the elevator hoistway can be reconfirmed at the stage of making a replacement plan.

  By the way, since the data finally required for replacement is a dimension, in the case of three-dimensional measurement, the dimension is measured later from the shape data. In a general replacement operation, the inner dimensions between the wall surfaces, the distance from the rail to the wall surface, the distance between brackets attached to the wall surface at regular intervals to support the rail, and the like are measured. In other words, the measurement is repeated for a surface facing in a direction that is approximately the same. Such a situation appears not only in the elevator hoistway but also in various places such as a guide for gripping piping in the factory and a fixture for fixing the rail in the factory.

  As described above, measurement can be performed by a method such as selecting a measurement reference point on the shape data displayed on the display. At this time, in order to clarify the dimension measured by the set measurement reference point, a line segment having the measurement reference point as an end point is often defined and displayed on the display. Since the dimension measurement is performed a plurality of times, this method is also useful for clarifying the combination of measurement reference points.

  Thus, when it is going to obtain a dimension from shape data, a measurement position is determined from the rendering result displayed on the display. However, it may be difficult to grasp the sense of distance, the inclination of the shape, or the like only from the shape data. That is, it may be difficult to identify the measurement target.

  On the other hand, there is a method of imaging the inside of the hoistway with a camera. By displaying the captured image and the shape data together, the measurement target can be easily identified. However, the image is not helpful when the measurement target is hidden from the viewpoint of the camera or when the measurement target is located outside the angle of view of the camera.

  There is also a reference method for coloring a point group according to the distance from the viewpoint. By performing coloring, two points having different distances from the viewpoint are displayed in different colors. Thereby, identification of a measuring object can be made easy. However, when dimension measurement is considered, unless the coloring suitable for the measurement target is performed, it becomes difficult to identify the measurement target. For example, when it is desired to measure the distance between two planes having a normal vector orthogonal to the line-of-sight vector, if the coloring is simply performed based on the distance from the viewpoint, the two planes are not colored in a single color. For this reason, it becomes difficult to see the measurement target.

  On the other hand, in the embodiment, the reference direction vector is calculated based on the positional relationship of the measuring instrument with respect to the hoistway and the arrangement information of the measurement object in the hoistway (that is, measurement target information). Then, color information is assigned to the point cloud data based on the reference direction vector. Thereby, it becomes possible to express the anteroposterior relation of the selected measurement object or a part thereof among the plurality of measurement objects.

FIG. 5A to FIG. 5C are schematic views illustrating point group data to which color information is assigned.
For example, the first position C1 (viewpoint) as shown in FIG. For example, based on the point cloud data, an image in which the hoistway is viewed along the Z-axis direction from the first position C1 is generated.
FIG. 5B and FIG. 5C illustrate an image generated based on the point cloud data to which color information is assigned by the color information determination apparatus 101 according to the embodiment. In FIG. 5B and FIG. 5C, the hatching density represents the color of the image. In other words, areas having different hatching densities mean areas displayed in different colors.

  For example, consider a case where the distance from the wall 63 to the rail 62 is measured. At this time, for example, the user selects the wall 63 as the measurement object. In this case, the direction indicated by the measurement target information (or the reference direction vector) is, for example, the direction Db shown in FIG. The direction Db is, for example, a direction perpendicular to the Z-axis direction. Based on this direction Db, color information is assigned to the point cloud data. Thereby, for example, an image as shown in FIG. 5B is obtained. In FIG. 5B, colors are added so that the unevenness along the X-axis direction and the unevenness along the Y-axis direction can be easily grasped. For example, since the color of the wall 63 is different from the color of the rail 62, the front-rear relationship between the wall 63 and the rail 62 is easy to grasp.

  For example, consider a case where the distance between the brackets 61 is measured. At this time, for example, the user selects the bracket 61 as the measurement object. In this case, the direction indicated by the measurement target information (or the reference direction vector) is, for example, the direction Dc shown in FIG. The direction Dc is, for example, a direction parallel to the Z-axis direction. Based on this direction Dc, color information is assigned to the point cloud data. Thereby, for example, an image as shown in FIG. 5C is obtained. In FIG. 5 (c), colors are added so that the unevenness along the Z-axis direction can be easily grasped. For example, the wall 63 is periodically colored along the Z-axis direction. One bracket 61 is represented by a single color, for example. Thereby, it is easy to grasp the positions of the two brackets 61.

  According to the embodiment, it becomes easy to grasp the inclination of the surface and the front-rear relationship of the measurement locations that appear repeatedly. Thereby, an easy-to-see image can be provided. A plurality of types of vectors such as a posture vector of the distance meter 71, a vector indicating the extending direction of the hoistway (Z-axis direction), a vector indicating the arrangement of the measurement object (measurement target information), and the like in one coordinate system. It can be difficult to handle properly. On the other hand, according to the above-described steps S101 to S105, it is possible to appropriately process these and assign color information.

FIG. 6 is a block diagram illustrating an image generation apparatus according to the first embodiment.
FIG. 7 is a flowchart illustrating the process of the image generation device according to the first embodiment.
As illustrated in FIG. 6, the image generation device 201 according to the present embodiment includes a color information determination device 101, a display image generation unit 3, and a second acquisition unit 4.
The same description as the color information determination apparatus described with reference to FIGS. 1 to 5 can be applied to the color information determination apparatus 101 included in the image generation apparatus 201.

  In this example, the measuring instrument 75 further includes an imaging device 73 (camera) (see FIG. 2). A first image is generated based on the obtained point cloud data. The first image is, for example, a display image displayed to the user.

  For example, the positional relationship between the imaging device 73 and the distance meter 71 is defined in advance. The arrangement of the imaging device 73 in the entire measuring instrument 75 is defined in advance. The imaging device 73 images at least a part of the hoistway (target object 60). For example, the imaging device 73 captures a plurality of images (hereinafter referred to as second images).

  First, in step S <b> 106, the second acquisition unit 4 acquires the second image captured by the image capturing device 73 and information on the positional relationship of the image capturing device 73 with respect to the object 60 when the second image is captured. . This positional relationship information includes information on the position of the image pickup device 73 (position data) and information on the posture of the image pickup device 73 (posture data). Here, the attitude data of the image pickup device 73 is information on the orientation of the image pickup device 73.

  As a method for obtaining posture data and position data, there are a method of combining sensors such as a gyroscope and a GPS, a method of estimating the motion of the image pickup device 73 itself (motion of the measuring device 75) from a camera image, and a method of combining these. . Thereby, the position with respect to the hoistway of the imaging device 73 and the direction (posture) of the imaging device 73 with respect to the hoistway are acquired.

  For example, the posture data includes a vector (vector Z ′) indicating the forward direction of the image pickup device 73, a vector (vector Y ′) indicating the upward direction of the image pickup device 73, and a normal vector (vector) of a plane including these vectors. X ′). Note that the forward direction of the image pickup device 73 refers to a direction from the image pickup device 73 toward an area (image pickup region) where the image pickup device 73 takes an image. For example, the attitude data of the imaging device 73 is represented by the same coordinate system as the attitude data of the distance meter 71.

  Thus, the second acquisition unit 4 acquires the second image, the posture data and the position data at the time of imaging. Hereinafter, the second image captured by the image capturing device 73, the attitude data of the image capturing device 73 when the second image is captured, and the position data of the image capturing device 73 when the second image is captured. , Is referred to as sensor image data. For example, the second acquisition unit 4 acquires a plurality of sensor image data. The number of sensor image data is the same as the number of acquired second images.

In steps S107 to S109, the display image generation unit 3 generates a first image based on the point cloud data for which color information is determined (allocated).
First, viewpoint information for generating a first image is input to the display image generation unit 3. Then, the display image generation unit 3 performs CG rendering at a given viewpoint.

  The viewpoint information includes the viewpoint position (first position C1) and the direction of the line of sight. The position of the viewpoint (first position C1) may be an arbitrary position in the three-dimensional space. The direction of the viewpoint is represented by, for example, a vector (LookAt vector) in the line-of-sight direction (first direction Da) and a vector (Up vector) indicating the upward direction viewed from the viewpoint. The first image is an image of the object 60 viewed from the first position C1 along the first direction Da.

Below, the flow of the process in the display image generation part 3 is shown.
(Step S107)
The display image generation unit 3 determines whether there is sensor image data corresponding to the given viewpoint. Specifically, the display image generation unit 3 checks whether sensor image data that meets a predetermined criterion exists among the plurality of sensor image data acquired by the second acquisition unit 4.

  The predetermined reference is, for example, a reference determined for two values (first value and second value). When the first value is less than or equal to a predetermined first threshold value and the second value is less than or equal to a predetermined second threshold value, the display image generation unit 3 matches the sensor image data with the reference. It is determined that

Here, the first value is a value based on the position data of the imaging device 73 and the position of the viewpoint. Specifically, the first value is the distance between the position of the viewpoint (first position C1) and the position of the imaging device 73.
The second value is a value based on the orientation of the viewpoint and the attitude data of the imaging device 73. The second value is based on at least the angle between the first direction (line-of-sight direction) and the second direction (front direction of the image pickup device 73). Specifically, the second value is defined by the following first to third angles. The first angle is an angle formed by the LookAt vector of the viewpoint and the vector Z ′ of the image pickup device 73. The second angle is an angle formed by the viewpoint Up vector and the vector Y ′ of the image pickup device 73. The third angle is an angle formed by a vector perpendicular to the LookAt vector and the Up vector and the vector X ′ of the image pickup device 73. For example, the second value is defined as the sum of the first angle, the second angle, and the third angle. The second value may be defined as the maximum value of the first angle, the second angle, and the third angle. Note that the value of the first threshold and the value of the second threshold are arbitrary.

  If there is sensor image data that meets the reference, the display image generation unit 3 proceeds to step S108. If there is no sensor image data that matches the reference, the display image generation unit 3 proceeds to step S109.

(Step S108)
FIG. 8A to FIG. 8C are schematic views illustrating processing in the display image generation unit.
As illustrated in FIG. 8A, in step S <b> 108, the display image generation unit 3 arranges and renders the second image of the sensor image data in the view volume 76. Thereby, for example, an image 77 as shown in FIG. 8B is generated. Note that the view volume 76 at this time is obtained from the angle of view, focal length, and the like of the image pickup device 73. The internal parameters of the image pickup device 73 may be obtained by camera calibration generally performed in the field of image recognition, and the view volume 76 may be set based on the internal parameters.

(Step S109)
As illustrated in FIG. 7, the display image generation unit 3 performs step S109 after step S107 or after step S108. The display image generation unit 3 renders the point cloud data 78 to which colors are assigned based on the viewpoint information. At this time, when the image 77 is generated in step S <b> 108, the display image generation unit 3 adds the point cloud data 78 to the image 77. When additionally writing, the result of step S108 and the result of rendering the point cloud data 78 may be blended by alpha blending. For example, an image as shown in FIG. 8C is obtained. The view volume used at this time is obtained by the same method as in step S108. That is, the view volume used in step S109 is the same as the view volume obtained from the parameters of the imaging device 73. In this case, the location indicated by the second image of the sensor image data is the same as the location indicated by the result of rendering the point cloud. Thereby, visual information such as the texture of the object 60 can be obtained from the second image, and the depth of the object 60 can be grasped from the color assigned to the point cloud data.

  As described above, in step S108 and step S109, the first image obtained by rendering the point cloud data and the second image captured by the imaging device 73 are superimposed. With the second image, an image related to the measurement location can be grasped. Furthermore, it becomes possible to grasp the depth feeling that cannot be grasped from the two-dimensional image by the color of the point cloud.

(Second Embodiment)
FIG. 9 is a block diagram illustrating a color information determination device according to the second embodiment.
FIG. 10 is a flowchart illustrating the processing of the color information determination device according to the second embodiment.
As illustrated in FIG. 9, the color information determination apparatus 102 according to the present embodiment includes a first acquisition unit 1b, a determination unit 2b, and a designation unit 5. The determination unit 2 b includes a measurement direction determination unit 21 b and a color information determination unit 22.

  The 1st acquisition part 1b acquires point cloud data etc. similarly to the 1st acquisition part 1 explained about a 1st embodiment (Step S101). Furthermore, the first acquisition unit 1b acquires surface data calculated from the point cloud data (step S201). The surface data includes information regarding a plurality of surfaces representing the three-dimensional shape of the object 60. The surface data closely approximates the point cloud data.

  For example, the surface data represents a three-dimensional shape by a plurality of triangular surfaces. A unique surface ID is defined for each triangular surface. The three-dimensional shape may be expressed by a polygonal surface having a quadrangle or more, or may be parametrically expressed by a curved surface or the like. In this case, a polygonal surface or a curved surface may be converted into a triangular surface.

  For example, the surface data is acquired from an external storage device in the same manner as the point cloud data. When surface data is not acquired from the outside, surface data can be generated by constructing a triangular surface from point cloud data. For example, when a point is obtained by a laser range finder, there is a method of configuring a surface using a detector of the laser range finder. In addition, when a point is obtained by depth estimation of a stereo camera, there is a method of constructing a surface using pixel adjacency relationships. In addition, there are various methods such as a method of directly estimating a surface from a point cloud.

  For example, the user inputs information regarding a location to be measured to the designation unit 5 via an input device. The designation unit 5 designates a measurement reference point based on the user input (step S202). The measurement reference point is designated on one of the plurality of surfaces of the surface data. For the processing in the designation unit 5, polygon data picking processing used in the CG field is used. Note that an arbitrary device such as a touch pen, a touch panel, a mouse, or a keyboard is used as the input device.

The measurement direction determination unit 21b calculates a reference direction vector based on the measurement reference point designated by the designation unit 5 and the surface data (step S203). In the second embodiment, the measurement object as described with respect to the first embodiment is not specified.
Details of step S203 will be described below.
First, a measurement target surface group is calculated from the measurement reference point designated by the designation unit 5. The measurement target surface group is a set of surface IDs of triangular surfaces that can be regarded as a plane in the vicinity of the triangular surface including the measurement reference point.

FIG. 11 is a flowchart illustrating the calculation of the measurement target surface group.
FIG. 12 is a schematic diagram illustrating the calculation of the measurement target surface group.
In step S1201 shown in FIG. 11, the surface ID of a triangular surface (hereinafter referred to as the first surface F1) including the measurement reference point is added to the stack 1.
Thereafter, one surface ID is extracted from the stack 1 (step S1202), and the surface ID extracted in step S1202 is added to the evaluated surface list (step S1203).

  Thereafter, the search target surface F2 is calculated (step S1204). The search target surface F2 is a triangular surface that shares a ridge line with the triangular surface (search reference surface F3) corresponding to the surface ID extracted in step S1202 and whose surface ID is not registered in the evaluated surface list. . For example, a plurality of search target surfaces F2 (search target surface group) are calculated.

Thereafter, one unevaluated search target surface F2 is selected from the search target surface group (step S1205), and the surface ID of the selected surface is added to the evaluated surface list (step S1206). An angle θn formed between the normal vector Vn2 of the selected search target surface F2 and the normal vector Vn3 of the search reference surface F3 is calculated (step S1207).
Thereafter, it is determined whether the angle θn is equal to or smaller than a predetermined threshold (step S1208). If the angle θn is larger than the threshold value, the process proceeds to step S1210. When the angle θn is equal to or smaller than the threshold value, it is assumed that the direction of the search target surface F2 is the same as the direction of the search reference surface F3, and the process proceeds to step S1209. In step S1209, the surface ID of the search target surface F2 is added to the stack 1 and the measurement target surface group.

  Thereafter, it is determined whether all search target planes F2 included in the search target plane group have been evaluated (step S1210). If there is an unevaluated search target surface F2, the processing from step S1205 to step S1210 is repeated. If there is no unevaluated search target surface F2, the process proceeds to step S1211. In step S1211, it is determined whether stack 1 is empty. If the stack 1 is not empty, the processing from step S1202 to step S1211 is repeated. If the stack 1 is empty, the process ends.

  As described above, the processing is repeated according to the flowchart of FIG. Thereby, a measurement object surface group is obtained. At least one set of measurement target surface groups is calculated for each measurement reference point.

「Ｎ」は、計測対象面群に含まれる面の数である。「ｉ」は、計測対象面群に含まれる面の通し番号である。「v_normal_i」は、計測対象面群に含まれるｉ番目の面の正規化された法線ベクトルである。基準方向ベクトルは、例えば、計測基準点を含む第１面Ｆ１の法線の方向に基づいて算出されたベクトルである。 A reference direction vector is calculated from the obtained measurement target plane group. The reference direction vector is obtained by the following equation.

“N” is the number of surfaces included in the measurement target surface group. “I” is a serial number of a surface included in the measurement target surface group. “V_normal _i ” is a normalized normal vector of the i-th surface included in the measurement target surface group. The reference direction vector is a vector calculated based on, for example, the direction of the normal line of the first surface F1 including the measurement reference point.

  The calculation method of the reference direction vector (v_base) is not limited to the above. For calculating the reference direction vector, various methods for determining the representative vector of the surface group can be used. For example, the principal component analysis is performed on the position coordinates of the vertices constituting the surface included in the measurement target surface group. The main axis corresponding to the smallest eigenvalue may be used as the reference direction vector. For example, a quadratic or higher order polynomial is applied to the vertices constituting the surfaces included in the measurement target surface group. The normal vector at the position of the measurement reference point mapped onto the polynomial curved surface may be used as the reference direction vector.

  Thereafter, as shown in FIG. 10, the distance between the plane defined by the reference direction vector (the plane corresponding to the measurement target plane group) and each point represented by the point group data is obtained (step S104). ). For example, the distance measured along the reference direction vector is obtained. Thereafter, color information is assigned to the point cloud data according to the value obtained in step S104 (step S105).

  That is, the color information determination unit 22 assigns color information to the point cloud data according to the distance between the plane including the first surface F1 and each of the plurality of points on the object 60. Here, the plane including the first surface F1 is, for example, a part of a plurality of surfaces of the surface data. The part includes the first surface F1 and can be regarded as a substantially flat surface. Note that the same description as that of the first embodiment can be applied to step S104 and step S105.

  According to the above, at the stage where the user designates one point for measurement, for example, coloring that makes it easy to see the measurement object arranged in a certain direction is automatically performed. For this reason, it becomes easy to input the other measurement reference point. Further, as long as the same measurement object is continuously measured, the measurement object is, for example, always easy to see and colored. For this reason, continuous measurement of the measurement object arranged in a fixed direction becomes easy.

FIG. 13 is a block diagram illustrating an image generation apparatus according to the second embodiment.
FIG. 14 is a flowchart illustrating the processing of the image generation apparatus according to the second embodiment.
As illustrated in FIG. 13, the image generation device 202 according to the present embodiment includes a color information determination device 102, a display image generation unit 3, and a second acquisition unit 4.
The same description as the color information determination device described with reference to FIGS. 9 to 12 can be applied to the color information determination device 102 included in the image generation device 202.
The same description as the image generation device 201 according to the first embodiment can be applied to the display image generation unit 3 and the second acquisition unit 4.

  Steps S101 to S105 illustrated in FIG. 14 are the same as Steps S101 to S105 described with reference to FIG. Through these processes, color information is assigned to the point cloud data.

  Further, the image generation device 202 executes Steps S106 to S109. Steps S106 to S109 in the image generation apparatus 202 are the same as the processes described with reference to FIG.

  As described above, based on the information of the measurement reference point designated by the user, color information is assigned to the point cloud data, and an image that is easy for the user to view can be displayed. Furthermore, the image obtained by rendering the point cloud data and the camera image are superimposed. Thereby, it becomes easier to grasp the shape of the object, and the measurement reference point can be easily set.

  As described above, the color information determination device and the image generation device have been described as examples. However, the embodiment is a form of a program for causing a computer to execute processing in the color information determination device and processing in the image generation device. May be. Alternatively, the embodiment may be in the form of a computer-readable recording medium that records this program.

  Specific examples of the recording medium include CD-ROM (-R / -RW), magneto-optical disk, HD (hard disk), DVD-ROM (-R / -RW / -RAM), and FD (flexible disk). ), A flash memory, a memory card, a memory stick, and other various ROMs and RAMs.

  According to the embodiment, a color information determination device and an image generation device that can provide an easy-to-view image can be provided.

  In the present specification, “vertical” and “parallel” include not only strictly vertical and strictly parallel, but also include, for example, variations in the manufacturing process, and may be substantially vertical and substantially parallel. It ’s fine.

The embodiments of the present invention have been described above with reference to specific examples. However, embodiments of the present invention are not limited to these specific examples. For example, regarding a specific configuration of each element such as a first acquisition unit, a second acquisition unit, a determination unit, a measurement direction determination unit, a color information determination unit, a display image generation unit, a designation unit, etc. As long as the present invention can be carried out in the same manner and the same effect can be obtained by appropriately selecting from the above, it is included in the scope of the present invention.
Moreover, what combined any two or more elements of each specific example in the technically possible range is also included in the scope of the present invention as long as the gist of the present invention is included.

  In addition, based on the color information determination device and the image display device described above as the embodiments of the present invention, all color information determination devices and image display devices that can be implemented by those skilled in the art with appropriate design changes are also included in the present invention. As long as the gist is included, it belongs to the scope of the present invention.

  In addition, in the category of the idea of the present invention, those skilled in the art can conceive of various changes and modifications, and it is understood that these changes and modifications also belong to the scope of the present invention. .

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  DESCRIPTION OF SYMBOLS 1, 1b ... 1st acquisition part, 2, 2b ... Determination part, 3 ... Display image generation part, 4 ... Second acquisition part, 5 ... Designation part 21, 21b ... Measurement direction determination part, 22 ... Color information determination part , 60 ... object, 61 ... bracket, 62 ... rail, 63, 64 ... wall, 71 ... distance meter, 73 ... imaging device, 75 ... measuring instrument, 76 ... view volume, 77 ... image, 78 ... point cloud data, DESCRIPTION OF SYMBOLS 101,102 ... Color information determination apparatus, 201, 202 ... Image generation apparatus, C1 ... 1st position, D1-D4 1st-4th point, Da, Db, Dc ... Direction, F1 ... 1st surface, F2 ... Search Target plane, F3 ... search reference plane, La, Lb, Lc ... distance, P1 ... first part, S101-S109, S1201-S1211, S201-S203 ... step, Vn2, Vn3 ... normal vector

Claims (14)

Point cloud data representing a three-dimensional shape of the target object, including a plurality of point data measured by a measuring instrument including a distance meter and representing each position of a plurality of points included in the target object including the first portion;
Information on the positional relationship of the distance meter with respect to the object when the point cloud data is measured;
A first acquisition unit for acquiring
Determination to determine color information for each of the plurality of point data based on arrangement information based on at least one of the position, shape, and size of the first portion and the information on the positional relationship of the distance meter And
A color information determination device. The object has a portion extending in the first extending direction,
The color information determination apparatus according to claim 1, wherein the first portion is arranged based on a direction based on the first extending direction.   3. The direction based on the first extending direction is one of a direction parallel to the first extending direction and a direction perpendicular to the first extending direction. Color information determination device. The determination unit
A measurement direction determining unit that calculates a reference direction based on the arrangement information of the first part and the information on the positional relationship of the distance meter;
A color information determination unit that determines the color information based on the reference direction;
The color information determination device according to claim 1, comprising:   The color information is determined based on a distance between a plane calculated by the position of the distance meter when the point cloud data is measured and the reference direction, and each of the plurality of points. The color information determination apparatus according to claim 4. A first acquisition unit that is measured by a measuring instrument including a distance meter and includes a plurality of point data representing the positions of a plurality of points included in the object, and acquires point cloud data representing the three-dimensional shape of the object When,
A designation unit for designating a reference point on one of a plurality of surfaces calculated from the point cloud data and representing the three-dimensional shape of the object;
A determining unit that determines color information for each of the plurality of point data according to a distance between the plane based on the one of the plurality of surfaces and each of the plurality of points;
A color information determination device. The determination unit
A measurement direction determination unit that calculates a reference direction based on the reference point and the plurality of surfaces;
A color information determination unit that determines the color information based on the reference direction;
The color information determination apparatus according to claim 6, comprising:   The color information determination apparatus according to claim 6 or 7, wherein the reference direction is calculated based on a direction of the one normal line of the plurality of surfaces.   The color information determination apparatus according to claim 7 or 8, wherein the plane based on the one of the plurality of faces is calculated based on the reference direction. The plurality of points includes a first point, a second point, and a third point located between the first point and the second point,
The distance between the first point and the third point is a first distance;
The distance between the second point and the third point is the first distance;
The plurality of point data includes first point data corresponding to the first point, second point data corresponding to the second point, and third point data corresponding to the third point,
The color determined for the first point data is the same as the color determined for the second point data;
The color information determination apparatus according to claim 1, wherein the color determined for the third point data is the same as the color determined for the second point data. The plurality of points further includes a fourth point located between the first point and the third point;
The distance between the first point and the fourth point is shorter than the first distance,
The plurality of point data further includes fourth point data corresponding to the fourth point,
The color information determination apparatus according to claim 10, wherein the color determined for the fourth point data is different from the color determined for the first point data. The color information determination device according to any one of claims 1 to 11,
A display image generating unit that generates a first image of the object viewed from a first position in a three-dimensional space along a first direction based on the point cloud data for which the color information is determined;
An image generation apparatus comprising: A second acquisition unit;
The measuring instrument further includes an imager that images the object,
The second acquisition unit acquires a second image in which an imaging region of the object is imaged by the imaging device, and a positional relationship of the imaging device with respect to the object when the second image is imaged. And
The positional relationship of the image pickup device includes a position of the image pickup device with respect to the object and a second direction from the image pickup device toward the image pickup area,
In the display image generation unit, a distance between the first position and the position of the imaging device is equal to or less than a first threshold, and a value based on an angle between the first direction and the second direction is a first value. The image generation apparatus according to claim 12, wherein the first image and the second image are superimposed when the threshold value is equal to or less than two thresholds.   The image generation apparatus according to claim 13, wherein a positional relationship between the measuring instrument and the imaging device is defined in advance.

Images