JP2021140429A - Three-dimentional model generation method - Google Patents

Abstract

To generate a three-dimentional model capable of identifying a certain region where existence or absence of an object is unknown.SOLUTION: A point cloud generation unit generates point cloud data showing a three-dimensional shape of a monitored object on the basis of an image data set. A registration unit aligns the point cloud data and reference point cloud data. A voxelization unit generates voxel data on the basis of the point cloud data. A pixel position estimation unit estimates world coordinate values of each of pixels making up the image data set for each image data making up the image data set. A label update unit calculates a camera visual line straight line passing through each pixel and a camera center for each pixel of each image data making up the image data set and executes processing for updating a label of the voxel crossing the camera visual line straight line among the voxels indicated by voxel data. A voxel comparison unit compares the voxel data with reference voxel data.SELECTED DRAWING: Figure 2

Description

The present invention relates to a method of generating a 3D model.

Conventionally, there is known a method of shooting a moving image with a camera mounted on a drone or a vehicle and generating 3D point cloud data based on the shot moving image. As a method of using this 3D point cloud data, for example, Patent Document 1 describes a method of accurately generating a polygon model by analyzing a three-dimensional point cloud.

Japanese Patent No. 60080640

Another method of using the 3D point cloud data is to detect an abnormal part. In this usage method, the 3D point cloud data representing the monitored object at the time of inspection is compared with the 3D point cloud data representing the monitored object at the time of normal operation, and the difference is detected as an abnormal part. In this usage method, a method of determining the similarity based on the distance of a point cloud is generally used.

However, there is an occlusion problem in abnormality detection using 3D point cloud data. Here, the occlusion problem is that there is no object in the area where the point does not exist in the 3D point cloud represented by the 3D point cloud data, or the object is blocked by a wall or the like and cannot be photographed. As a result, it is not possible to distinguish whether or not an object exists in the area. If the object could not be photographed because it was shielded by a wall or the like, and then the object was removed, the object was detected as an abnormal part even though it had existed before. Will end up.

The present invention has been made in view of such circumstances, and an object of the present invention is to generate a 3D model capable of identifying a region where the existence or nonexistence of an object is unknown.

In order to solve the above problems, the 3D model generation method according to the present invention is a 3D model generation method executed by a computer, and is generated by continuously photographing a monitored object from a large number of points with a camera. The first step of setting a point group representing the three-dimensional shape of the monitored object, which is a point group represented by the world coordinate system based on the image group, and a voxel group represented by the world coordinate system. Therefore, each of the second step of setting the voxel group occupying the area of the three-dimensional shape represented by the point group and the pixel group of the first image included in the image group is represented by the image coordinate system. The third step of converting the coordinate values into the coordinate values represented by the world coordinate system, and the fourth step of specifying the camera line-of-sight straight line passing through the pixel and the camera for each of the pixel groups to which the coordinate values have been converted. For each of the step and the specified camera line-of-sight line, when the camera line-of-sight line intersects a plurality of voxels in the voxel group, a point constituting the point group among the plurality of voxels is determined. The first category includes a fifth step of classifying the voxels containing a number or more and closest to the camera into the first category, while classifying the other voxels into the second category. Indicates that an object exists in the area occupied by the voxels classified into the first category, and the second category indicates whether or not the object exists in the area occupied by the voxels classified into the second category. Is characterized by indicating that is unknown.

In a preferred embodiment, in the fifth step, for each of the specified camera line-of-sight lines, the camera line-of-sight line is a voxel included in the voxel group, and the number of points constituting the point cloud is equal to or more than the predetermined number. When intersecting with a voxel that does not include, the voxel is classified into a third category, and the third category indicates that there is no object in the area occupied by the voxels classified into the third category.

In a further preferred embodiment, in the third step, the coordinate values represented by the image coordinate system are converted into the coordinate values represented by the world coordinate system for each of the pixel groups of the plurality of images included in the image group. do.

In a more preferred embodiment, the 3D model generation method compares the categories of voxels having a common coordinate value between the voxel group classified into each category and another voxel group classified into each category. A sixth step of outputting information indicating the result of the comparison is further included.

According to the present invention, it is possible to generate a 3D model capable of identifying a region where the existence or nonexistence of an object is unknown.

Block diagram showing the functional configuration of the abnormality detection system 1 Flow diagram showing anomaly detection processing The figure which shows an example of the point cloud data D2 The figure which shows an example of a 3D point cloud Flow diagram showing voxel data generation processing Diagram showing an example of voxel group The figure which shows an example of the voxel data D4 Diagram showing an example of an image Flow diagram showing coordinate conversion processing The figure which shows an example of the camera attribute data D5 The figure which shows an example of the pixel position data D6 Flow diagram showing label update processing The figure which shows an example of the state which the camera is not included in the rectangular parallelepiped A diagram showing an example of a state in which a camera is included in a rectangular parallelepiped. Flow diagram showing the first update process Top view showing an example of voxel group Flow diagram showing the second update process Diagram showing label update processing Diagram showing an example of the screen

1. 1. Embodiment An abnormality detection system 1 according to an embodiment of the present invention will be described with reference to the drawings. The abnormality detection system 1 according to the present embodiment is a system for detecting an abnormality of the monitored object by generating a 3D model of the monitored object and comparing it with a normal 3D model of the monitored object. be. The abnormality detection system 1 is composed of one or more information processing devices.

FIG. 1 is a block diagram showing a functional configuration of the abnormality detection system 1. The abnormality detection system 1 shown in the figure includes an image data storage unit 101, a point group generation unit 102, a point group data storage unit 103, a reference point group data storage unit 104, a registration unit 105, a voxelization unit 106, and a voxel data storage unit. It has functions such as a unit 107, a camera attribute data storage unit 108, a pixel position estimation unit 109, a pixel position data storage unit 110, a label update unit 111, a reference voxel data storage unit 112, and a voxel comparison unit 113.

Among these functions, various storage units are realized by a storage device such as an HDD. Other functions are realized by an arithmetic processing unit such as a CPU executing an abnormality detection program stored in the storage device.

Hereinafter, the abnormality detection process executed by using these functions will be described. FIG. 2 is a flow chart showing this abnormality detection process.

The point cloud generation unit 102 of the abnormality detection system 1 generates point cloud data D2 representing the three-dimensional shape of the monitored object based on the image data set D1 stored in the image data storage unit 101 (step Sa1). Here, the image data set D1 stored in the image data storage unit 101 is a set of image data generated by continuously photographing the monitored object from a large number of points with a camera at the time of inspection. In other words, it is video data. This image data set D1 is generated by, for example, a camera attached to a moving body such as an aircraft or a vehicle.

In this step Sa1, the point cloud generation unit 102 specifically represents the three-dimensional shape of the object to be monitored from the image data set D1 by using SfM (Structure from Motion) and MVS (Multi-view Stereo) techniques. Restore the 3D point cloud. When the 3D point cloud is restored, the point cloud data D2 is generated by converting the relative coordinate values of the restored 3D point cloud into world coordinate values based on the ground reference point specified by the user.

FIG. 3 is a diagram showing an example of this point cloud data D2. The point cloud data D2 shown in the figure is composed of point IDs, coordinate values, and color information of each point constituting the point cloud. Here, the point ID is the point identification information, the coordinate value is the coordinate value of the world coordinate system, and the color information is the RGB value.

FIG. 4 is a diagram showing an example of a 3D point cloud represented by the point cloud data D2. The position of each point of the 3D point cloud PG shown in the figure is represented by the world coordinate values (X, Y, Z).

When the point cloud generation unit 102 generates the point cloud data D2, the point cloud generation unit 102 stores it in the point cloud data storage unit 103.

The point cloud generator 102 is realized, for example, by executing Pix4D (registered trademark) mapper.

When the point cloud data D2 is generated, the registration unit 105 uses the ICP (Iterative Closest Point) algorithm to generate the point cloud data D2 and the reference stored in the reference point cloud data storage unit 104 in advance. The point cloud data D3 is aligned (step Sa2). Here, the reference point cloud data D3 stored in advance in the reference point cloud data storage unit 104 is the data of the 3D point cloud representing the three-dimensional shape of the monitored object at the normal time. The data structure of the reference point cloud data D3 is the same as that of the point cloud data D2 illustrated in FIG.

Specifically, the registration unit 105 calculates the amount of deviation between the point cloud data D2 and the reference point cloud data D3, and corrects the coordinate values of each point of the point cloud data D2 so as to minimize the calculated deviation amount. do.

When the alignment is completed, the voxelization unit 106 generates voxel data D4 based on the point cloud data D2 (step Sa3). FIG. 5 is a flow chart showing this voxel data generation process.

In the voxel data generation process shown in the figure, the voxelization unit 106 includes the maximum and minimum values of the coordinate value X of the point cloud data D2, the maximum and minimum values of the coordinate value Y, and the maximum and minimum values of the coordinate value Z. Specify the value (step Sb1). In other words, the length, width and height of the rectangular parallelepiped circumscribing the three-dimensional shape represented by the point cloud data D2 are specified. When the length of the rectangular parallelepiped is specified, a voxel group that divides the world coordinate space and occupies at least the above-mentioned rectangular parallelepiped region is set (step Sb2). Each voxel constituting this voxel group has a predetermined size. Further, each surface of the rectangular parallelepiped formed by this voxel group is parallel to any coordinate axis of the world coordinate system.

FIG. 6 is a diagram showing an example of this voxel group. The voxel group shown in the figure is composed of 27 voxels, and constitutes a cube CB as a whole. The cube CB composed of this voxel group occupies the region of the 3D point cloud PG represented by the point cloud data D2. Further, this cube CB has a plane parallel to any coordinate axis of the world coordinate system. For example, the plane PL1 shown in FIG. 6 is parallel to the X axis.

After setting the voxel group, the voxel conversion unit 106 generates voxel data D4 for the set voxel group (step Sb3). FIG. 7 is a diagram showing an example of the voxel data D4. The voxel data D4 shown in the figure is composed of an index, a label, and a score of each voxel constituting the voxel group. Here, the index is a coordinate value of the world coordinate system for identifying a voxel. More specifically, it is the coordinate value of the vertex having the maximum coordinate value Y and the minimum coordinate value Z among the eight vertices constituting the voxel. For example, the index of voxel VX1 shown in FIG. 6 is the coordinate value of the vertex PT1.

The label is information indicating whether or not an object exists in the area occupied by the box, or whether or not the existence or nonexistence of the object is unknown. There are three types of labels: "unknown", "empty" and "occupancy". Of these, the label "unknown" indicates that the area occupied by the voxels is blocked by a wall or the like and cannot be photographed, and as a result, it is unknown whether or not an object exists in that area. This label "unknown" is the initial value. The label "empty" indicates that there are no objects in the area occupied by the voxels. The label "occupancy" indicates that the object is in the area occupied by the voxels.

The score is the number of points included in the voxel among the points constituting the point cloud data D2.

When the voxel conversion unit 106 generates the voxel data D4, it stores it in the voxel data storage unit 107.

When the voxel data generation process is completed, the pixel position estimation unit 109 estimates the world coordinate values of each pixel constituting the image for each image data constituting the image data set D1 (step Sa4). More specifically, the image coordinate value of each pixel constituting the image is converted into the world coordinate value. Here, the image coordinate value is a coordinate value indicating a position on a plane represented by the image coordinate system.

FIG. 8 is a diagram showing an example of an image represented by image data. The image IM shown in the figure is an image generated by photographing the monitored object with the camera CA. The monitored object shown in this image IM is arranged at the position of the cube CB in the figure. An image coordinate system is defined in this image IM. This image coordinate system is a coordinate system having a U-axis extending in the horizontal direction and a V-axis extending in the vertical direction with the origin as the upper left corner.

FIG. 9 is a flow chart showing a coordinate conversion process executed by the pixel position estimation unit 109.

In the coordinate conversion process shown in the figure, the pixel position estimation unit 109 selects image data from the image data set D1 (step Sc1). When the image data is selected, the camera attribute data D5 stored in the camera attribute data storage unit 108 is referred to to specify the camera parameters corresponding to the selected image data (step Sc2). Here, the camera attribute data D5 stored by the camera attribute data storage unit 108 is data indicating the parameters of the camera that generated the image data set D1.

FIG. 10 is a diagram showing an example of the camera attribute data D5. The camera attribute data D5 shown in the figure is composed of an image ID of the image data constituting the image data set D1 and a plurality of sets of camera parameters when the image data is generated. Here, the camera parameters include the focal length and the center of the image as internal parameters, and the rotation axis angle and the translation matrix as external parameters.

この式（１）において、Ｕ、Ｖは、画像座標値である。ｆｘ、ｆｙは、カメラの横軸及び縦軸の焦点距離である。Ｃｘ、Ｃｙは、画像座標系における光軸と画像面との交点の位置（画像中心）である。ｒ１１〜ｒ３３は、カメラの回転軸角度を表す回転行列の成分である。ｔ１〜ｔ３は、カメラの平行移動を表す行列の成分である。Ｘ、Ｙ、Ｚは、世界座標値である。 When the parameter is specified, the pixel position estimation unit 109 selects a pixel from the selected image data (step Sc3). When a pixel is selected, the image coordinate value of the selected pixel and the specified parameter are substituted into the following equation (1) to calculate the world coordinate value of the pixel (step Sc4).

In this equation (1), U and V are image coordinate values. fx and fy are focal lengths on the horizontal and vertical axes of the camera. Cx and Cy are the positions (center of the image) of the intersections of the optical axis and the image plane in the image coordinate system. r11 to r33 are components of a rotation matrix representing the rotation axis angle of the camera. t1 to t3 are components of a matrix representing the translation of the camera. X, Y, and Z are world coordinate values.

When the pixel position estimation unit 109 calculates the world coordinate value, the calculated world coordinate value is stored in the pixel position data storage unit 110 (step Sc5). At that time, the calculated world coordinate value is stored in association with the image coordinate value of the pixel and the image ID of the selected image data. Here, the pixel position data storage unit 110 stores the pixel position data D6. FIG. 11 is a diagram showing an example of pixel position data D6. The pixel position data D6 shown in the figure is composed of an image ID, an image coordinate value of the pixel, and a world coordinate value.

When the storage of the world coordinate values is completed, it is determined whether or not there are unselected pixels (step Sc6). As a result of this determination, if there are unselected pixels (YES in step Sc6), the process returns to step Sc3. On the other hand, when there are no unselected pixels (NO in step Sc6). It is determined whether or not there is unselected image data (step Sc7). As a result of this determination, if there is unselected image data (YES in step Sc7), the process returns to step Sc1. On the other hand, if there is no unselected image data (NO in step Sc7), the coordinate conversion process ends. As a result of this coordinate conversion process, pixel position data D6 is generated.

When the coordinate conversion process is completed, the label updating unit 111 calculates a camera line-of-sight straight line passing through the pixel and the camera center for each pixel of each image data constituting the image data set D1, and the voxel represented by the voxel data D4. Among them, the process of updating the voxel label that intersects the straight line of sight of the camera is executed (step Sa5). FIG. 12 is a flow chart showing this label update process.

この条件式（２）において、Ｘｖｍａｘ、Ｘｖｍｉｎは、ボクセルデータＤ４の座標値Ｘの最大値、最小値である。Ｙｖｍａｘ、Ｙｖｍｉｎは、ボクセルデータＤ４の座標値Ｙの最大値、最小値である。Ｚｖｍａｘ、Ｚｖｍｉｎは、ボクセルデータＤ４の座標値Ｚの最大値、最小値である。Ｘｃ、Ｙｃ、Ｚｃは、カメラの座標値である。 In the label update process shown in the figure, the label update unit 111 selects image data from the image data set D1 (step Sd1). When the image data is selected, the coordinate values of the camera corresponding to the selected image data are specified by referring to the camera attribute data D5 stored in the camera attribute data storage unit 108 (step Sd2). When the coordinate value of the camera is specified, it is determined whether or not the specified coordinate value is included in the voxel group represented by the voxel data D4 (step Sd3). In other words, it is determined whether or not the camera is included in the rectangular parallelepiped composed of the voxels as a whole. At that time, the label updating unit 111 determines whether or not the following conditional expression (2) is satisfied.

In this conditional expression (2), Xvmax and Xvmin are the maximum and minimum values of the coordinate value X of the voxel data D4. Yvmax and Yvmin are the maximum and minimum values of the coordinate value Y of the voxel data D4. Zvmax and Zvmin are the maximum and minimum values of the coordinate value Z of the voxel data D4. Xc, Yc, and Zc are the coordinate values of the camera.

If the conditional expression (2) is not satisfied, that is, if the rectangular parallelepiped does not include the camera (NO in step Sd3), the first update process is executed (step Sd4). FIG. 13 is a diagram showing an example of a state in which the rectangular parallelepiped does not include a camera. The camera CA shown in the figure exists outside the cube CB. On the other hand, when the conditional expression (2) is satisfied, that is, when the rectangular parallelepiped includes the camera (YES in step Sd3), the second update process is executed (step Sd5). FIG. 14 is a diagram showing an example of a state in which a camera is included in a rectangular parallelepiped. The camera CA shown in the figure exists inside the cube CB.

When the first or second update process is completed, it is determined whether or not there is unselected image data (step Sd6). As a result of this determination, if there is unselected image data (YES in step Sd6), the process returns to step Sd1. On the other hand, if there is no unselected image data (NO in step Sd6), the label update process ends.

Next, the first update process will be described. FIG. 15 is a flow chart showing the first update process.

この式（３）において、Ｘｃ、Ｙｃ、Ｚｃは、カメラの座標値である。Ｘｐ、Ｙｐ、Ｚｐは、画素の座標値である。ｔは媒介変数である。 In the first update process shown in the figure, the label update unit 111 selects pixels from the image data selected in step Sd1 (step Se1). When a pixel is selected, the coordinate value (specifically, the world coordinate value) of the selected pixel is specified with reference to the pixel position data D6 (step Se2). When the coordinate values are specified, the specified coordinate values and the straight line of the camera line of sight passing through the coordinate values of the camera specified in step Sd2 are calculated (step Se3). At that time, the label updating unit 111 calculates the camera line-of-sight line by substituting the coordinate values of the pixels and the camera into the following equation (3).

In this equation (3), Xc, Yc, and Zc are the coordinate values of the camera. Xp, Yp, and Zp are the coordinate values of the pixels. t is a parameter.

The camera line-of-sight line calculated here corresponds to, for example, the camera line-of-sight line L1 shown in FIG.

これらの式（４）〜（９）において、Ｘｖｍａｘ、Ｘｖｍｉｎは、ボクセルデータＤ４の座標値Ｘの最大値、最小値である。Ｙｖｍａｘ、Ｙｖｍｉｎは、ボクセルデータＤ４の座標値Ｙの最大値、最小値である。Ｚｖｍａｘ、Ｚｖｍｉｎは、ボクセルデータＤ４の座標値Ｚの最大値、最小値である。 When the camera line-of-sight line is calculated, the surface of the rectangular parallelepiped where the calculated camera line-of-sight line intersects is calculated (step Se4). Here, each surface of the rectangular parallelepiped is represented by the following equations (4) to (9).

In these equations (4) to (9), Xvmax and Xvmin are the maximum and minimum values of the coordinate values X of the voxel data D4. Yvmax and Yvmin are the maximum and minimum values of the coordinate value Y of the voxel data D4. Zvmax and Zvmin are the maximum and minimum values of the coordinate value Z of the voxel data D4.

The label updating unit 111 calculates the parameter t for each surface of the rectangular parallelepiped by sequentially solving the calculated equation of the line of sight of the camera and any one of the simultaneous equations (4) to (9). As a result, if the denominator of the calculated parameter t is "0" for any of the surfaces, that is, if the straight line of the camera line of sight does not intersect the rectangular parallelepiped (NO in step Se5), the process proceeds to step Se19. On the other hand, when the denominator of the calculated parameter t is not "0" for any of the surfaces, that is, when the straight line of the camera line of sight intersects the square (YES in step Se5), the parameter t is "0". It is determined whether or not the number of non-faces (in other words, the faces where the straight lines of the camera line of sight intersect) is "2" (step Se6). As a result of this determination, when the number of surfaces whose parameter t is not "0" is "2" (YES in step Se6), the surface on which the smaller parameter t is calculated is set as the entrance surface (step Se7). ). In other words, the surface closer to the camera is the entrance surface. The entrance surface specified here corresponds to, for example, the surface PL2 shown in FIG. On the other hand, when the number of faces whose parameter t is not "0" is more than "2" (NO in step Se6), that is, when the camera line-of-sight line intersects the apex of the rectangular parallelepiped, the camera line-of-sight line intersects. Any surface to be used is used as the entrance surface (step Se8).

When the entrance surface is specified, one of the specified entrance surfaces, the opposite surface thereof, and a plurality of cross sections existing between the entrance surface and the opposite surface is selected (step Se9). At that time, the label updating unit 111 selects in order from the surface closest to the camera. Here, the opposite surface and the cross section will be described before proceeding to the description of the next step.

FIG. 16 is a plan view showing an example of the voxel group. The voxel group shown in the figure is composed of nine voxels, and constitutes a rectangular parallelepiped RP as a whole. The rectangular parallelepiped RP composed of this voxel group intersects the camera line-of-sight line L2. Of the surfaces that intersect the camera line-of-sight line L2, the surface PL3 corresponds to the entrance surface. The surface PL4 parallel to the entrance surface corresponds to the opposite surface. Then, the cross sections CS1 and CS2 exist between the entrance surface and the opposite surface. These cross sections CS1 and CS2 are cross sections that are parallel to the entrance surface and the opposite surface and are arranged at voxel length intervals.

When any surface is selected from the plurality of surfaces described above, the label updating unit 111 determines whether or not the selected surface is an entrance surface (step Se10). As a result of this determination, if the selected surface is the entrance surface (YES in step Se10), step Se11 is skipped and the process proceeds to step Se12. On the other hand, when the selected surface is not the entrance surface (NO in step Se10), it is determined whether or not the selected surface intersects the straight line of the camera line of sight (step Se11). Specifically, the parameter t is calculated for the surface by solving the simultaneous equations of the selected surface equation and the calculated equation of the line of sight of the camera. As a result of this determination, if the denominator of the calculated parameter t is "0", that is, if the selected surface does not intersect the straight line of the camera line of sight (NO in step Se11), the process proceeds to step Se19. On the other hand, when the denominator of the calculated parameter t is not "0", that is, when the selected surface intersects the straight line of the camera line of sight (YES in step Se11), the intersection is calculated (step Se12). Specifically, the intersection is calculated by substituting the calculated parameter t into the calculated straight line of the camera line of sight.

When the intersection point is calculated, the voxel data D4 is referred to, and the voxel having the calculated intersection point on the surface is specified (step Se13). More specifically with reference to FIG. 16, when the intersection PT2 is calculated, the voxel VX2 is specified, when the intersection PT3 is calculated, the voxel VX3 is specified, and when the intersection PT4 is calculated, the voxel VX4 is specified. Is identified. When two voxels having the calculated intersection on the surface are specified, the voxel farther from the camera is selected.

When the voxel is specified, the number of points included in the selected voxel is specified with reference to the voxel data D4 (step Se14). Then, it is determined whether or not the number of the specified points is "0" (step Se15). As a result of this determination, if the number of specified points is "0" (YES in step Se15), the label of the selected voxel is updated from "unknown" to "empty" (step Se16). The label "empty" indicates that there are no objects in the area occupied by the voxel. Then, the process proceeds to step Se18. On the other hand, if the number of specified points is not "0" (NO in step Se15), the label of the selected voxel is updated from "unknown" to "occupancy" (step Se17). The label "occupancy" indicates that an object exists in the area occupied by the voxel. Then, step Se18 is skipped and the process proceeds to step Se19.

Here, by skipping step Se18, the intersection with the straight line of the camera line of sight is not specified for the surface existing behind the selected surface. Therefore, even if there is an intersection with the straight line of the camera line of sight on the back surface, the label of the voxel having the intersection on the surface remains "unknown". This is because the area behind the voxel whose label has been updated to "occupancy" cannot be photographed because it is shielded by a wall or the like, and it is considered that the existence or nonexistence of the object is unknown.

Next, in step Se18, the label updating unit 111 determines whether or not there is an unselected surface. As a result of this determination, if there is an unselected surface (YES in step Se18), the process returns to step Se9. On the other hand, when there is no unselected surface (NO in step Se18). It is determined whether or not there are unselected pixels (step Se19). As a result of this determination, if there are unselected pixels (YES in step Se19), the process returns to step Se1. On the other hand, if there are no unselected pixels (NO in step Se19), the first update process is terminated.
The above is the description of the first update process.

Next, the second update process executed in step Sd5 described above will be described. This second update process is a process executed when the camera is included in the rectangular parallelepiped formed by the voxel group as a whole.

FIG. 17 is a flow chart showing the second update process. The second update process shown in the figure is different from the first update process only in that steps Sf1 to Sf4 are included instead of steps Se6 to Se8. Therefore, in the following, only this difference, steps Sf1 to Sf4, will be described.

In step Sf1, the label updating unit 111 determines whether or not the number of surfaces whose parameter t is not "0" (in other words, the surfaces where the straight lines of the camera line of sight intersect) is "1". As a result of this determination, when the number of surfaces whose parameter t is not "0" is "1" (YES in step Sf1), the surface whose parameter t is not "0" is set as the exit surface (step Sf2). ). The exit surface specified here corresponds to, for example, the surface PL5 shown in FIG. On the other hand, when the number of faces whose parameter t is not "0" is larger than "1" (NO in step Sf1), that is, when the camera line-of-sight line intersects the apex of the rectangular parallelepiped, the camera line-of-sight line intersects. Let any surface to be the exit surface (step Sf3).

When the exit surface is specified, the entrance surface is specified based on the specified exit surface (step Sf4). Specifically, among the cross sections parallel to the specified exit surface and existing between the exit surface and the camera, the cross section closest to the camera is specified as the entrance surface. The entrance surface specified here corresponds to, for example, the surface PL6 shown in FIG.

After specifying the entrance surface, the processes after step Se9 are executed in the same manner as the first update process.
The above is the description of the second update process.

By executing the label update process described above, the label update unit 111 passes through the pixels and the center of the camera for a plurality of pixels of the plurality of image data constituting the image data set D1 as illustrated in FIG. Calculate the line-of-sight line. Then, among the voxels represented by the voxel data D4, the process of updating the label of the voxel intersecting with the straight line of the camera line of sight is executed.

When the label update process is completed, the voxel comparison unit 113 compares the voxel data D4 stored in the voxel data storage unit 107 with the reference voxel data D7 stored in advance in the reference voxel data storage unit 112 (step Sa6). ). Here, the reference voxel data D7 stored in advance in the reference voxel data storage unit 112 is voxel data generated for the monitored object at the normal time. The data structure of the reference voxel data D7 is the same as that of the voxel data D4 illustrated in FIG.

Specifically, the voxel comparison unit 113 compares the labels of voxels having a common index between the voxel data D4 and the reference voxel data D7. Then, among the voxels represented by the voxel data D4, a voxel having a label different from that of the voxel represented by the reference voxel data D7 is specified.

Here, in both the voxel data D4 and the reference voxel data D7, the label "empty" is given to the voxels that occupy the area where no object exists. Therefore, it is possible to compare the label with the voxel having a common index even for the voxel that occupies the area where the object does not exist. Therefore, even when a new object is added to the monitored object, the existence of the object can be detected. On the other hand, in the prior art, since the region where no point does not exist in the reference data is not a comparison target, even if a new object is added to the monitored object, the existence of the object cannot be detected.

Next, when the voxels are specified, a screen showing the voxel group represented by the voxel data D4 is generated so that the specified voxels can be identified (step Sa7). FIG. 19 is a diagram showing an example of this screen. In the screen shown in the figure, the voxel group represented by the reference voxel data D7 and the voxel group represented by the voxel data D4 are juxtaposed. Each voxel that composes these voxels is given a letter that indicates the type of label. The letter "u" stands for "unknown", the letter "e" stands for "empty", and the letter "o" stands for "occupancy". Of the characters attached to each voxel, the characters of the specified voxel are underlined. Therefore, the user who sees this screen can identify the difference between these voxel groups. In other words, it is possible to identify the difference between the monitored object during normal operation and the monitored object during inspection (in other words, the location where an abnormality occurs).

In addition, "unknown" is included in the type of label attached to each voxel. This label "unknown" indicates that the area occupied by the voxel is blocked by a wall or the like and cannot be photographed, and as a result, it is unknown whether or not an object exists in the area. Therefore, the user can identify a voxel whose existence or nonexistence of an object is unknown from the voxel group.

2. Modification Example The above embodiment may be modified as described below. One or more modifications described below may be combined with each other.

2-1. Modification 1
The label update unit 111 does not necessarily have to execute the first or second update process for all the images constituting the image data set D1. For example, the label update unit 111 may extract image data from the image data set D1 at predetermined sampling intervals and execute the first or second update process only for the extracted image data.

Similarly, the label updating unit 111 does not necessarily have to execute the first or second updating process for all the pixels constituting the image data. For example, the label updating unit 111 may extract pixels from the image data at predetermined sampling intervals and execute the first or second updating process only for the extracted pixels.

2-2. Modification 2
When the label update unit 111 determines that the number of points included in the voxel is "0" in the first or second update process, the label update unit 111 updates the label of the voxel to "empty" and "0". If it is determined that it is not, it is updated to "occupancy". Instead, when it is determined that the number of points contained in the voxel is "1" or more and less than a predetermined number, the label of the voxel is updated to "empty", and when it is determined that the number is larger than the predetermined number, " You may update to "occupancy". That is, the threshold value for determining the label type may be arbitrarily changed.

2-3. Modification 3
The method in which the voxel comparison unit 113 outputs the comparison result is not limited to the above example. The voxel comparison unit 113 may output the comparison result by another method in which the difference between the normal state and the inspection time of the monitored object can be identified.

1 ... Abnormality detection system, 101 ... Image data storage unit, 102 ... Point cloud generation unit, 103 ... Point cloud data storage unit, 104 ... Reference point cloud data storage unit, 105 ... Registration unit, 106 ... Voxelization unit, 107 ... Voxel data storage unit, 108 ... Camera attribute data storage unit, 109 ... Pixel position estimation unit, 110 ... Pixel position data storage unit, 111 ... Label update unit, 112 ... Reference voxel data storage unit, 113 ... Voxel comparison unit, CA ... camera, CB ... cube, CS1, CS2 ... cross section, IM ... image, L1, L2 ... camera line of sight, PG ... 3D point cloud, PL1-PL6 ... plane, PT1 ... apex, PT2-PT4 ... intersection, RP ... square , VX1-VX4 ... Voxels

Claims (4)

A 3D model generation method executed by a computer.
A point cloud represented in the world coordinate system based on an image group generated by continuously photographing the monitored object from a large number of points with a camera, and is a point cloud representing the three-dimensional shape of the monitored object. The first step to set and
The second step of setting the voxel group represented by the world coordinate system and occupying the three-dimensional shape region represented by the point cloud, and the second step.
For each of the pixel groups of the first image included in the image group, the third step of converting the coordinate values represented by the image coordinate system into the coordinate values represented by the world coordinate system, and
For each of the pixel groups whose coordinate values have been converted, the fourth step of identifying the pixel and the camera line-of-sight line passing through the camera, and
For each of the specified camera line-of-sight lines, when the camera line-of-sight line intersects a plurality of voxels in the voxel group, a predetermined number or more of the points constituting the point group are included in the plurality of voxels. The voxels that are closest to the camera are classified into the first category, while the other voxels are classified into the second category, and the fifth step is included.
The first category indicates that an object exists in the area occupied by the voxels classified in the first category, and the second category is in the area occupied by the voxels classified in the second category. A 3D model generation method comprising showing that it is unknown whether or not an object exists. In the fifth step, for each of the specified camera line-of-sight lines, the camera line-of-sight line is a voxel included in the voxel group, and the voxels do not include the points constituting the point cloud in excess of the predetermined number. When crossing, classify the voxels into a third category and
The 3D model generation method according to claim 1, wherein the third category indicates that no object exists in the area occupied by the voxels classified into the third category. The third step is characterized in that, for each of the pixel groups of a plurality of images included in the image group, the coordinate values represented by the image coordinate system are converted into the coordinate values represented by the world coordinate system. The 3D model generation method according to claim 1 or 2. A sixth that compares the categories of voxels with common coordinate values between the voxel group classified into each category and the other voxel groups classified into each category, and outputs information indicating the result of the comparison. The 3D model generation method according to any one of claims 1 to 3, further comprising a step.

Images