JP2016170610A - Three-dimensional model processing device and camera calibration system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve a problem that when extracting an edge in a spatial model obtained by three-dimensional measurement, there is difficulty extracting a true edge present at a boundary with an unmeasured area correctly.SOLUTION: Spatial model storage means 510 stores the three-dimensional model of an object composed of a plurality of three-dimensional coordinate data obtained by measuring a depth from a base point to the surface of the object. Edge presence addition means 531 makes the spatial model storage means 510 store an edge affirmative value indicating that an object edge exists in front data among the front data and back data whose directions from the base point are adjacent to each other and depths from the base point are different by or more than a prescribed value, each of which is three-dimensional coordinate data located at a boundary with a depth unmeasured area that is a dead corner from the base point.SELECTED DRAWING: Figure 2

Description

  The present invention uses a three-dimensional model processing device that generates a three-dimensional model from three-dimensional coordinate data obtained by measuring a predetermined target, a three-dimensional model processing device that generates an edge image of an arbitrary viewpoint from a three-dimensional model, and a three-dimensional model. The present invention relates to a camera calibration system for calibrating a camera.

  If the space where the surveillance camera is installed is three-dimensionally measured and a three-dimensional model (spatial model) of the space is generated in advance, by searching for a viewpoint that matches the captured image of the surveillance camera using the spatial model, The surveillance camera can be calibrated. In the search, a virtual model with various positions and orientations is simulated and the virtual camera position where the edge position is the best is obtained by comparing the simulated image with the image actually captured by the surveillance camera. The posture is determined as the position and posture of the monitoring camera.

  In Non-Patent Document 1 below, an arbitrary viewpoint image (generated image) is generated for a three-dimensional model of a room by a virtual camera with known camera parameters, and extracted from a real image and a generated image captured by a portable camera, respectively. It describes that the position / posture of a portable camera is estimated by performing a matching process on SIFT (Scale Invariant Feature Transform), which is a kind of edge.

Jun Nishida, et al., “Towards Position Estimation of Mobile Camera Based on Virtual Reality Environment Model”, IEICE Technical Report, vol. 111, no. 101, MVE2011-28, pp. 73-78, June 2011

  However, in a spatial model obtained by actually performing three-dimensional measurement, an unmeasured region caused by occlusion can be included. Due to this unmeasured area, there is a problem that the number of edges useful for calibration of the surveillance camera is reduced.

  For example, it is difficult to three-dimensionally measure a room that is a security target, and a blind spot caused by the shadow of a desk when viewed from a measuring device such as a distance image sensor becomes an unmeasured area. Then, if the spatial model is simulated and photographed from other than the viewpoint at the time of measurement, the edge of the desk that made the blind spot appears next to the unmeasured area on the simulated image. Here, in the non-measurement area, information on whether the desk surface measured adjacent to the non-measurement area continuously spreads or ends at the boundary with the non-measurement area cannot be obtained. For this reason, it is impossible to extract the edge of the desk which is useful for calibration of the surveillance camera even though the edge of the desk itself can be measured. This leads to a decrease in the calibration accuracy of the surveillance camera.

  The unmeasured area adjacent to the shaded desk edge on the simulated image is adjacent to the floor other than the desk edge. Therefore, if the boundary between the unmeasured area and the measurement area is regarded as an edge in order to extract the edge of the desk edge from the simulated image, a false edge is extracted from the floor portion. The extraction of false edges also leads to a decrease in the calibration accuracy of the surveillance camera.

  In addition, there is a problem that useful edges are similarly reduced in object recognition using a three-dimensional model obtained by measuring an object and a process of connecting three-dimensional models obtained by measuring a common space. It was.

  The present invention has been made in view of the above problems, and an object of the present invention is to realize a three-dimensional model processing apparatus that can provide a three-dimensional model that can extract a true edge at the boundary with an unmeasured region. Another object of the present invention is to provide a three-dimensional model processing apparatus capable of generating an edge image of an arbitrary viewpoint including a true edge at the boundary with an unmeasured region from a three-dimensional model. Furthermore, another object of the present invention is to improve the accuracy of camera calibration using a three-dimensional model.

  (1) A three-dimensional model processing device according to the present invention stores a three-dimensional model of the target composed of a plurality of three-dimensional coordinate data obtained by measuring the depth from a base point to the surface of the target. The three-dimensional coordinate data located at the boundary between the original model storage means and the depth-unmeasured area that is a blind spot from the base point, the directions from the base point are adjacent to each other, and the depth from the base point is a predetermined value Edge presence degree giving means for storing an edge affirmation value indicating that the target edge exists in the near data among the different near data and the back data, in the three-dimensional model storage means.

  (2) Another three-dimensional model processing apparatus according to the present invention, in addition to the configuration of (1) above, a virtual camera setting means for setting a virtual camera for simulating and photographing the three-dimensional model at an arbitrary position and orientation; Virtual edge image generation means for generating a virtual edge image with the projected position of the three-dimensional coordinate data having the edge positive value as an edge pixel on the imaging surface of the virtual camera.

  (3) The camera calibration system according to the present invention acquires a real image obtained by photographing the target with the three-dimensional model processing device of (2) above and a real camera, and represents the edge of the target on the real image. Real edge image generation means for extracting edge pixels to generate a real edge image; and calculating the degree of similarity between the real edge image and the virtual edge image and maximizing the degree of similarity. And an edge image collating means for determining the position and orientation of the real camera.

  (4) Another three-dimensional model processing apparatus according to the present invention stores the three-dimensional model of the object configured by a plurality of three-dimensional coordinate data obtained by measuring the depth from the base point to the surface of the object, Each of the three-dimensional coordinate data located at the boundary with the depth-unmeasured area, which is a blind spot from the base point, and the front data and the back side that are adjacent to each other in the direction from the base point and whose depths from the base point are different from each other by a predetermined value or more. The front data out of the data includes a three-dimensional model storage means to which an edge affirmation value indicating that the target edge exists, and a virtual camera that simulates and shoots the three-dimensional model at an arbitrary position and orientation. A virtual edge setting unit, and a virtual edge image having an edge pixel as a projection position of the three-dimensional coordinate data having the edge affirmative value is generated on the imaging surface of the virtual camera. Includes a virtual edge image generation unit that, the.

  (5) In the three-dimensional model processing apparatus according to (4), the three-dimensional model stored in the three-dimensional model storage means further has a luminance difference with an adjacent pixel in an image obtained by photographing the target being a predetermined value or more. The edge affirmative value can be given to the three-dimensional coordinate data corresponding to a certain pixel.

  (6) In the three-dimensional model processing apparatus of (4) and (5), the three-dimensional model stored in the three-dimensional model storage means is further a method of the surface of the object with adjacent three-dimensional coordinate data. The edge affirmative value may be assigned to three-dimensional coordinate data in which the difference between the line vectors is equal to or greater than a predetermined value.

  (7) Another camera calibration system according to the present invention is configured by a plurality of three-dimensional coordinate data representing the position of the target surface, and there is a region where the three-dimensional coordinate data is not obtained on the target surface. A three-dimensional model for storing a three-dimensional model in which the three-dimensional coordinate data is given an edge presence level indicating a high possibility that the target edge exists at the position represented by the three-dimensional coordinate data. Storage means, and calibration means for obtaining the position and orientation of the camera that captures the object using the three-dimensional coordinate data in which the edge presence degree is a predetermined value or more.

  According to the present invention, it is possible to generate a three-dimensional model that can extract an edge at a boundary with an unmeasured region. Further, according to the present invention, an edge image of an arbitrary viewpoint including an edge at a boundary with an unmeasured region can be generated from a three-dimensional model including the unmeasured region. In addition, according to the present invention, camera calibration can be performed with high accuracy using a three-dimensional model obtained by measurement.

1 is a block diagram showing a schematic configuration of a camera calibration system according to an embodiment of the present invention. It is a functional block diagram of a three-dimensional data processing device when generating a spatial model using measurement data input from a three-dimensional measurement device. It is a typical perspective view which shows the mode of the measurement of the room using a three-dimensional measuring apparatus. It is a schematic diagram which shows the example of data of a space model. It is a schematic diagram of the distance image which measured the room from the measurement position. It is the schematic diagram which looked at the space model obtained by measuring a room from the upper part. It is a schematic diagram of the space model of the room for demonstrating the example of edge presence degree. It is a functional block diagram of the outline of the three-dimensional data processing apparatus when calibrating the surveillance camera installed in the room using the room space model. It is a schematic flowchart of operation | movement of the camera calibration system at the time of a space model production | generation. It is a schematic flowchart of operation | movement of the camera calibration system at the time of a space model production | generation. It is a schematic flowchart of operation | movement of the camera calibration system at the time of camera calibration.

  Hereinafter, as an embodiment of the present invention (hereinafter referred to as an embodiment), an example of a camera calibration system that calibrates a surveillance camera using a three-dimensional model (space model) of a space in which the surveillance camera is installed will be described. This camera calibration system includes an example of a three-dimensional model processing device that generates a space model of the space from measurement data obtained by three-dimensionally measuring the space and generates an edge image of an arbitrary viewpoint from the space model. .

[Configuration of Camera Calibration System 1]
FIG. 1 is a block diagram showing a schematic configuration of the camera calibration system 1. The camera calibration system 1 includes a three-dimensional measuring device 2, a monitoring camera 3, a monitoring device 4, and a three-dimensional data processing device 5.

  The three-dimensional measuring device 2 three-dimensionally measures the room in which the surveillance camera 3 is installed to obtain a distance image, and captures the room in synchronization with it to obtain an RGB image. The distance image of the room and the RGB image Is output to the three-dimensional data processing device 5. For example, the three-dimensional measuring device 2 is an RGBD camera, and is the same in synchronization with the distance sensor of the TOF (Time Of Flight) method that measures the distance from the time required for the projected laser light to reciprocate with the target surface, and the distance sensor. This is a device combined with a color camera that captures an RGB image of the field of view. The camera may be a monochrome camera that acquires gray images. That is, the three-dimensional measuring device 2 three-dimensionally measures the depth from the measurement position (base point) to the surface of the predetermined target, and outputs the three-dimensional data of the predetermined target to the three-dimensional data processing device 5, preferably further A predetermined object is photographed from the same base point in synchronization with the three-dimensional measurement, and a two-dimensional image of the predetermined object is output to the three-dimensional data processing device 5.

  The surveillance camera 3 is a camera to be calibrated. For example, a plurality of surveillance cameras 3 are respectively installed on the ceiling of the room, the room is photographed from the ceiling, and a surveillance image and a camera ID assigned in advance are output to the surveillance device 4. When the monitoring camera 3 is calibrated, the monitoring camera 3 outputs a monitoring image and a camera ID to the three-dimensional data processing device 5.

  The monitoring device 4 receives camera parameters including the position and orientation (imaging position and orientation) of the monitoring camera 3 from the three-dimensional data processing device 5 and stores them. The shooting position and orientation are a shooting position (light spot position) and a shooting posture (optical axis direction and its rotation angle) of the monitoring camera 3 in the world coordinate system imitating real space. The process for obtaining the shooting position and orientation of the monitoring camera 3 is calibration.

  In addition, the monitoring device 4 analyzes a monitoring image from the monitoring camera 3 to track a person who moves in the room and detects suspicious behavior. When detecting a suspicious person, the monitoring device 4 determines the actual size of the change area extracted from the monitoring image based on the calibrated camera parameters, and determines whether or not the change area is a person.

  The three-dimensional data processing device 5 is configured by a computer such as a PC (Personal Computer). The three-dimensional data processing device 5 generates a spatial model using the measurement data input from the three-dimensional measurement device 2. Also, the three-dimensional data processing device 5 receives the monitoring image and the camera ID from the monitoring camera 3, calibrates the monitoring camera 3 based on the monitoring image and the space model, and the shooting position and orientation of the monitoring camera 3 as a calibration result. Are output to the monitoring device 4 together with the corresponding camera ID. Camera parameters include focal length, resolution (number of pixels in the width direction, number of pixels in the height direction), lens distortion, and the like in addition to the shooting position and orientation. Of these, there is basically no change except for the shooting position and orientation. Therefore, for camera parameters other than the shooting position and orientation, values obtained by referring to the specifications of the monitoring camera 3 or values obtained by previous measurement are stored in the three-dimensional data processing device 5 in advance.

  The three-dimensional data processing device 5 includes a measurement data input unit 50, a storage unit 51, a monitoring image input unit 52, a control unit 53, an output unit 54 and a user interface unit 55.

  The measurement data input unit 50 is a communication interface circuit that connects the three-dimensional measurement device 2 and the control unit 53, receives measurement data from the three-dimensional measurement device 2, and outputs this to the control unit 53.

  The storage unit 51 is a storage device such as a ROM (Read Only Memory) or a RAM (Random Access Memory), stores various programs and various data, and inputs / outputs these information to / from the control unit 53.

  The monitoring image input unit 52 is a communication interface circuit that connects the monitoring camera 3 and the control unit 53. The monitoring image input unit 52 receives the captured image and the camera ID from the monitoring camera 3 and outputs them to the control unit 53.

  The control unit 53 is configured using an arithmetic device such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or an MCU (Micro Control Unit). The control unit 53 reads out and executes the program from the storage unit 51 and functions as each unit described later.

  The output unit 54 is a communication interface circuit that connects the control unit 53 and the monitoring device 4, receives the camera parameters and camera ID of the monitoring camera 3 from the control unit 53, and outputs them to the monitoring device 4.

  The user interface unit 55 is a user interface device including a keyboard, a mouse, a display, and the like. For example, it is used by a user such as an installation worker, and is used for inputting various parameters such as designating a floor area in measurement data.

[Three-dimensional data processing device 5 when generating a spatial model]
FIG. 2 is a functional block diagram of the three-dimensional data processing device 5 when a spatial model is generated using measurement data input from the three-dimensional measurement device 2. The storage unit 51 functions as a space model storage unit 510 and a measurement parameter storage unit 511. The control unit 53 functions as a coordinate conversion unit 530 and an edge presence degree giving unit 531. The edge presence degree providing unit 531 includes a change point extracting unit 532 and an edge presence degree determining unit 533.

  The spatial model storage unit 510 is a three-dimensional model storage unit that stores a spatial model, that is, a three-dimensional model of a predetermined target obtained by measurement by the three-dimensional measurement apparatus 2. The spatial model includes a plurality of three-dimensional coordinate data (referred to as a point cloud) converted by the coordinate conversion means 530, luminance data associated with each three-dimensional coordinate data, and each tertiary by the edge presence degree giving means 531. It consists of the edge presence given to the original coordinate data. Here, the three-dimensional coordinate data means that the object surface exists at that position. The luminance data represents the color and luminance of the object surface at that position. The edge presence level is a value representing the high possibility that an edge exists at the position of the corresponding three-dimensional coordinate data. In this embodiment, the edge presence level is defined as 0.0 or more and 1.0 or less. To do. The edge presence will be further described later.

  The measurement parameter storage unit 511 stores measurement parameters including the position and orientation (measurement position and orientation) at the time of measurement of the three-dimensional measurement apparatus 2. The measurement position / orientation is represented by a measurement position (light spot position) and a measurement attitude (optical axis direction and its rotation angle) of the three-dimensional measurement apparatus 2 in a world coordinate system imitating real space. The measurement position / orientation is calculated by the coordinate conversion means 530. Alternatively, separately measured measurement positions and orientations may be input in advance from the user interface unit 55 and stored.

  The measurement parameters include a focal length, resolution (number of pixels in the width direction, number of pixels in the height direction) and the like in addition to the measurement position and orientation. Since there is basically no change except for the measurement position and orientation, a value obtained by referring to the specification of the three-dimensional measurement apparatus 2 or a value obtained by a previous measurement is stored in advance. The angle of view can be derived from the focal length and the resolution. In addition, the projection plane at the time of measurement can be derived from the measurement parameters.

  The coordinate conversion unit 530 performs coordinate conversion of the measurement data input from the three-dimensional measurement apparatus 2 via the measurement data input unit 50 into three-dimensional coordinate data of the world coordinate system, and stores it in the space model storage unit 510. Also, the coordinate conversion unit 530 calculates the measurement position and orientation of the three-dimensional measurement apparatus 2 at the time of measurement from the measurement data, and stores the calculated measurement position and orientation in the measurement parameter storage unit 511.

  The outline of the coordinate conversion process is as follows: (a) calculating a three-dimensional point group in the local coordinate system with the measurement position as the origin from the distance image; and (b) setting the reference of the world coordinate system in the local coordinate system. The measurement position and orientation RT in the coordinate system is estimated, and (c) the 3D point group in the local coordinate system is converted into 3D coordinate data in the world coordinate system using RT. Each procedure will be described below.

In step (a), the coordinate system of the distance image (x, y, d) in which the pixel value of each pixel (x, y), that is, the depth value is d, is represented by the light spot ( It is converted into a three-dimensional point group (Xc, Yc, Zc) in the local coordinate system with the measurement position) as the origin.

  However, the center pixel position of the distance image is (cx, cy). f is a focal length of the three-dimensional measuring apparatus 2.

  In step (b), in order to calculate the measurement position / orientation RT, a point on the floor surface of the three-dimensional point group of the local coordinate system is selected and input using the user interface unit 55, and measurement is performed based on the selection input. Calculate the coordinate value of the floor directly under the position. Thereby, for example, the measurement position and orientation RT (3 × 4 matrix) in the world coordinate system with the floor surface as the XY plane, the height direction as the Z axis, and the point immediately below the measurement position on the floor surface as the origin can be calculated. .

In step (c), using the RT obtained in step (b), the three-dimensional point group in the local coordinate system is converted into three-dimensional coordinate data (X, Y, Z) in the world coordinate system according to the following equation.

  Further, the coordinate conversion unit 530 stores each pixel value of the RGB image as luminance data in the spatial model storage unit 510 in association with the three-dimensional coordinate data of the conversion destination of the pixel of the distance image corresponding to each pixel.

  FIG. 3 is a schematic perspective view showing a state of measurement of the room 600 using the three-dimensional measuring apparatus 2. The room 600 is a room to be guarded, for example, and the monitoring camera 3 to be calibrated is installed on the ceiling. The room 600 includes a floor 601 and a desk 602, and a pattern 603 is formed on the floor 601. Incidentally, as will be described later, the camera calibration system 1 uses an edge such as an edge of the desk 602 as a reference for calibration of the surveillance camera 3.

  The installation worker of the monitoring camera measures the room 600 three-dimensionally using the three-dimensional measuring device 2. The three-dimensional measurement apparatus 2 outputs a distance image in which each pixel represents a depth from the measurement position 650 to each of a plurality of points on the surface of the floor 601 or the desk 602 at the measurement position and posture represented by the measurement position 650 and the measurement posture 651. To do. Further, the three-dimensional measurement apparatus 2 outputs an RGB image obtained by photographing the room 600 at the same measurement position and orientation. Here, the three-dimensional measuring apparatus 2 basically measures the room 600 at a measurement position and orientation different from the imaging position and orientation of the monitoring camera 3 represented by the imaging position 652 and the imaging orientation 653.

  The area represented by the distance image and the RGB image output from the three-dimensional measuring apparatus 2 is an area displayed in white in FIG. This area is called a measurement area. The measurement area cannot cover the entire room 600, and an unmeasured area occurs due to occlusion. In FIG. 3, the unmeasured area shaded is an area caused by occlusion by the desk 602, and is a blind spot from the base point (measurement position 650). In FIG. 3, the region outside the angle of view displayed with diagonal lines is the region outside the angle of view of the three-dimensional measurement apparatus 2.

  FIG. 4 is a schematic diagram showing an example of data of a spatial model, and is stored in the spatial model storage unit 510 in a table format in which, for example, three-dimensional coordinate data, luminance data, and edge presence are associated with each other. Incidentally, all the data shown in FIG. 4 is data of the measurement region. The first three-dimensional coordinate data has the surface of the object at coordinates (X1, Y1, Z1) in the spatial model, the color is (R1, G1, B1) in the RGB color system, and the value of edge presence “1.0” indicates that an edge such as the edge of the desk 602 exists at the coordinates. Similarly, the second three-dimensional coordinate data indicates that an edge of the object surface exists at coordinates (X2, Y2, Z2) in the spatial model, and the color thereof is (R2, G2, B2). The third three-dimensional coordinate data has the surface of the object at coordinates (X3, Y3, Z3) in the spatial model, the color is (R3, G3, B3) in the RGB color system, and the edge presence value “0.0” indicates that the possibility that an edge exists there is small.

  Now, the point where the depth changes in the distance image is an edge useful for camera calibration or the like that occurs at the edge of the object. However, if the depth change point is three-dimensional coordinate data located at the boundary with the unmeasured area in the spatial model, there arises a problem that it cannot be used even if an edge actually exists. Moreover, since the occlusion is likely to occur near the edge of the object, useful edges are likely to be lost. For example, the monitoring image captured by the monitoring camera 3 of FIG. 3 includes an unmeasured area. When a monitoring image captured by the monitoring camera 3 is simulated using a space model obtained by measuring the room 600, the edge of the desk 602 and the plane of the floor 601 are mixed at the boundary of the unmeasured area. Both useful edges appearing on the edge of the desk 602 must be discarded unless the 3D coordinate data of the edge of the desk 602 and the 3D coordinate data of the plane of the floor 601 that are both located at the boundary with the unmeasured area can be identified. In the present invention, the edge presence degree giving means 531 identifies the three-dimensional coordinate data in which the useful edge is present, and the edge positive value indicating that the edge exists is assigned to the identified three-dimensional coordinate data. Generate a 3D model that can use edges.

  The edge presence degree giving unit 531 gives the edge presence degree to a plurality of three-dimensional coordinate data constituting the space model. As described above, the edge presence level is a value indicating the high possibility that an edge exists at the position of the corresponding three-dimensional coordinate data, and is used to identify whether the three-dimensional coordinate data is the position of the edge. Information.

  The edge on the target surface is the position indicated by the 3D coordinate data where the depth from the specific viewpoint changes greatly due to the unevenness of the object, the position indicated by the 3D coordinate data where the normal direction changes greatly due to the unevenness of the object, the object surface Appear at the position indicated by the three-dimensional coordinate data where the luminance varies greatly depending on the pattern of the pattern. The edge presence degree assigning means 531 assigns an edge affirmative value to these three-dimensional coordinate data by the change point extracting means 532 and the edge presence degree determining means 533 and stores them in the space model storage means 510.

  The change point extraction unit 532 extracts depth change points, normal direction change points, and luminance change points from a plurality of three-dimensional coordinate data constituting the space model, and outputs these to the edge presence degree determination unit 533.

  First, extraction of depth change points will be described. The change point extraction unit 532 calculates a difference in depth between adjacent pixels on the distance image measured from the measurement position, compares the difference with a predetermined threshold value Td, and calculates a difference in depth greater than or equal to the threshold value Td. Three-dimensional coordinate data corresponding to the selected pixel is extracted as a depth change point. The three-dimensional coordinate data corresponding to adjacent pixels on the distance image is three-dimensional coordinate data in which the directions (gaze direction) from the base point (measurement position) are adjacent to each other in the spatial model. For example, a predetermined edge operator is applied to the distance image to calculate the depth edge strength of each pixel, and 3D coordinate data corresponding to a pixel whose depth edge strength is equal to or greater than a predetermined threshold Te is extracted as a depth change point. do it. Te is set to a range in which a difference in depth equal to or greater than Td is extracted and not impaired.

  As the edge operator, a Sobel operator can be used, but other known operators such as a Laplacian filter may be used.

  In the present embodiment, the distance image input from the three-dimensional measurement apparatus 2 is used for extracting the depth change point. On the other hand, instead of directly using the distance image from the three-dimensional measurement apparatus 2, the distance image may be reproduced from the spatial model and the measurement parameter, and the depth change point may be extracted from the reproduced distance image. Specifically, the distance image can be reproduced by calculating the depth from the measurement position for each of a plurality of three-dimensional coordinate data constituting the space model and projecting the calculated depth onto the projection plane represented by the measurement parameter. Note that the angle of view of the distance image to be reproduced is matched to the angle of view of the three-dimensional measuring apparatus 2. At the time of measurement, the distance image is not stored in the storage unit 51 and only the process of the coordinate conversion unit 530 is performed, and the process of the edge presence degree providing unit 531 can be executed at another time after the measurement. This configuration is effective when the capacity of the storage unit 51 and the processing capability of the control unit 53 are low.

  Next, extraction of normal direction change points will be described. The change point extraction means 532 extracts, as a normal direction change point, three-dimensional coordinate data indicating a change in the normal direction of the target surface that is greater than or equal to a predetermined threshold value Tn in the spatial model representing the shape of the target surface. First, the change point extraction unit 532 calculates a normal vector using each pixel and the surrounding three-dimensional coordinate data. Specifically, the three-dimensional coordinate data of the pixel of interest is P0, the three-dimensional coordinate data of a point around the pixel of interest (for example, the right neighbor) is P1, and the three-dimensional data of another point around the pixel of interest (eg, the lower neighbor) The coordinate data is P2. The vectors from the three-dimensional position of the target pixel to the peripheral points are set as Vec1 = P1-P0 and Vec2 = P2-P0, respectively. The normal vector can be calculated from the outer product of Vec1 and Vec2, and the normal vector Norm = Vec1 × Vec2. Next, the change point extraction means 532 calculates the angle formed by the normal vector of each pixel and the normal vectors of the surrounding pixels, and extracts pixels whose sum is equal to or greater than the threshold value Tn as the normal direction change point. To do. The normal direction change point is extracted at the uneven portion of the object existing in the space.

  Next, extraction of luminance change points will be described. The change point extraction unit 532 extracts a luminance change point where the luminance difference between adjacent pixels is equal to or greater than a predetermined threshold value Ti from an RGB image obtained by photographing the space represented by the space model at the measurement position and orientation. In the present embodiment, edge strength is used as the luminance difference. The change point extraction means 532 applies a predetermined edge operator to each pixel of the RGB image described above to calculate the edge strength between the pixel and its surrounding pixels, and changes the luminance of the pixel whose edge strength is equal to or greater than the threshold value Ti. Extract as a point. The luminance change point is extracted by a pattern portion of an object existing in the space.

  In the present embodiment, the RGB image input from the three-dimensional measuring device 2 is used for extracting the luminance change point. On the other hand, instead of directly using the RGB image from the three-dimensional measurement apparatus 2, the RGB image may be reproduced from the spatial model and the measurement parameter, and the luminance change point may be extracted from the reproduced RGB image. Specifically, an RGB image can be reproduced by a rendering process in which luminance data corresponding to a plurality of three-dimensional coordinate data constituting a space model is projected onto a projection plane represented by a measurement parameter. Note that the angle of view of the RGB image to be reproduced is matched to the angle of view of the three-dimensional measuring apparatus 2. At the time of measurement, only the process of the coordinate conversion unit 530 is performed without storing the RGB image in the storage unit 51, and the process of the edge presence degree providing unit 531 can be executed at another time after the measurement. This configuration is effective when the capacity of the storage unit 51 and the processing capability of the control unit 53 are low.

  The edge presence degree determination means 533 determines the edge presence degree for each of the depth change point, the normal direction change point, and the luminance change point extracted by the change point extraction means 532 and stores it in the space model storage means 510.

  The edge presence degree determination means 533 gives an edge affirmative value as an edge presence degree indicating that an edge is present to the near side three-dimensional coordinate data (front side data) of the three-dimensional coordinate data of the depth change point. No edge affirmative value is given to the three-dimensional coordinate data (back data). This handling will be described. From the depth gap between the front data and the back data, other 3D coordinate data cannot be obtained between the 3D coordinate data, and the target surface shape is not measured, that is, it is an unmeasured area. It is presumed that the near data and the back data are located at the boundary of the unmeasured area. However, the near-side data is extracted at the tip position of the convex portion of the object as viewed from the three-dimensional measuring apparatus 2, and can be regarded as the position of the edge. On the other hand, for the back data, since it is impossible to deny the possibility that the target surface where it is located and the target surface in the unmeasured area adjacent to it are connected smoothly, it is not appropriate to set the position of the edge. Therefore, only the near data is set as the edge position as described above. Edge presence degree (edge positive value) indicating that an edge exists in 3D coordinate data in which the edge of the external part of the object actually exists among the 3D coordinate data located at the boundary with the unmeasured area. The edge presence degree (edge negative value) indicating that the edge does not exist can be assigned to the three-dimensional coordinate data that is unlikely to actually have an edge.

  In this embodiment, the edge positive value is 1.0 and the edge negative value is 0.0. That is, the edge presence degree determination means 533 determines the edge presence degree for the three-dimensional coordinate data on the near side of the three-dimensional coordinate data of the depth change point as 1.0, and determines the edge presence degree for the three-dimensional coordinate data on the back side. It is determined as 0.0, and the determined edge presence degree is stored in the space model storage unit 510. For example, the edge presence degree determination unit 533 pays attention to each depth change point, and the depth value adjacent to the depth change point measured at the measurement position / posture is larger than the threshold value Td, that is, the back side. If the 3D coordinate data is present, the depth change point of interest is determined to be the 3D coordinate data on the near side, and the edge presence is determined to be 1.0. Determined to be 0.

  Next, the edge presence degree determination means 533 gives an edge positive value to the three-dimensional coordinate data corresponding to the normal direction change point. Specifically, the edge presence degree determination means 533 determines the edge presence degree for the three-dimensional coordinate data that is not the depth change point among the normal direction change points extracted by the change point extraction means 532 as 1.0. The edge presence degree is stored in the space model storage unit 510. Thereby, an edge affirmative value can be given to uneven portions other than the outer shape of the object.

  Further, the edge presence determination unit 533 gives an edge affirmative value to the three-dimensional coordinate data at the position corresponding to the luminance change point for each luminance change point extracted by the change point extraction unit 532. Specifically, the edge presence degree determination unit 533 determines the edge presence degree for the three-dimensional coordinate data that is not the depth change point among the luminance change points as 1.0, and the determined edge presence degree is the space model storage unit 510. Remember me. Thereby, an edge affirmative value can be given to a pattern portion or the like.

  Further, the edge presence degree determination means 533 sets the edge presence degree to three-dimensional coordinate data other than the above, that is, the edge existence degree for the three-dimensional coordinate data that is not the depth change point, the normal direction change point, and the brightness change point. .0, and the determined edge presence degree is stored in the space model storage means 510.

  Here, an example of giving the edge presence level to the depth change point will be described with reference to FIGS. 5 and 6. FIG. 5 is a schematic diagram of a distance image 750 obtained by measuring the room 600 from the measurement position 650. FIG. 5A shows the entire distance image 750, and FIG. 5B shows a partial region 751 of the distance image 750. Is shown. FIG. 6 is a schematic view of a space model 800 obtained by measuring the room 600 as viewed from above. In FIG. 6, the shaded area is an unmeasured area 803, the area 804 is a three-dimensional coordinate data group representing the plane of the floor 601, and the area 805 is a three-dimensional coordinate data group representing the upper surface of the desk 602.

  FIG. 5A is a distance image acquired by the three-dimensional measurement apparatus 2 with the positional relationship as shown in FIG. 3, and a desk 602 and a floor 601 exist within the angle of view. The partial area 751 is an area of 5 × 5 pixels including the desk edge 602, and a small square represents a pixel, and a numerical value in the square represents a depth value. The unit of the depth value is centimeter (cm). The threshold value Td is 10 cm, and in the region 751, 10 pixels having high depth edge strength shown as the region 752 and the region 753 are extracted as depth change points. The three-dimensional coordinate data 801 in FIG. 6 corresponding to the pixel 754 in the region 752 represents one point of the edge portion on the side surface of the desk 602 and is the actual three-dimensional coordinate data, but the pixel 755 in the region 753 corresponds to this. The three-dimensional coordinate data 802 in FIG. 6 corresponding to is one-dimensional coordinate data representing one point on the plane of the floor 601 and having no actual edge. These three-dimensional coordinate data 801 and 802 are both located at the boundary with the unmeasured area 803.

  Comparing the depth value of the pixel 754 with the adjacent pixel on the distance image, there is a pixel that is 40 cm larger than the threshold value Td. Therefore, the pixel 754 is determined as the depth change point in the foreground, and corresponds to the pixel 754. An edge positive value 1.0 is given to the three-dimensional coordinate data 801 of the edge portion of the desk 602.

  On the other hand, when comparing the depth value of the pixel 755 with the adjacent pixel on the distance image, there is no pixel having a depth value larger than that of the pixel 755, so that the pixel 755 is determined to be the depth change point in the back and corresponds to the pixel 755. An edge negative value 0.0 is given to the three-dimensional coordinate data 802 of the floor 601 to be processed.

  In this way, among the three-dimensional coordinate data located at the boundary with the non-measurement region 803, the three-dimensional coordinate data in which the edge is real can be given to the three-dimensional coordinate data in which the edge is real, and the possibility that the edge is real is low. An edge negative value can be given to the data.

  FIG. 7 is a schematic diagram of a space model of the room 600 for explaining an example of the edge presence level. In FIG. 7, the solid line parts (straight line P1-P2, P1-P3, P1-P5, P2-P4, P3-P4, P3-P6, P4-P7, P5-P6, P6-P7 part and straight line group S). An edge positive value 1.0 is assigned to the three-dimensional coordinate data, and an edge negative value 0.0 is assigned to the other three-dimensional coordinate data.

  Among these solid line portions, the edge positive value 1.0 is given to the three-dimensional coordinate data of the straight lines P1-P2, P1-P5, P2-P4 and P4-P7 as the depth change point in the foreground. These three-dimensional coordinate data determined as the depth change point in the foreground are located at the boundary with the unmeasured area (shaded portion) and are edges that could not be used for camera calibration or the like in the prior art. An edge affirmative value is given and it can be used for camera calibration and the like.

  The edge positive value 1.0 is given to the three-dimensional coordinate data of the straight lines P1-P3, P3-P4, P3-P6, P5-P6 and P6-P7 in the solid line as the normal direction change point or the luminance change point. In addition, an edge positive value 1.0 is given to the straight line group S in the solid line portion as a luminance change point.

[3D data processor 5 during camera calibration]
FIG. 8 is a schematic functional block diagram of the three-dimensional data processing apparatus 5 when the surveillance camera 3 installed in the room 600 is calibrated using the space model of the room 600.

  The storage unit 51 functions as the space model storage unit 510, and the control unit 53 functions as the real edge image generation unit 535, the virtual camera setting unit 536, the virtual edge image generation unit 537, the edge image collation unit 538, and the like.

  The real edge image generation means 535 extracts an edge from each monitoring image (real image) input from the monitoring camera 3 (real camera) to generate an edge image, and the generated edge image together with the corresponding camera ID is used as an edge image. The result is output to the verification unit 538. In order to distinguish from an edge image generated by a virtual edge image generation means 537 described later, the edge image generated here is called a real edge image.

  The real edge image generation means 535 has a pixel value “1” for the pixel corresponding to the luminance change point where the luminance difference with the surrounding pixels in the monitoring image is equal to or greater than a predetermined threshold value Ti, and a pixel value “0” for the other pixels. "" Is set to generate an actual edge image. In the present embodiment, edge strength is used as the luminance difference. The real edge image generation means 535 calculates edge strength by applying a predetermined edge operator such as a Sobel operator to each pixel of the monitoring image.

  The virtual camera setting unit 536 sets a plurality of virtual cameras for simulating and capturing the room 600 at an arbitrary position and orientation in the space model, and outputs the set camera parameters of the virtual camera to the virtual edge image generation unit 537. The position and orientation included in the camera parameters of the virtual camera are candidates for the photographing position and orientation of the surveillance camera 3 to be calibrated. Of the camera parameters, values of the surveillance camera 3 stored in advance are used for the focal length, resolution, lens distortion, etc. other than the position and orientation.

  The virtual camera setting means 536 moves the position of the virtual camera in the X, Y, and Z directions at a predetermined minute interval from the origin in the XYZ coordinate system of the space model, and also the attitude of the virtual camera (pitch angle, yaw angle, roll The angle may be changed at a predetermined minute interval to set a plurality of virtual cameras. However, the setting range based on the installation specifications of the monitoring camera 3 is input in advance from the user interface unit 55, By setting a virtual camera within the range, the amount of camera calibration processing can be reduced. For example, if the installation specification is a specification in which the ceiling is installed vertically downward, the range of movement of the virtual camera in the Z direction and the variation range of the pitch angle can be significantly narrowed by inputting the ceiling height in advance. Also, the X coordinate, Y coordinate, Z coordinate, pitch angle, yaw angle, and roll angle defined in the installation specifications are input in advance, and within a predetermined installation error range centered on each input value at a fixed minute interval. A plurality of virtual cameras may be set, or a plurality of virtual cameras may be set at a minute interval that spreads away from each input value.

  The virtual edge image generation unit 537 generates an edge image by projecting the space model of the room 600 onto the imaging plane represented by each camera parameter set by the virtual camera setting unit 536, and the generated edge image is sent to the edge image matching unit 538. Output. The edge image generated here is called a virtual edge image.

  Specifically, among the three-dimensional coordinate data constituting the space model, three-dimensional coordinate data whose edge presence is an edge positive value “1.0” is projected onto the imaging surface, and the edge presence is an edge negative value “0”. By not projecting the 3D coordinate data of .0 ", a virtual edge image is generated with the projected position of the 3D coordinate data having an edge positive value as an edge pixel on the imaging surface of the virtual camera. At this time, only the three-dimensional coordinate data of the front surface as viewed from the shooting position of the virtual camera is projected by a hidden surface elimination method such as a Z buffer method or a Z sort method.

  By defining the edge presence at the depth change point of the distance image as described above, the virtual edge image includes the edge that exists at the boundary of the unmeasured area generated in the spatial model corresponding to the depth change point. Can be made.

  Here, in the monitoring image obtained when the monitoring camera 3 exists at the virtual viewpoint even if the edge actually exists in the three-dimensional coordinate data whose distance from the virtual viewpoint is farther than the distance from the measurement viewpoint. In some cases, the resolution of the pixel position corresponding to the three-dimensional coordinate data is lowered and the edge is not extracted. Therefore, an edge image is generated from a projection image (rendered image) obtained by projecting the luminance data associated with the three-dimensional coordinate data constituting the spatial model onto the imaging plane represented by each virtual viewpoint, and the virtual edge image is generated as the generated edge. You may replace with the logical product image of an image and a virtual edge image. In particular, at this time, for the three-dimensional coordinate data that is far from the virtual viewpoint, a plurality of three-dimensional coordinate data is projected onto one pixel of the rendered image. The virtual edge image generation means 537 sets the average value of the luminance data associated with the plurality of three-dimensional coordinate data to the projection destination pixel. An outlier (for example, RGB values (255, 255, 255)) may be given to the unmeasured area projected on the rendered image. In this way, an edge that does not actually exist can be extracted from the boundary with the unmeasured area in the rendered image, but this edge does not cause a problem because it is deleted by a logical product operation with the virtual edge image. By averaging the luminance data, non-extraction of edges due to a decrease in resolution can be reproduced, and the matching accuracy between the real edge image and the virtual edge image can be improved.

  The edge image collating unit 538 calculates the similarity between the real edge image generated by the real edge image generating unit 535 and the virtual edge image generated by the virtual edge image generating unit 537 for each camera parameter of the virtual camera, and converts the similarity into the real edge image. The position and orientation of the virtual camera that gives a similar virtual edge image is determined as the position and orientation of the monitoring camera 3. For example, the edge image collating unit 538 determines the position and orientation of the virtual camera having the highest similarity as the position and orientation of the monitoring camera 3.

  Here, the similarity is defined to be a value corresponding to the degree of matching of the edge positions. For example, the similarity may be a value obtained by dividing the number of coincidence of edge positions by the number of edge pixels of the virtual edge image or the sum of the number of edge pixels of the real edge image and the number of edge pixels of the virtual edge image. Also, for example, the distance from each edge position of the virtual edge image to the nearest edge position of the real edge image or the distance from each edge position of the real edge image to the nearest edge position of the virtual edge image is accumulated, and the accumulated value A value obtained by dividing the value corresponding to the reciprocal number by the number of edge pixels of the virtual edge image or the sum of the number of edge pixels of the real edge image and the number of edge pixels of the virtual edge image may be used as the similarity. Further, the similarity may be corrected by weighting and adding the matching degree of the luminance data to the similarity. In this case, the normalized correlation value of the pixel value between the corresponding pixels between the monitoring image and the rendered image can be set as the degree of coincidence of the luminance data.

  Incidentally, the three-dimensional model processing apparatus according to the present invention includes a spatial model storage unit 510, an edge presence degree assigning unit 531, a virtual camera setting unit 536, and a virtual edge image generation unit 537. The camera calibration system according to the present invention includes: The three-dimensional model processing apparatus is configured to include an actual edge image generating unit 535 and an edge image collating unit 538.

[Operation of camera calibration system 1 when generating spatial model]
FIG. 9 and FIG. 10 are schematic flowcharts of the operation of the camera calibration system 1 when generating a spatial model. When the three-dimensional measurement device 2 measures the room 600, measurement data is input from the three-dimensional measurement device 2 to the three-dimensional data processing device 5. The three-dimensional data processing device 5 performs a series of processes shown in FIGS. 9 and 10 on the input measurement data, and generates a space model of the room 600.

  The measurement data input unit 50 acquires measurement data, that is, a distance image measured at the measurement position and orientation and an RGB image captured at the measurement position and orientation from the three-dimensional measurement apparatus 2, and inputs these to the control unit 53. (Step S10).

  The control unit 53 operates as the coordinate conversion unit 530, and the coordinate conversion unit 530 calculates the measurement position and orientation from the distance image acquired in step S10 and stores the measurement position and orientation in the storage unit 51 that functions as the measurement parameter storage unit 511 ( Step S11).

  The coordinate conversion unit 530 uses the measurement position and orientation calculated in step S11 to convert each pixel of the distance image acquired in step S10 into three-dimensional coordinate data, and stores it in the storage unit 51 that functions as the space model storage unit 510. Store (step S12). Further, the correspondence relationship between the pixels of the distance image and the three-dimensional coordinate data is also temporarily stored in the space model storage unit 510 for subsequent processing. In addition, the coordinate conversion unit 530 associates each pixel value of the RGB image acquired in step S10 as luminance data with the three-dimensional coordinate data that is the conversion destination of the pixel of the corresponding distance image, as a spatial model storage unit 510. Remember me.

  Further, the control unit 53 operates as the edge presence degree giving unit 531 and performs the processes of steps S13 to S25. First, the change point extraction unit 532 of the edge presence degree assigning unit 531 calculates the edge strength at each pixel of the distance image acquired in step S10, and sets a pixel whose edge strength is equal to or greater than the threshold Te to a depth change point. (Step S13).

  Next, the edge presence determination means 533 of the edge presence degree giving means 531 sequentially sets the depth change points extracted by the change point extraction means 532 as the attention change points, and gives the edge presence degree loops of steps S14 to S18. Execute the process.

  The edge presence degree determination unit 533 calculates the depth difference by subtracting the depth value of the target change point from the depth value of each depth change point extracted in the eight neighboring pixels of the target change point on the distance image, and calculates each difference The depth difference is compared with a threshold value Td (step S15). If at least one of the eight neighboring points has a depth difference equal to or greater than the threshold value Td (in the case of YES in S15), the depth change point of interest has an adjacent depth change greater than or equal to the threshold value Td. There is a point, and therefore the 3D coordinate data of interest is considered to be the 3D coordinate data on the near side of the 3D coordinate data of the change point where the depth from the measurement position shows a change of a predetermined value or more. The edge model 1.0 is stored in the space model storage unit 510 in association with the existing three-dimensional coordinate data (step S16).

  By providing the edge presence level, it is possible to identify that there is an edge in the three-dimensional coordinate data of the outer portion of the object observed at the measurement position and orientation, including those located at the boundary of the unmeasured region. By using a spatial model to which an edge presence degree of 1.0 is used, it is possible to effectively use an edge located at the boundary of an unmeasured area.

  On the other hand, if none of the eight neighboring points has a depth difference equal to or greater than the threshold value Td (NO in S15), the three-dimensional coordinate data of interest has a depth from the measurement position greater than or equal to a predetermined value. The edge presence degree 0.0 is stored in the spatial model storage unit 510 in association with the three-dimensional coordinate data of interest, as the three-dimensional coordinate data on the back side among the three-dimensional coordinate data of the change point indicating the change of (Step S17).

  By providing the edge presence degree, it is possible to identify the three-dimensional coordinate data that is unlikely to have an edge among the three-dimensional coordinate data located at the boundary of the unmeasured region.

  The edge presence degree determining means 533 checks whether all depth change points have been processed (step S18), and if there is an unprocessed item (NO in S18), the process returns to step S14, and the next depth change point is determined. Perform the process. On the other hand, if all depth change points have been processed, the process proceeds to step S19 (YES in S18).

  The change point extraction unit 532 calculates the normal direction of each pixel of the distance image acquired in step S10 and calculates the difference between the normal direction of each pixel and the normal direction of the surrounding pixels. A pixel having surrounding pixels equal to or greater than the threshold value Tn is extracted as a normal direction change point (step S19).

  Further, the change point extraction means 532 calculates the edge intensity at each pixel of the two-dimensional image acquired in step S10, and extracts a pixel whose edge intensity is equal to or greater than the threshold value Ti as a luminance change point (step S20). .

  The edge presence degree determination unit 533 executes the loop processing of steps S21 to S24 in which the normal direction change point and the luminance change point extracted by the change point extraction unit 532 are sequentially set as the attention change point to determine the edge presence degree. To do. Incidentally, in the loop processing, the normal direction change point and the luminance change point corresponding to the same three-dimensional coordinate data may be processed together.

  The edge presence degree determining means 533 checks whether or not the attention change point has already been processed as a depth change point (step S22). If not processed as a depth change point (NO in S22), The edge presence degree 1.0 is stored in the space model storage unit 510 in association with the three-dimensional coordinate data of interest as the attention change point (step S23).

  On the other hand, if the target change point has already been processed as a depth change point (YES in S22), the edge presence given in steps S16 and S17 is maintained by skipping step S23.

  The edge presence determination means 533 checks whether all normal direction change points and luminance change points have been processed (step S24), and if there is any unprocessed item (NO in S24), the process returns to step S21. Then, the next change point is processed. On the other hand, if all normal direction change points and luminance change points have been processed, the process proceeds to step S25 (in the case of NO in S24).

  The edge presence degree determination means 533 extracts the three-dimensional coordinate data to which the edge presence degree is not yet given in the spatial model stored in the spatial model storage means 510, and adds the edge presence degree 0. 0 to the extracted three-dimensional coordinate data. 0 is associated and stored in the space model storage unit 510 (step S25).

  When measurement data is acquired at a plurality of measurement positions 650 and measurement postures 651, the above-described processing can be performed for each measurement data, thereby reducing an unmeasured area and extracting more edges. Is possible.

[Operation of camera calibration system 1 during camera calibration]
FIG. 11 is a schematic flowchart of the operation of the camera calibration system 1 during camera calibration. When the monitoring camera 3 installed in the room 600 captures an image of the room 600, a monitoring image is input from the monitoring camera 3 to the three-dimensional data processing device 5. The three-dimensional data processing device 5 performs a series of processes shown in FIG. 11 on the input monitoring image and calibrates the monitoring camera 3.

  The monitoring image input unit 52 acquires a monitoring image from the monitoring camera 3 and inputs it to the control unit 53 (step S50). The control unit 53 operates as the real edge image generation unit 535, and generates an actual edge image by applying an edge operator to each pixel of the monitoring image acquired in Step S50 (Step S51).

  Next, the control unit 53 operates as the virtual camera setting unit 536, and sets a plurality of virtual cameras having different positions and orientations in the coordinate system of the space model (step S52).

  Then, the control unit 53 operates as the virtual edge image generation unit 537, and generates a virtual edge image on the imaging surface of each virtual camera set in step S52 (step S53). Specifically, the virtual edge image generation unit 537 reads the space model of the room 600 from the space model storage unit 510, and the edge presence degree is 1.0 on the photographing surface represented by the camera parameter of each virtual camera set in step S52. Is projected to generate a virtual edge image. In addition, luminance data is rendered on the projection plane, an edge operator is applied to each pixel of the rendered image to generate an edge image, and the virtual edge image is updated to a logical product image with the generated edge image.

  The control unit 53 operates as the edge image collating unit 538, and calculates the similarity between the real edge image generated in step S51 and the virtual edge image for each virtual camera generated in step S53 (step S54). The edge image collating unit 538 detects the position and orientation of the virtual camera for which the maximum similarity is calculated based on the similarity for each virtual camera calculated in step S54, and monitors the detected position and orientation of the virtual camera. 3 is input to the output unit 54 together with the camera ID of the monitoring camera 3 (step S55).

  The position and orientation of the monitoring camera 3 determined here is a highly accurate estimation result obtained using an edge located at the boundary of the unmeasured area in the spatial model.

  The output unit 54 inputs the position and orientation of the monitoring camera 3 and the camera ID to the monitoring device 4. The monitoring device 4 stores the input position and orientation of the monitoring camera 3 and uses them in subsequent monitoring processing.

[Modification]
(1) In the above embodiment, an example has been described in which the edge presence degree giving unit 531 gives either the edge negative value “0.0” or the edge positive value “1.0” as the edge presence degree. Two or more types of edge presence may be given as edge positive values. In that case, the edge presence degree giving means 531 is, for example, 0.5 for the depth change point, 0.8 for the normal direction change point, 1.0 for the luminance change point, and the normal line. An edge presence level of 0.8 is assigned to a change point that is a direction change point and a luminance change point. Then, the edge image collating unit 538 calculates the similarity with the actual edge image by weighting the edge position of the virtual edge image with the edge presence degree. For example, on a desk with a rounded edge, the position of the edge of the desk detected as the depth change point and the normal direction change point is slightly shifted due to the difference between the measurement position and the shooting position and posture. On the other hand, such a shift is unlikely to occur in the edge on the plane detected as the luminance change point. Thus, by using a plurality of types of edge affirmative values, more practical edge information can be given to the three-dimensional model.

  (2) In the above embodiment, a spatial model obtained by measuring a predetermined space is exemplified as the three-dimensional model. However, the three-dimensional model may be an object model obtained by measuring a predetermined object such as a fixture.

  (3) In the above embodiment, the point cloud format three-dimensional model in which the three-dimensional coordinate data represents a point in the three-dimensional space is exemplified, but the mesh model format three-dimensional model including a set of vertex coordinates of the object plane is used. There may be. In this case, for example, an arbitrary vertex coordinate pair is set as three-dimensional coordinate data (that is, three-dimensional coordinate data in units of straight lines), and the edge presence degree assigning unit 531 assigns an edge presence degree to the three-dimensional coordinate data. .

  (4) In the above embodiment, an example in which there is one monitoring camera 3 has been described. However, two or more monitoring cameras may be used, and in this case, the same processing as in the above example may be repeated for each monitoring camera.

  (5) In the above embodiment, the camera calibration system that calibrates the camera in which the photographing position and orientation is fixed is exemplified. However, the present invention is a camera that performs pan / tilt operation, a camera installed on a moving body such as a robot, The present invention can also be applied to a camera calibration system that calibrates a camera that changes its photographing position and orientation, such as a camera held by a person. That is, when these cameras are calibrated, it is sufficient to calibrate each time a captured image after the photographing position and orientation is changed is input.

  (6) In the above-described embodiment, an example in which a virtual edge image is generated by projecting a three-dimensional model to which edge presence is given is shown. However, a spatial model to which edge presence is given can be generated between three-dimensional data without projection. There is an effect of improving the accuracy of alignment and collation. For example, an edge presence is given to each of a spatial model or an object model obtained by measuring from a plurality of different measurement viewpoints, and an edge exists indicating that an edge exists in these multiple spatial models or multiple object models If the positioning is performed with emphasis on the three-dimensional coordinate data to which the degree is given, the positioning accuracy can be improved. Also, for example, when edge recognition is given to an object model obtained by measuring a predetermined object, and object recognition is performed by collating the object model with three-dimensional data obtained by measuring an unknown object, If the collation is performed with emphasis on the three-dimensional coordinate data to which the edge presence level indicating that the image exists, the collation accuracy can be improved.

  (7) In the above embodiment, an example in which the camera is calibrated using the virtual edge image has been described. However, the accuracy of object matching can be improved using the virtual edge image. For example, if a virtual edge image is generated by adding an edge presence to an object model obtained by measuring a predetermined object and collated with a real edge image generated from a captured image, the object shown in the captured image is The accuracy of determining whether or not the object is a predetermined object can be improved.

  1 camera calibration system, 2 3D measuring device, 3 monitoring camera, 4 monitoring device, 5 3D data processing device, 50 measurement data input unit, 51 storage unit, 52 monitoring image input unit, 53 control unit, 54 output unit, 55 User interface unit, 510 Spatial model storage unit, 511 Measurement parameter storage unit, 530 Coordinate conversion unit, 531 Edge presence degree provision unit, 532 Change point extraction unit, 533 Edge presence level determination unit, 535 Real edge image generation unit, 536 Virtual camera setting means, 537 virtual edge image generation means, 538 edge image collation means, 600 room, 601 floor, 602 desk, 603 pattern, 650 measurement position, 651 measurement position, 652 shooting position, 653 shooting position.

Claims (7)

3D model storage means for storing a 3D model of the object composed of a plurality of 3D coordinate data obtained by measuring the depth from the base point to the surface of the object;
Each of the three-dimensional coordinate data located at the boundary with the depth-unmeasured area, which is a blind spot from the base point, and the front data and the back side whose directions from the base point are adjacent to each other and whose depth from the base point is different by a predetermined value or more. Edge presence degree giving means for storing an edge affirmation value indicating that the target edge exists in the previous data among the data in the 3D model storage means,
A three-dimensional model processing apparatus comprising: The three-dimensional model processing apparatus according to claim 1, further comprising:
Virtual camera setting means for setting a virtual camera for simulating and photographing the three-dimensional model at an arbitrary position and orientation;
Virtual edge image generation means for generating a virtual edge image with the projection position of the three-dimensional coordinate data having the edge positive value as an edge pixel on the imaging surface of the virtual camera;
A three-dimensional model processing apparatus comprising: The three-dimensional model processing device according to claim 2,
A real edge image generating means for acquiring a real image obtained by photographing a target by the real camera, extracting an edge pixel representing the edge of the target on the real image, and generating a real edge image;
Edge image matching means for calculating the degree of similarity between the real edge image and the virtual edge image and determining the position and orientation of the virtual camera that maximizes the degree of similarity as the position and orientation of the real camera;
A camera calibration system characterized by comprising: Stores a 3D model of the object composed of a plurality of 3D coordinate data obtained by measuring the depth from the base point to the surface of the object, and at each boundary with a depth unmeasured area that is a blind spot from the base point The front data of the front data and the back data, which are the three-dimensional coordinate data that is positioned and whose directions from the base points are adjacent to each other and whose depths from the base points are different from each other by a predetermined value or more, include the target edge. 3D model storage means to which an edge positive value indicating
Virtual camera setting means for setting a virtual camera for simulating and photographing the three-dimensional model at an arbitrary position and orientation;
Virtual edge image generation means for generating a virtual edge image with the projection position of the three-dimensional coordinate data having the edge positive value as an edge pixel on the imaging surface of the virtual camera;
A three-dimensional model processing apparatus comprising: The three-dimensional model processing device according to claim 4,
The three-dimensional model stored in the three-dimensional model storage means further adds the edge affirmative value to the three-dimensional coordinate data corresponding to a pixel whose luminance difference with an adjacent pixel is not less than a predetermined value in an image obtained by photographing the target. A three-dimensional model processing device characterized by being provided. The three-dimensional model processing device according to claim 4 or 5,
The three-dimensional model stored in the three-dimensional model storage means further includes the edge affirmative value in three-dimensional coordinate data in which a difference between normal vectors of the surface of the target from adjacent three-dimensional coordinate data is a predetermined value or more. A three-dimensional model processing device characterized by being given. A three-dimensional model composed of a plurality of three-dimensional coordinate data representing the position of the target surface, and a region where the three-dimensional coordinate data is not obtained exists on the target surface, wherein the three-dimensional coordinate data is Three-dimensional model storage means for storing a three-dimensional model to which an edge presence level indicating a high possibility that the target edge exists at the position represented by:
Calibration means for determining the position and orientation of the camera that captures the object using the three-dimensional coordinate data in which the edge presence is equal to or greater than a predetermined value;
A camera calibration system characterized by comprising:

Images