JP6374812B2 - 3D model processing apparatus and camera calibration system - Google Patents

Description

  The present invention uses a three-dimensional model processing device that generates a three-dimensional model from three-dimensional coordinate data obtained by measuring a predetermined target, a three-dimensional model processing device that generates an edge image of an arbitrary viewpoint from a three-dimensional model, and a three-dimensional model. The present invention relates to a camera calibration system for calibrating a camera.

  If the space where the surveillance camera is installed is three-dimensionally measured and a three-dimensional model (spatial model) of the space is generated in advance, by searching for a viewpoint that matches the captured image of the surveillance camera using the spatial model, The surveillance camera can be calibrated. In the search, a virtual model with various positions and orientations is simulated and the virtual camera position where the edge position is the best is obtained by comparing the simulated image with the image actually captured by the surveillance camera. The posture is determined as the position and posture of the monitoring camera.

  In Non-Patent Document 1 below, an arbitrary viewpoint image (generated image) is generated for a three-dimensional model of a room by a virtual camera with known camera parameters, and extracted from a real image and a generated image captured by a portable camera, respectively. It describes that the position / posture of a portable camera is estimated by performing a matching process on SIFT (Scale Invariant Feature Transform), which is a kind of edge.

Jun Nishida, et al., “Towards Position Estimation of Mobile Camera Based on Virtual Reality Environment Model”, IEICE Technical Report, vol. 111, no. 101, MVE2011-28, pp. 73-78, June 2011

  However, in a spatial model obtained by actually performing three-dimensional measurement, an unmeasured region caused by occlusion can be included. Due to this unmeasured area, there is a problem that the number of edges useful for calibration of the surveillance camera is reduced.

  For example, it is difficult to three-dimensionally measure a room that is a security target, and a blind spot caused by the shadow of a desk when viewed from a measuring device such as a distance image sensor becomes an unmeasured area. Then, if the spatial model is simulated and photographed from other than the viewpoint at the time of measurement, the edge of the desk that made the blind spot appears next to the unmeasured area on the simulated image. Here, in the non-measurement area, information on whether the desk surface measured adjacent to the non-measurement area continuously spreads or ends at the boundary with the non-measurement area cannot be obtained. For this reason, it is impossible to extract the edge of the desk which is useful for calibration of the surveillance camera even though the edge of the desk itself can be measured. This leads to a decrease in the calibration accuracy of the surveillance camera.

  The unmeasured area adjacent to the shaded desk edge on the simulated image is adjacent to the floor other than the desk edge. Therefore, if the boundary between the unmeasured area and the measurement area is regarded as an edge in order to extract the edge of the desk edge from the simulated image, a false edge is extracted from the floor portion. The extraction of false edges also leads to a decrease in the calibration accuracy of the surveillance camera.

  In addition, there is a problem that useful edges are similarly reduced in object recognition using a three-dimensional model obtained by measuring an object and a process of connecting three-dimensional models obtained by measuring a common space. It was.

  The present invention has been made in view of the above problems, and an object of the present invention is to realize a three-dimensional model processing apparatus that can provide a three-dimensional model that can extract a true edge at the boundary with an unmeasured region. Another object of the present invention is to provide a three-dimensional model processing apparatus capable of generating an edge image of an arbitrary viewpoint including a true edge at the boundary with an unmeasured region from a three-dimensional model. Furthermore, another object of the present invention is to improve the accuracy of camera calibration using a three-dimensional model.

  (1) A three-dimensional model processing apparatus according to the present invention includes a three-dimensional model obtained by measuring at a plurality of measurement positions and orientations, which is constituted by a plurality of three-dimensional coordinate data representing a target surface shape, and the plurality of measurement positions. Storage means for storing an image obtained by photographing the object at each of the plurality of measurement positions and postures, and a luminance change point is extracted from each image, and the luminance change points are obtained using the measurement position and posture. Change point associating means for detecting three-dimensional coordinate data at a position corresponding to the edge, and an edge for assigning a degree of edge presence corresponding to the number of detected correspondences between the luminance change points to the three-dimensional coordinate data Presence degree providing means.

  (2) Another three-dimensional model processing apparatus according to the present invention, in addition to the configuration of (1) above, a virtual camera setting means for setting a virtual camera for simulating and photographing the three-dimensional model at an arbitrary position and orientation; Virtual edge image generation means for generating a virtual edge image with the projection position of the three-dimensional coordinate data whose edge presence is higher than a predetermined height on the imaging surface of the virtual camera as an edge pixel.

  (3) A camera calibration system according to the present invention acquires a real image obtained by photographing the target with the 3D model processing apparatus of (2) above and a real camera, and extracts an edge from the real image to obtain a real edge image. A real edge image generating means for generating the virtual edge image, calculating a similarity between the real edge image and the virtual edge image, and determining the position and orientation of the virtual camera that maximizes the similarity as the position and orientation of the real camera Edge image matching means.

  (4) Another three-dimensional model processing apparatus according to the present invention stores a three-dimensional model obtained by measuring at a plurality of measurement positions and postures, which is constituted by a plurality of three-dimensional coordinate data representing a target surface shape, The three-dimensional coordinate data is given a degree of edge presence according to the number of detected correspondences with luminance change points extracted from images obtained by photographing the target at each of the plurality of measurement positions and orientations. Storage means, virtual camera setting means for setting a virtual camera for simulating and photographing the three-dimensional model at an arbitrary position and orientation, and an edge presence level indicating that an edge exists on the photographing surface of the virtual camera Virtual edge image generation means for generating a virtual edge image having the projection position of the three-dimensional coordinate data as an edge pixel.

  According to the present invention, it is possible to generate a three-dimensional model that can extract an edge at a boundary with an unmeasured region. Further, according to the present invention, an edge image of an arbitrary viewpoint including an edge at a boundary with an unmeasured region can be generated from a three-dimensional model including the unmeasured region. In addition, according to the present invention, camera calibration can be performed with high accuracy using a three-dimensional model obtained by measurement.

1 is a block diagram showing a schematic configuration of a camera calibration system according to an embodiment of the present invention. It is a functional block diagram of a three-dimensional data processing device when generating a space model of a room using measurement data input from the three-dimensional measurement device. It is a typical perspective view which shows the mode of the measurement of the room using a three-dimensional measuring apparatus. It is a schematic diagram which shows the example of data of a space model. It is a schematic diagram which shows the example of the RGB image which image | photographed the room with a different measurement position orientation. It is a schematic diagram which shows the correspondence of the brightness | luminance change point between each of two RGB images from which a measurement position orientation differs, and a space model. It is a schematic diagram which shows the example of edge presence provision. It is a schematic diagram of the space model of the room for demonstrating the example of edge presence degree. It is a functional block diagram of the outline of the three-dimensional data processing apparatus when calibrating the surveillance camera installed in the room using the room space model. It is a schematic flowchart of operation | movement of the camera calibration system at the time of a space model production | generation. It is a schematic flowchart of operation | movement of the camera calibration system at the time of camera calibration.

  Hereinafter, as an embodiment of the present invention (hereinafter referred to as an embodiment), an example of a camera calibration system that calibrates a surveillance camera using a three-dimensional model (space model) of a space in which the surveillance camera is installed will be described. This camera calibration system includes an example of a three-dimensional model processing device that generates a space model of the space from measurement data obtained by three-dimensionally measuring the space and generates an edge image of an arbitrary viewpoint from the space model. .

[Configuration of Camera Calibration System 1]
FIG. 1 is a block diagram showing a schematic configuration of the camera calibration system 1. The camera calibration system 1 includes a three-dimensional measuring device 2, a monitoring camera 3, a monitoring device 4, and a three-dimensional data processing device 5.

  The three-dimensional measuring device 2 three-dimensionally measures the room in which the surveillance camera 3 is installed to obtain a distance image, and captures the room in synchronization with it to obtain an RGB image. The distance image of the room and the RGB image Is output to the three-dimensional data processing device 5. For example, the three-dimensional measuring device 2 is an RGBD camera, and is the same in synchronization with the distance sensor of the TOF (Time Of Flight) method that measures the distance from the time required for the projected laser light to reciprocate with the target surface, and the distance sensor. This is a device combined with a color camera that captures an RGB image of the field of view. The camera may be a monochrome camera that acquires gray images. Then, the three-dimensional measurement and the photographing by the three-dimensional measurement apparatus 2 are performed at a plurality of measurement positions and postures by including at least a common part of the target in the measurement region. That is, the three-dimensional measuring device 2 performs three-dimensional measurement on a predetermined target at a plurality of measurement positions and orientations, outputs three-dimensional data of the predetermined target to the three-dimensional data processing device 5, and further synchronizes with the three-dimensional measurement. A predetermined object is photographed at each measurement position and orientation, and a two-dimensional image obtained by photographing the predetermined object at each measurement position and orientation is output to the three-dimensional data processing device 5.

  The surveillance camera 3 is a camera to be calibrated. For example, a plurality of surveillance cameras 3 are respectively installed on the ceiling of the room, the room is photographed from the ceiling, and a surveillance image and a camera ID assigned in advance are output to the surveillance device 4. When the monitoring camera 3 is calibrated, the monitoring camera 3 outputs a monitoring image and a camera ID to the three-dimensional data processing device 5.

  The monitoring device 4 receives camera parameters including the position and orientation (imaging position and orientation) of the monitoring camera 3 from the three-dimensional data processing device 5 and stores them. The shooting position and orientation are a shooting position (light spot position) and a shooting posture (optical axis direction and its rotation angle) of the monitoring camera 3 in the world coordinate system imitating real space. The process for obtaining the shooting position and orientation of the monitoring camera 3 is calibration.

  In addition, the monitoring device 4 analyzes a monitoring image from the monitoring camera 3 to track a person who moves in the room and detects suspicious behavior. When detecting a suspicious person, the monitoring device 4 determines the actual size of the change area extracted from the monitoring image based on the calibrated camera parameters, and determines whether or not the change area is a person.

  The three-dimensional data processing device 5 is configured by a computer such as a PC (Personal Computer). The three-dimensional data processing device 5 generates a spatial model using the measurement data input from the three-dimensional measurement device 2. Also, the three-dimensional data processing device 5 receives the monitoring image and the camera ID from the monitoring camera 3, calibrates the monitoring camera 3 based on the monitoring image and the space model, and the shooting position and orientation of the monitoring camera 3 as a calibration result. Are output to the monitoring device 4 together with the corresponding camera ID. Camera parameters include focal length, resolution (number of pixels in the width direction, number of pixels in the height direction), lens distortion, and the like in addition to the shooting position and orientation. Of these, there is basically no change except for the shooting position and orientation. Therefore, for camera parameters other than the shooting position and orientation, values obtained by referring to the specifications of the monitoring camera 3 or values obtained by previous measurement are stored in the three-dimensional data processing device 5 in advance.

  The three-dimensional data processing device 5 includes a measurement data input unit 50, a storage unit 51, a monitoring image input unit 52, a control unit 53, an output unit 54 and a user interface unit 55.

  The measurement data input unit 50 is a communication interface circuit that connects the three-dimensional measurement device 2 and the control unit 53, receives measurement data from the three-dimensional measurement device 2, and outputs this to the control unit 53.

  The storage unit 51 is a storage device such as a ROM (Read Only Memory) or a RAM (Random Access Memory), stores various programs and various data, and inputs / outputs these information to / from the control unit 53.

  The monitoring image input unit 52 is a communication interface circuit that connects the monitoring camera 3 and the control unit 53. The monitoring image input unit 52 receives the captured image and the camera ID from the monitoring camera 3 and outputs them to the control unit 53.

  The control unit 53 is configured using an arithmetic device such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or an MCU (Micro Control Unit). The control unit 53 reads out and executes the program from the storage unit 51 and functions as each unit described later.

  The output unit 54 is a communication interface circuit that connects the control unit 53 and the monitoring device 4, receives the camera parameters and camera ID of the monitoring camera 3 from the control unit 53, and outputs them to the monitoring device 4.

  The user interface unit 55 is a user interface device including a keyboard, a mouse, a display, and the like. For example, it is used by a user such as an installation worker, and is used for inputting various parameters such as designating a floor area in measurement data.

[Three-dimensional data processing device 5 when generating a spatial model]
FIG. 2 is a functional block diagram of the three-dimensional data processing device 5 when a spatial model is generated using measurement data input from the three-dimensional measurement device 2. The storage unit 51 is a storage unit that functions as the space model storage unit 510 and the measurement parameter storage unit 511. The control unit 53 functions as a coordinate conversion unit 530, a change point association unit 531, and an edge presence degree providing unit 532.

  The space model storage unit 510 is a storage unit that stores a space model of a predetermined object obtained by the measurement of the three-dimensional measurement apparatus 2, that is, a three-dimensional model of the predetermined object obtained by the measurement of the three-dimensional measurement apparatus 2. The spatial model includes a plurality of three-dimensional coordinate data (referred to as a point cloud) measured by the three-dimensional measuring device 2 and converted by the coordinate conversion means 530, luminance data associated with each three-dimensional coordinate data, and edges It consists of the edge presence level given to each three-dimensional coordinate data by the presence level giving means 532. Here, the three-dimensional coordinate data means that the object surface exists at the position, and the surface shape of the object is represented by a plurality of three-dimensional coordinate data. The luminance data represents the color and luminance of the object surface at that position. The edge presence level is a value representing the high possibility that an edge exists at the position of the corresponding three-dimensional coordinate data. In this embodiment, the edge presence level is defined as 0.0 or more and 1.0 or less. To do. The edge appears at a position indicated by three-dimensional coordinate data whose luminance changes greatly depending on the unevenness of the object or the pattern on the object surface. The edge presence will be further described later.

  The measurement parameter storage unit 511 stores measurement parameters including the position and orientation (measurement position and orientation) at the time of measurement of the three-dimensional measurement apparatus 2. The measurement position / orientation is represented by a measurement position (light spot position) and a measurement attitude (optical axis direction and its rotation angle) of the three-dimensional measurement apparatus 2 in a world coordinate system imitating real space. The measurement position / orientation is calculated by the coordinate conversion means 530. Alternatively, separately measured measurement positions and orientations may be input in advance from the user interface unit 55 and stored. The measurement of the three-dimensional measuring apparatus 2 is performed at a plurality of positions and orientations, and the measurement parameter storage unit 511 stores the positions and orientations at the time of each measurement.

  The measurement parameters include a focal length, resolution (number of pixels in the width direction, number of pixels in the height direction) and the like in addition to the measurement position and orientation. Since there is basically no change except for the measurement position and orientation, a value obtained by referring to the specification of the three-dimensional measurement apparatus 2 or a value obtained by a previous measurement is stored in advance. The angle of view can be derived from the focal length and the resolution. In addition, the projection plane at the time of measurement can be derived from the measurement parameters.

  The coordinate conversion unit 530 performs coordinate conversion of the measurement data input from the three-dimensional measurement apparatus 2 via the measurement data input unit 50 into three-dimensional coordinate data of the world coordinate system, and stores it in the space model storage unit 510. Also, the coordinate conversion unit 530 calculates the measurement position and orientation of the three-dimensional measurement apparatus 2 at the time of measurement from the measurement data, and stores the calculated measurement position and orientation in the measurement parameter storage unit 511.

  The outline procedure of the coordinate conversion process is as follows: (a) calculating a three-dimensional point group in a local coordinate system having the measurement position as the origin from the distance image for each measurement position and orientation; A coordinate system is used as a reference coordinate system, a three-dimensional point group of a plurality of measurement positions and orientations is integrated into the reference coordinate system, and (c) a reference of the world coordinate system is set in the reference coordinate system, and the measurement position and orientation RT in the world coordinate system is And (d) converting the three-dimensional point group of the reference coordinate system into the three-dimensional coordinate data of the world coordinate system using RT. Each procedure will be described below.

In step (a), the coordinate system of the distance image (x, y, d) in which the pixel value of each pixel (x, y), that is, the depth value is d, is first three-dimensionally measured according to the following equation for each measurement position and orientation. Conversion to a three-dimensional point group (Xc, Yc, Zc) in the local coordinate system with the light spot (measurement position) of the apparatus 2 as the origin.

  However, the center pixel position of the distance image is (cx, cy). f is a focal length of the three-dimensional measuring apparatus 2.

  In step (b), one of the local coordinate systems is defined as a reference coordinate system, and a plurality of local coordinate systems are integrated into the reference coordinate system. For example, in order to integrate the local coordinate system of the measurement position and orientation α2 into the reference coordinate system of the measurement position and orientation α1, the three-dimensional point group measured with the measurement position and orientation α1 and the three-dimensional point group measured with the measurement position and orientation α2. Estimate multiple 3D points that measure common positions and associate them. Then, a conversion matrix RT1 for converting the local coordinate system of the measurement position and orientation α2 into the reference coordinate system of the measurement position and orientation α1 is calculated from the plurality of associations.

In step (c), in order to calculate the measurement position / orientation RT, a point that is a floor surface among the three-dimensional point group of the reference coordinate system is selected and input using the user interface unit 55, and based on this selection input, The coordinate value of the floor surface immediately below the measurement position in the reference coordinate system is calculated. Thereby, for example, the measurement position / orientation RT0 in the world coordinate system can be calculated in which the floor is the XY plane, the height direction is the Z axis, and the point on the floor immediately below the measurement position defined as the reference is the origin. Then, the measurement position / orientation RT is calculated according to the following equation.
RT = RT0 ・ RT1

In step (d), using the RT obtained in step (c) for each measurement position and orientation, the three-dimensional point group of the local coordinate system at the measurement position and orientation is converted into the three-dimensional coordinate data ( X, Y, Z).

  Further, the coordinate conversion unit 530 stores the luminance data, which is the pixel value of each pixel of the RGB image, in the spatial model storage unit 510 in association with the conversion destination three-dimensional coordinate data of the pixel of the distance image corresponding to each pixel. Let

  FIG. 3 is a schematic perspective view showing a state of measurement of the room 600 using the three-dimensional measuring apparatus 2. The room 600 is a room to be guarded, for example, and the monitoring camera 3 to be calibrated is installed on the ceiling. The room 600 includes a floor 601 and a desk 602, and a pattern 603 is formed on the floor 601. Incidentally, as will be described later, the camera calibration system 1 uses an edge such as an edge of the desk 602 as a reference for calibration of the surveillance camera 3.

  The installation worker of the monitoring camera measures the room 600 three-dimensionally using the three-dimensional measuring device 2. The three-dimensional measuring apparatus 2 is a distance in which each pixel represents a depth from the measurement position 650a to each of a plurality of points on the surface of the floor 601 or the desk 602 in the first measurement position and orientation represented by the measurement position 650a and the measurement orientation 651a. While outputting an image, the RGB image which image | photographed the room 600 in the 1st measurement position attitude | position is output. The measurement at this time is referred to as a first measurement.

  Further, the installation operator moves the three-dimensional measurement apparatus 2 and three-dimensionally measures the room 600 at different measurement positions and postures. This measurement is referred to as a second measurement. That is, the three-dimensional measurement apparatus 2 has a depth from the measurement position 650b to each of a plurality of points on the surface of the floor 601 or the desk 602 in the second measurement position and orientation represented by the measurement position 650b and the measurement orientation 651b. Is output, and an RGB image obtained by capturing the room 600 at the second measurement position and orientation is output. Incidentally, the monitoring camera 3 basically shoots the room 600 at a shooting position 652 and a shooting attitude 653 different from the first measurement position and posture.

  The area represented by the distance image and the RGB image output from the three-dimensional measurement apparatus 2 is called a measurement area. The measurement area cannot cover the entire room 600, and an unmeasured area occurs due to occlusion. In FIG. 3, a white area is a measurement area by the first measurement. In FIG. 3, the shaded area is an unmeasured area generated by the first measurement, and is a blind spot from the first measurement position and orientation caused by occlusion by the desk 602. In addition, the hatched area in FIG. 3 is an out-of-view-angle area that was out of the view angle of the three-dimensional measurement apparatus 2 in the first measurement.

  FIG. 4 is a schematic diagram showing an example of data of a spatial model, and is stored in the spatial model storage unit 510 in a table format in which, for example, three-dimensional coordinate data, luminance data, and edge presence are associated with each other. Incidentally, all the data shown in FIG. 4 is data of the measurement region. The first three-dimensional coordinate data has the surface of the object at coordinates (X1, Y1, Z1) in the spatial model, the color is (R1, G1, B1) in the RGB color system, and the value of edge presence “1.0” indicates that an edge such as the edge of the desk 602 exists at the coordinates. Similarly, the second three-dimensional coordinate data indicates that an edge of the object surface exists at coordinates (X2, Y2, Z2) in the spatial model, and the color thereof is (R2, G2, B2). The third three-dimensional coordinate data has the surface of the object at coordinates (X3, Y3, Z3) in the spatial model, the color is (R3, G3, B3) in the RGB color system, and the edge presence value “0.0” indicates that the possibility that an edge exists there is small.

  An edge is a feature point that is useful for camera calibration or the like that occurs at the edge of an object. However, there is a problem that the spatial model cannot be used even if an edge exists in the three-dimensional coordinate data located at the boundary with the unmeasured area. Moreover, since the occlusion is likely to occur near the edge of the object, useful edges are likely to be lost. For example, the monitoring image captured by the monitoring camera 3 in FIG. 3 includes an unmeasured area generated in the first measurement. When a monitoring image captured by the monitoring camera 3 is simulated using a space model obtained by measuring the room 600, the edge of the desk 602 and the plane of the floor 601 are mixed at the boundary of the unmeasured area. Both useful edges appearing on the edge of the desk 602 must be discarded unless the 3D coordinate data of the edge of the desk 602 and the 3D coordinate data of the plane of the floor 601 that are both located at the boundary with the unmeasured area can be identified. In the present invention, the change point associating means 531 and the edge presence degree assigning means 532 identify the three-dimensional coordinate data in which the useful edge is present, and an edge positive value indicating that the edge exists in the identified three-dimensional coordinate data. By assigning, a three-dimensional model that can utilize useful edges is generated.

  The change point association unit 531 extracts a luminance change point from each RGB image, and detects the three-dimensional coordinate data of the position corresponding to the luminance change point using the measurement position and orientation. Specifically, the change point associating unit 531 extracts a luminance change point where the luminance difference with the surrounding pixels is equal to or greater than a predetermined value in each RGB image, and uses the measurement position and orientation to determine the three-dimensional position corresponding to the luminance change point. The coordinate data is detected, and the detected correspondence is output to the edge presence degree assigning means 532.

  Here, extraction of luminance change points will be described. In the present embodiment, edge strength is used as the luminance difference. The change point association unit 531 applies a predetermined edge operator to each pixel of the RGB image photographed at the first measurement position and orientation to calculate the edge strength between the pixel and the surrounding pixels, and the edge strength is the threshold value. A pixel that is equal to or greater than the value Ti is extracted as a first luminance change point. Similarly, a predetermined edge operator is applied to each pixel of the RGB image photographed at the second measurement position and orientation to calculate the edge strength between the pixel and its surrounding pixels, and the edge strength is equal to or greater than the threshold value Ti. A pixel is extracted as a second luminance change point.

  In the present embodiment, the RGB image input from the three-dimensional measuring device 2 is used for extracting the luminance change point. On the other hand, instead of directly using the RGB image from the three-dimensional measurement apparatus 2, the RGB image may be reproduced from the spatial model and the measurement parameter, and the luminance change point may be extracted from the reproduced RGB image. Specifically, an RGB image can be reproduced by a rendering process in which luminance data corresponding to a plurality of three-dimensional coordinate data constituting a space model is projected onto a projection plane represented by a measurement parameter. Note that the angle of view of the RGB image to be reproduced is matched to the angle of view of the three-dimensional measuring apparatus 2. At the time of measurement, only the process of the coordinate conversion unit 530 is performed without storing the RGB image in the storage unit 51, and the process of the change point association unit 531 can be executed at another time after the measurement. This configuration is effective when the capacity of the storage unit 51 and the processing capability of the control unit 53 are low. As the edge operator, a Sobel operator can be used, but other known operators such as a Laplacian filter may be used.

  Next, detection of the correspondence between the brightness change points extracted at each of the plurality of measurement positions and orientations and the three-dimensional coordinate data will be described. Even if the same position in the real space is measured by the first measurement and the second measurement, the three-dimensional coordinate data obtained by both is shifted by one pixel at maximum due to the influence of quantization. The correspondence is detected in consideration of this deviation.

  By the way, the pixels of the RGB image photographed in each measurement and the three-dimensional coordinate data obtained in the measurement are already linked one-to-one in the conversion process performed by the coordinate conversion unit 530 using the measurement parameters at the time of each measurement. The change point association means 531 has already been associated one-to-one by a projection process performed using measurement parameters at the time of each measurement in order to reproduce an RGB image. This association is utilized in the following correspondence detection.

  The three-dimensional coordinate data associated with the luminance change point extracted from the RGB image photographed at the first measurement position and orientation is specified. Then, the specified three-dimensional coordinate data is projected onto the RGB image captured at the second measurement position and orientation using the measurement parameters including the second measurement position and orientation, and the projection destination pixel is the luminance change point. Make sure.

  If the projection destination pixel is a luminance change point, the three-dimensional distance between the three-dimensional coordinate data associated with the projection destination pixel and the three-dimensional coordinate data of the projection source is calculated, and the three-dimensional distance is calculated. It is confirmed whether it is within the error range for one pixel. When the three-dimensional distance is within the error range, the first measurement position and orientation are set for each of the three-dimensional coordinate data associated with the projection source three-dimensional coordinate data and the projection destination pixel. That is, the correspondence between the luminance change point of the RGB image photographed in this way and the luminance change point of the RGB image photographed at the second measurement position and orientation is detected.

  On the other hand, if the projection destination pixel is not a luminance change point, the correspondence between the projection source three-dimensional coordinate data and the luminance change point has not been detected.

  In addition, instead of the method described above, each three-dimensional coordinate data is projected onto the RGB image captured by the first measurement using the first measurement position and orientation, and each tertiary using the second measurement position and orientation. By projecting the original coordinate data onto the RGB image captured by the second measurement and confirming whether the projection destination pixel is a luminance change point, the correspondence between the three-dimensional coordinate data and the luminance change point is determined. It may be detected.

  The edge presence degree giving means 532 gives edge presence degrees according to the number of detected correspondences with luminance change points to a plurality of three-dimensional coordinate data constituting the spatial model. That is, referring to the correspondence relationship between the luminance change point detected by the change point association unit 531 and the three-dimensional coordinate data, an edge having a height corresponding to the number of correspondences detected with the luminance change point in the three-dimensional coordinate data. Give presence. Specifically, the number of first luminance change points and second luminance change points at which correspondence with the three-dimensional coordinate data is detected is counted for each three-dimensional coordinate data. The edge presence degree giving means 532 determines the edge presence degree to be 1.0 for 3D coordinate data having a count value of 2 or more, and the edge for 3D coordinate data having a count value of less than 2. The presence is determined to be 0.0. For the three-dimensional coordinate data for which no correspondence relationship has been detected, the edge presence is determined to be 0.0. The edge presence degree assigning unit 532 stores the determined edge presence degree in the space model storage unit 510. As a result, among the three-dimensional coordinate data located at the boundary with the unmeasured area, the edge presence degree indicating that the edge exists can be given to the three-dimensional coordinate data in which the edge of the outer portion of the object actually exists. The edge presence degree indicating that no edge exists can be given to the three-dimensional coordinate data that is unlikely to have an edge.

  5 and 6 are schematic diagrams showing examples of the above-described correspondence relationship. FIG. 5 is a schematic diagram illustrating an example of an RGB image obtained by photographing the room 600 at different measurement positions and orientations. The RGB image 750 in FIG. 5A is a first measurement position and orientation (measurement position 650a and measurement orientation 651a). The RGB image 760 in FIG. 5B is a photograph of the room 600 taken at the second measurement position / posture (measurement position 650b, measurement posture 651b). FIG. 6 is a schematic diagram showing the correspondence between the luminance change points between the RGB image 750 and the space model 800 and the correspondence between the luminance change points between the RGB image 760 and the space model 800. FIG. 6A is a schematic perspective view of a space model of the room 600 showing the positional relationship such as each measurement position / posture and luminance change point, and FIG. 6B shows the correspondence relationship in a tabular form. is there.

  Pixels 751 and 752 adjacent to each other in the RGB image 750 shown in FIG. 5A are extracted as luminance change points. Among these, as shown in FIG. 6A, the edge of the desk 852 actually exists at the position represented by the three-dimensional coordinate data 801 corresponding to the pixel 751. On the other hand, the position represented by the three-dimensional coordinate data 803 corresponding to the pixel 752 is actually a position on the plane of the floor 851 and no edge exists.

  On the RGB image 750, adjacent pixels (751, 752) are extracted as luminance change points, but the corresponding three-dimensional coordinates (801, 803) in the spatial model are separated as shown in FIG. An unmeasured area 850 in the first measurement is interposed between them. That is, the three-dimensional coordinate data corresponding to the luminance change point of the RGB image 750 photographed at the first measurement position / posture includes a mixture of data having no edge at that position. The location of the three-dimensional coordinate data related to the mixture in the space model is the boundary of the unmeasured area in the first measurement.

  The three-dimensional coordinate data 801 is projected onto the RGB image 760 using the measurement parameters including the second measurement position and orientation. The projection destination is the pixel 761. The edge of the desk 852 is shown in the pixel 761, and the pixel 761 is extracted as a luminance change point. The three-dimensional coordinate data 802 associated with the pixel 761 is data obtained by measuring the edge of the desk 852 in the same way as the three-dimensional coordinate data 801, and one pixel exists between the three-dimensional coordinate data 802 and the three-dimensional coordinate data 801. A three-dimensional distance that is less than a minute error is calculated. As a result, the correspondence between the pixel 751 and the pixel 761 is detected for the three-dimensional coordinate data 801 and 802 (see FIG. 6B). That is, for the three-dimensional coordinate data 801 and 802, the correspondence between two luminance change points is detected.

  On the other hand, another pixel 752 extracted as a luminance change point in the RGB image 750 corresponds to the three-dimensional coordinate data 803. When the three-dimensional coordinate data 803 is projected onto the RGB image 760 using the measurement parameters including the second measurement position, the projection destination is the pixel 762. However, the pixel 762 in which the planar portion on the floor 851 is not extracted as a luminance change point. As a result, only the correspondence with the luminance change point of the RGB image 750 is detected for the three-dimensional coordinate data 803 (see FIG. 6B). That is, for the three-dimensional coordinate data 803, only one correspondence with the luminance change point is detected.

  FIG. 7 is a schematic diagram showing an example of edge presence assignment. FIG. 7A is a schematic perspective view of a space model of the room 600 and shows positions of Q1 to Q14 which are examples of three-dimensional coordinate data. FIG. 7B shows the number of corresponding luminance change points and the edge presence in Q1 to Q14 in a tabular form. As described with reference to FIG. 6, the three-dimensional coordinate data 801 (Q7) has a correspondence relationship with the luminance change point (pixel 751) extracted from the RGB image 750 taken at the first measurement position and orientation. , And a correspondence relationship with the luminance change point (pixel 761) extracted from the RGB image 760 captured at the second measurement position and orientation, and the luminance change point corresponding to the three-dimensional coordinate data Q7 is counted as 2. The Depending on the number of luminance change points, the edge presence degree 1.0 is given to the three-dimensional coordinate data Q7. Similarly, with respect to the three-dimensional coordinate data Q1 to Q6, Q8, and Q9 located on the edge of the desk 602, the number of luminance change points is counted as 2, and the edge presence degree 1.0 is given.

  Further, as described with reference to FIG. 6, the three-dimensional coordinate data 803 (Q11) has a correspondence relationship with the luminance change point 751 extracted from the RGB image 750 captured at the first measurement position and orientation. Although detected, the correspondence with the luminance change point extracted from the RGB image 760 taken at the second measurement position and orientation was not detected. Therefore, the number of luminance change points corresponding to the three-dimensional coordinate data Q11 is counted as 1. Depending on the number of luminance change points, the edge presence degree of 0.0 is given to the three-dimensional coordinate data Q11. Similarly, the luminance change point is counted as 1 for the three-dimensional coordinate data Q10 and Q12 located on the boundary with the unmeasured area (the unmeasured area generated in the first measurement) on the plane of the floor 601. Edge presence degree 0.0 is given.

  Further, the number of luminance change points with respect to the three-dimensional coordinate data Q14 located on the edge of the pattern 603 shown in FIG. 3 on the plane of the floor 601 is 2, and the edge presence degree 1.0 is given according to the number of luminance change points. .

  Further, the number of luminance change points for the three-dimensional coordinate data Q13 at a position that is neither the boundary with the unmeasured area nor the edge of the pattern 603 on the plane of the floor 601 is 0, and the edge presence degree is 0.0 according to the number of luminance change points. Is granted.

  FIG. 8 is a schematic diagram of a space model of the room 600 for explaining an example of the edge presence degree. In FIG. 8, the solid line parts (straight line P1-P2, P1-P3, P1-P5, P2-P4, P3-P4, P3-P6, P4-P7, P5-P6, P6-P7 part and straight line group S). An edge presence of 1.0 is assigned to the three-dimensional coordinate data, and an edge presence of 0.0 is assigned to the other three-dimensional coordinate data.

  Of these solid lines, the three-dimensional coordinate data of the straight lines P1-P2, P1-P5, P2-P4 and P4-P7 are located at the boundary with the unmeasured area (shaded area). Although it was an edge that could not be used, in the present invention, an edge presence indicating that an edge is present is given and can be used for camera calibration and the like.

[3D data processor 5 during camera calibration]
FIG. 9 is a schematic functional block diagram of the three-dimensional data processing apparatus 5 when the surveillance camera 3 installed in the room 600 is calibrated using the space model of the room 600.

  The storage unit 51 functions as the space model storage unit 510, and the control unit 53 functions as the real edge image generation unit 535, the virtual camera setting unit 536, the virtual edge image generation unit 537, the edge image collation unit 538, and the like.

  The real edge image generation means 535 extracts an edge from each monitoring image (real image) input from the monitoring camera 3 (real camera) to generate an edge image, and the generated edge image together with the corresponding camera ID is used as an edge image. The result is output to the verification unit 538. In order to distinguish from an edge image generated by a virtual edge image generation means 537 described later, the edge image generated here is called a real edge image.

  The real edge image generation means 535 has a pixel value “1” for the pixel corresponding to the luminance change point where the luminance difference with the surrounding pixels in the monitoring image is equal to or greater than a predetermined threshold value Ti, and a pixel value “0” for the other pixels. "" Is set to generate an actual edge image. In the present embodiment, edge strength is used as the luminance difference. The real edge image generation means 535 calculates edge strength by applying a predetermined edge operator such as a Sobel operator to each pixel of the monitoring image.

  The virtual camera setting unit 536 sets a plurality of virtual cameras for simulating and capturing the room 600 at an arbitrary position and orientation in the space model, and outputs the set camera parameters of the virtual camera to the virtual edge image generation unit 537. The position and orientation included in the camera parameters of the virtual camera are candidates for the photographing position and orientation of the surveillance camera 3 to be calibrated. Of the camera parameters, values of the surveillance camera 3 stored in advance are used for the focal length, resolution, lens distortion, etc. other than the position and orientation.

  The virtual camera setting means 536 moves the position of the virtual camera in the X, Y, and Z directions at a predetermined minute interval from the origin in the XYZ coordinate system of the space model, and also the attitude of the virtual camera (pitch angle, yaw angle, roll The angle may be changed at a predetermined minute interval to set a plurality of virtual cameras. However, the setting range based on the installation specifications of the monitoring camera 3 is input in advance from the user interface unit 55, By setting a virtual camera within the range, the amount of camera calibration processing can be reduced. For example, if the installation specification is a specification in which the ceiling is installed vertically downward, the range of movement of the virtual camera in the Z direction and the variation range of the pitch angle can be significantly narrowed by inputting the ceiling height in advance. Also, the X coordinate, Y coordinate, Z coordinate, pitch angle, yaw angle, and roll angle defined in the installation specifications are input in advance, and within a predetermined installation error range centered on each input value at a fixed minute interval. A plurality of virtual cameras may be set, or a plurality of virtual cameras may be set at a minute interval that spreads away from each input value.

  The virtual edge image generation unit 537 generates an edge image by projecting the space model of the room 600 onto the imaging plane represented by each camera parameter set by the virtual camera setting unit 536, and the generated edge image is sent to the edge image matching unit 538. Output. The edge image generated here is called a virtual edge image.

  Specifically, among the three-dimensional coordinate data constituting the space model, three-dimensional coordinate data with an edge presence of “1.0” is projected onto the imaging surface, and a tertiary with an edge presence of “0.0” is projected. By not projecting the original coordinate data, a virtual edge image is generated with the projection position of the three-dimensional coordinate data having an edge presence of 1.0 as an edge pixel on the imaging surface of the virtual camera.

  By defining the edge presence according to the number of corresponding brightness change points using RGB images at multiple measurement positions and orientations as described above, at the boundary of the unmeasured area generated in the spatial model due to occlusion during measurement Real edges can be included in the virtual edge image, while non-existing edges can be eliminated.

  Here, in the monitoring image obtained when the monitoring camera 3 exists at the virtual viewpoint even if the edge actually exists in the three-dimensional coordinate data whose distance from the virtual viewpoint is farther than the distance from the measurement viewpoint. In some cases, the resolution of the pixel position corresponding to the three-dimensional coordinate data is lowered and the edge is not extracted. Therefore, an edge image is generated from a projection image (rendered image) obtained by projecting the luminance data associated with the three-dimensional coordinate data constituting the spatial model onto the imaging plane represented by each virtual viewpoint, and the virtual edge image is generated as the generated edge. You may replace with the logical product image of an image and a virtual edge image. In particular, at this time, for the three-dimensional coordinate data that is far from the virtual viewpoint, a plurality of three-dimensional coordinate data is projected onto one pixel of the rendered image. The virtual edge image generation means 537 sets the average value of the luminance data associated with the plurality of three-dimensional coordinate data to the projection destination pixel. An outlier (for example, RGB values (255, 255, 255)) may be given to the unmeasured area projected on the rendered image. In this way, an edge that does not actually exist can be extracted from the boundary with the unmeasured area in the rendered image, but this edge does not cause a problem because it is deleted by a logical product operation with the virtual edge image. By averaging the luminance data, non-extraction of edges due to a decrease in resolution can be reproduced, and the matching accuracy between the real edge image and the virtual edge image can be improved.

  The edge image collating unit 538 calculates the similarity between the real edge image generated by the real edge image generating unit 535 and the virtual edge image generated by the virtual edge image generating unit 537 for each camera parameter of the virtual camera, and converts the similarity into the real edge image. The position and orientation of the virtual camera that gives a similar virtual edge image is determined as the position and orientation of the monitoring camera 3. For example, the edge image collating unit 538 determines the position and orientation of the virtual camera having the highest similarity as the position and orientation of the monitoring camera 3.

  Here, the similarity is defined to be a value corresponding to the degree of matching of the edge positions. For example, the similarity may be a value obtained by dividing the number of coincidence of edge positions by the number of edge pixels of the virtual edge image or the sum of the number of edge pixels of the real edge image and the number of edge pixels of the virtual edge image. Also, for example, the distance from each edge position of the virtual edge image to the nearest edge position of the real edge image or the distance from each edge position of the real edge image to the nearest edge position of the virtual edge image is accumulated, and the accumulated value A value obtained by dividing the value corresponding to the reciprocal number by the number of edge pixels of the virtual edge image or the sum of the number of edge pixels of the real edge image and the number of edge pixels of the virtual edge image may be used as the similarity. Further, the similarity may be corrected by weighting and adding the matching degree of the luminance data to the similarity. In this case, the normalized correlation value of the pixel value between the corresponding pixels between the monitoring image and the rendered image can be set as the degree of coincidence of the luminance data.

  Incidentally, the 3D model processing apparatus according to the present invention includes a spatial model storage unit 510, a change point association unit 531, an edge presence degree assigning unit 532, a virtual camera setting unit 536, and a virtual edge image generation unit 537. The camera calibration system according to the embodiment includes the three-dimensional model processing apparatus, an actual edge image generation unit 535, and an edge image collation unit 538.

[Operation of camera calibration system 1 when generating spatial model]
FIG. 10 is a schematic flowchart of the operation of the camera calibration system 1 when generating a spatial model. When the three-dimensional measurement device 2 measures the room 600, measurement data is input from the three-dimensional measurement device 2 to the three-dimensional data processing device 5. The three-dimensional data processing device 5 performs a series of processes shown in FIG. 10 on the input measurement data, and generates a space model of the room 600.

  The measurement data input unit 50 receives the measurement data from the three-dimensional measurement apparatus 2, that is, the distance image measured at the first measurement position and orientation, the RGB image captured at the first measurement position and orientation, and the second measurement position and orientation. The measured distance image and the RGB image photographed at the second measurement position and orientation are acquired and input to the control unit 53 (step S10).

  The control unit 53 operates as the coordinate conversion unit 530. The coordinate conversion unit 530 calculates the first measurement position and orientation and the second measurement position and orientation from the distance image acquired in step S10, and the measurement parameter storage unit 511. (Step S11).

  The coordinate conversion means 530 uses the first measurement position and orientation calculated in step S11 and the second measurement position and orientation to convert each pixel of the distance image acquired in step S10 into three-dimensional coordinate data, thereby creating a spatial model. It memorize | stores in the memory | storage part 51 which functions as a memory | storage means 510 (step S12). Further, the correspondence relationship between the pixels of the distance image and the three-dimensional coordinate data is also temporarily stored in the space model storage unit 510 for subsequent processing. In addition, the coordinate conversion unit 530 associates each pixel value of the RGB image acquired in step S10 as luminance data with the three-dimensional coordinate data that is the conversion destination of the pixel of the corresponding distance image, as a spatial model storage unit 510. Remember me. Note that the luminance data for the three-dimensional coordinate data corresponding to the plurality of pixels of the RGB image may be an average pixel value of the corresponding plurality of pixels.

  In addition, the control unit 53 operates as the changing point association unit 531 and performs the processes of steps S13 to S16. First, the change point associating unit 531 sequentially sets the plurality of measurement position / postures acquired in step S11 as the target position / posture (step S13).

  The change point associating means 531 extracts a luminance change point from the RGB image of the target position and orientation among the RGB images acquired in step S10 (step S14). Specifically, the edge strength in each pixel of the RGB image of the target position and orientation is calculated, and a pixel whose edge strength is equal to or greater than the threshold value Ti is extracted as a luminance change point.

  The change point association unit 531 detects the three-dimensional coordinate data corresponding to the luminance change point extracted in step S14 from the space model (step S15).

  The change point associating means 531 confirms whether or not all measurement positions and orientations have been processed (step S16), and if there are unprocessed items (NO in S16), the process returns to step S13, and the next measurement position and orientation is determined. Process.

  On the other hand, when all the measurement positions and orientations have been processed (YES in S16), the control unit 53 operates as the edge presence degree assigning unit 532 and performs the processes in steps S17 to S21.

  The edge presence level assigning unit 532 sequentially sets a plurality of three-dimensional coordinate data constituting the space model as attention data (step S17), refers to the correspondence detected in step S15, and changes the luminance corresponding to the attention data. The number of points (that is, the number of measurement positions and orientations) is counted (step S18). If the number of luminance change points whose correspondence with the attention data is detected is 2 or more, the edge presence degree assigning means 532 stores the edge presence degree 1.0 in association with the attention data in the spatial model storage means 510. On the other hand, if the number of luminance change points at which the correspondence with the attention data is detected is less than 2, the edge presence degree 0.0 is stored in association with the attention data in the space model storage unit 510. Note that the case where the correspondence with the data of interest is not detected is also included in the case of less than 2.

  The edge presence degree assigning means 532 confirms whether or not all the three-dimensional coordinate data have been processed (step S21), and if there is any unprocessed data (NO in S21), the process returns to step S17, and the next three-dimensional Perform coordinate data processing. On the other hand, when all the three-dimensional coordinate data have been processed, the model generation process ends (in the case of YES in S21).

[Operation of camera calibration system 1 during camera calibration]
FIG. 11 is a schematic flowchart of the operation of the camera calibration system 1 during camera calibration. When the monitoring camera 3 installed in the room 600 captures an image of the room 600, a monitoring image is input from the monitoring camera 3 to the three-dimensional data processing device 5. The three-dimensional data processing device 5 performs a series of processes shown in FIG. 11 on the input monitoring image and calibrates the monitoring camera 3.

  The monitoring image input unit 52 acquires a monitoring image from the monitoring camera 3 and inputs it to the control unit 53 (step S50). The control unit 53 operates as the real edge image generation unit 535, and generates an actual edge image by applying an edge operator to each pixel of the monitoring image acquired in Step S50 (Step S51).

  Next, the control unit 53 operates as the virtual camera setting unit 536, and sets a plurality of virtual cameras having different positions and orientations in the coordinate system of the space model (step S52).

  Then, the control unit 53 operates as the virtual edge image generation unit 537, and generates a virtual edge image on the imaging surface of each virtual camera set in step S52 (step S53). Specifically, the virtual edge image generation unit 537 reads the space model of the room 600 from the space model storage unit 510, and the edge presence degree is 1.0 on the photographing surface represented by the camera parameter of each virtual camera set in step S52. Is projected to generate a virtual edge image. In addition, luminance data is rendered on the projection plane, an edge operator is applied to each pixel of the rendered image to generate an edge image, and the virtual edge image is updated to a logical product image with the generated edge image.

  The control unit 53 operates as the edge image collating unit 538, and calculates the similarity between the real edge image generated in step S51 and the virtual edge image for each virtual camera generated in step S53 (step S54). The edge image collating unit 538 detects the position and orientation of the virtual camera for which the maximum similarity is calculated based on the similarity for each virtual camera calculated in step S54, and monitors the detected position and orientation of the virtual camera. 3 is input to the output unit 54 together with the camera ID of the monitoring camera 3 (step S55).

  The position and orientation of the monitoring camera 3 determined here is a highly accurate estimation result obtained using an edge located at the boundary of the unmeasured area in the spatial model.

  The output unit 54 inputs the position and orientation of the monitoring camera 3 and the camera ID to the monitoring device 4. The monitoring device 4 stores the input position and orientation of the monitoring camera 3 and uses them in subsequent monitoring processing.

[Modification]
(1) In the above embodiment, an example in which measurement is performed at two different measurement positions and orientations has been described, but measurement may be performed at three or more measurement positions and orientations.

  (2) In the above embodiment, the example in which the edge presence degree giving unit 532 assigns either “0.0” or “1.0” edge presence degree has been described. It may be given. In that case, the edge presence degree assigning means 532 converts the edge presence degree “0.0” into the three-dimensional coordinate data with the luminance change point correspondence number 0 or 1 and the three-dimensional coordinate data with the luminance change point correspondence number 2, for example. Edge presence degree “0.625” in 3D coordinate data with edge presence “0.5”, luminance change point correspondence number 3 and edge presence degree “0.3” in 3D coordinate data with luminance change point correspondence number 4. Edge presence “0.875” is assigned to 3D coordinate data with 75 ”, luminance change point correspondence number 5 and edge presence degree“ 1.0 ”is assigned to 3D coordinate data with luminance change point correspondence number 6 or more. To do. Then, the edge image collating unit 538 calculates the similarity with the actual edge image by weighting the edge position of the virtual edge image with the edge presence degree.

  (3) In the above embodiment, a space model obtained by measuring a predetermined space is exemplified as a three-dimensional model. However, the three-dimensional model may be an object model obtained by measuring a predetermined object such as a fixture.

  (4) In the above embodiment, the point cloud format three-dimensional model in which the three-dimensional coordinate data represents a point in the three-dimensional space is exemplified, but the mesh model format three-dimensional model including a set of vertex coordinates of the object plane is used. There may be. In this case, for example, an arbitrary vertex coordinate pair is set as three-dimensional coordinate data (that is, three-dimensional coordinate data in units of straight lines), and the edge presence degree giving unit 532 gives edge presence degree to the three-dimensional coordinate data. .

  (5) In the above embodiment, an example in which the number of monitoring cameras 3 is one has been described. However, two or more monitoring cameras may be used, and in this case, the same processing as in the above example may be repeated for each monitoring camera.

  (6) In the above embodiment, the camera calibration system that calibrates the camera in which the photographing position and orientation is fixed is exemplified. However, the present invention is a camera that is installed in a moving body such as a pan / tilt camera, a robot, The present invention can also be applied to a camera calibration system that calibrates a camera that changes its photographing position and orientation, such as a camera held by a person. That is, when these cameras are calibrated, it is sufficient to calibrate each time a captured image after the photographing position and orientation is changed is input.

  (7) In the above embodiment, an example is shown in which a virtual edge image is generated by projecting a three-dimensional model to which edge presence is given. There is an effect of improving the accuracy of alignment and collation. For example, an edge presence is given to each of a spatial model or an object model obtained by measuring from a plurality of different measurement viewpoints, and an edge exists indicating that an edge exists in these multiple spatial models or multiple object models If the positioning is performed with emphasis on the three-dimensional coordinate data to which the degree is given, the positioning accuracy can be improved. Also, for example, when edge recognition is given to an object model obtained by measuring a predetermined object, and object recognition is performed by collating the object model with three-dimensional data obtained by measuring an unknown object, If the collation is performed with emphasis on the three-dimensional coordinate data to which the edge presence level indicating that the image exists, the collation accuracy can be improved.

  (8) In the above-described embodiment, an example in which the camera is calibrated using the virtual edge image has been described. However, the accuracy of object matching can be improved using the virtual edge image. For example, if a virtual edge image is generated by adding an edge presence to an object model obtained by measuring a predetermined object and collated with a real edge image generated from a captured image, the object shown in the captured image is The accuracy of determining whether or not the object is a predetermined object can be improved.

  1 camera calibration system, 2 3D measuring device, 3 monitoring camera, 4 monitoring device, 5 3D data processing device, 50 measurement data input unit, 51 storage unit, 52 monitoring image input unit, 53 control unit, 54 output unit, 55 User interface unit, 510 Spatial model storage unit, 511 Measurement parameter storage unit, 530 Coordinate conversion unit, 531 Change point association unit, 532 Edge presence degree provision unit, 535 Real edge image generation unit, 536 Virtual camera setting unit, 537 Virtual edge image generation means, 538 edge image collation means, 600 rooms, 601 floor, 602 desks, 603 patterns, 650 measurement positions, 651 measurement postures, 652 photographing positions, 653 photographing postures.

Claims (4)

The three-dimensional model composed of a plurality of three-dimensional coordinate data representing the surface shape of the object and obtained by measuring at a plurality of measurement positions and orientations, the plurality of measurement positions and orientations, and the plurality of measurement positions and orientations, respectively. Storage means for storing an image of the subject,
A change point associating means for extracting a brightness change point from each image and detecting three-dimensional coordinate data of a position corresponding to the brightness change point using the measurement position and orientation;
Edge presence giving means for giving an edge presence having a height corresponding to the number of detected correspondences with the luminance change points in the three-dimensional coordinate data;
Three-dimensional model alignment for aligning a plurality of three-dimensional models with different measurement positions and orientations, with the three-dimensional coordinate data having a high edge presence being emphasized over the three-dimensional coordinate data having a low edge presence Means,
A three-dimensional model processing apparatus comprising: The three-dimensional model composed of a plurality of three-dimensional coordinate data representing the surface shape of the object and obtained by measuring at a plurality of measurement positions and orientations, the plurality of measurement positions and orientations, and the plurality of measurement positions and orientations, respectively. Storage means for storing an image of the subject,
A change point associating means for extracting a brightness change point from each image and detecting three-dimensional coordinate data of a position corresponding to the brightness change point using the measurement position and orientation;
Edge presence giving means for giving an edge presence having a height corresponding to the number of detected correspondences with the luminance change points in the three-dimensional coordinate data;
Virtual camera setting means for setting a virtual camera for simulating and photographing the three-dimensional model at an arbitrary position and orientation;
A virtual edge image generating means for generating a virtual edge image having an edge pixel as a projection position of three-dimensional coordinate data in which the edge presence degree is higher than a predetermined height on the imaging surface of the virtual camera;
A three-dimensional model processing apparatus comprising: The three-dimensional model processing device according to claim 2,
An actual image obtained by capturing an image of the object by an actual camera, and an actual edge image generating unit that extracts an edge from the actual image and generates an actual edge image;
Edge image matching means for calculating the degree of similarity between the real edge image and the virtual edge image and determining the position and orientation of the virtual camera that maximizes the degree of similarity as the position and orientation of the real camera;
A camera calibration system characterized by comprising: Stores a 3D model that is composed of a plurality of 3D coordinate data representing the surface shape of the object and is obtained by measurement at a plurality of measurement positions and orientations, and the 3D coordinate data is stored at each of the plurality of measurement positions and orientations. Three-dimensional model storage means to which an edge presence corresponding to the number of detected correspondences with luminance change points extracted from an image obtained by photographing the target is provided;
Virtual camera setting means for setting a virtual camera for simulating and photographing the three-dimensional model at an arbitrary position and orientation;
A virtual edge image generating means for generating a virtual edge image having an edge pixel as a projection position of three-dimensional coordinate data having an edge presence level indicating that an edge exists on the imaging surface of the virtual camera;
A three-dimensional model processing apparatus comprising:

Images