WO2021020062A1 - 三次元変位計測方法及び三次元変位計測装置 - Google Patents

三次元変位計測方法及び三次元変位計測装置 Download PDF

Info

Publication number
WO2021020062A1
WO2021020062A1 PCT/JP2020/026801 JP2020026801W WO2021020062A1 WO 2021020062 A1 WO2021020062 A1 WO 2021020062A1 JP 2020026801 W JP2020026801 W JP 2020026801W WO 2021020062 A1 WO2021020062 A1 WO 2021020062A1
Authority
WO
WIPO (PCT)
Prior art keywords
dimensional
camera
viewpoints
cameras
subject
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/026801
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
徹 松延
敏康 杉尾
哲史 吉川
将貴 福田
研翔 寺西
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Panasonic Intellectual Property Management Co Ltd
Original Assignee
Panasonic Intellectual Property Management Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Panasonic Intellectual Property Management Co Ltd filed Critical Panasonic Intellectual Property Management Co Ltd
Priority to JP2021536879A priority Critical patent/JP7442072B2/ja
Publication of WO2021020062A1 publication Critical patent/WO2021020062A1/ja
Priority to US17/577,733 priority patent/US20220138971A1/en
Anticipated expiration legal-status Critical
Priority to JP2024016929A priority patent/JP7607229B2/ja
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/292—Multi-camera tracking
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G06T2207/10021—Stereoscopic video; Stereoscopic image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30164—Workpiece; Machine component

Definitions

  • the present disclosure relates to a three-dimensional displacement measuring method and a three-dimensional displacement measuring device.
  • 3D reconstruction technology in the field of computer vision, feature points are associated between multiple 2D images, and the position and orientation of the camera and the 3D position of the subject are estimated. Further, in the three-dimensional reconstruction technique, camera calibration and three-dimensional point cloud reconstruction are performed.
  • the device described in Patent Document 1 calibrates between three or more cameras, and converts each camera coordinate system into a virtual camera coordinate system of an arbitrary viewpoint according to the acquired camera parameters.
  • the device estimates the distance information by associating the images after coordinate conversion by block matching.
  • the present disclosure provides a technique capable of effectively measuring the displacement of a subject over time in a three-dimensional space based on a plurality of images obtained by photographing the subject from a plurality of viewpoints.
  • the three-dimensional displacement measuring method is imaged from M (M is an integer larger than N) viewpoints different from each other by a plurality of cameras including N (N is an integer larger than 1) cameras. It is generated by a camera calibration step that calculates camera parameters of the N cameras based on M images for calibration, and (1) imaging a subject by each of the N cameras at different times. It includes a displacement measurement step of measuring the temporal displacement of the subject in a three-dimensional space based on N images and (2) the camera parameters.
  • FIG. 1 is a schematic view when three-dimensional displacement measurement according to the embodiment is performed.
  • FIG. 2 is a diagram showing the principle of three-dimensional modeling in the embodiment.
  • FIG. 3 is a diagram showing the principle of three-dimensional displacement measurement in the embodiment.
  • FIG. 4A is a diagram showing an example of the shooting cycle and the exposure time of the camera according to the embodiment.
  • FIG. 4B is a diagram showing another example of the shooting cycle and the exposure time of the camera according to the embodiment.
  • FIG. 5 is a block diagram of the three-dimensional displacement measuring device according to the embodiment.
  • FIG. 6 is a flowchart of the three-dimensional reconstruction system according to the embodiment.
  • FIG. 7 is a block diagram of the control unit according to the embodiment.
  • FIG. 1 is a schematic view when three-dimensional displacement measurement according to the embodiment is performed.
  • FIG. 2 is a diagram showing the principle of three-dimensional modeling in the embodiment.
  • FIG. 3 is a diagram showing the principle of three-dimensional displacement measurement in the embodiment.
  • FIG. 8 is a flowchart showing the operation of the control unit according to the embodiment.
  • FIG. 9 is a diagram showing a baseline length, a distance between an imaging target and a camera, and a convergence angle used for three-dimensional displacement measurement in the embodiment.
  • FIG. 10 is a diagram showing a method of determining a viewpoint position used for camera calibration in the embodiment.
  • FIG. 11 is a diagram showing an example of a multi-view frame set in the embodiment.
  • FIG. 12 is a block diagram of the three-dimensional image processing unit according to the embodiment.
  • FIG. 13 is a flowchart showing the operation of the three-dimensional image processing unit according to the embodiment.
  • FIG. 14 is a flowchart showing an operation when the three-dimensional displacement measuring device according to the embodiment generates a free viewpoint image.
  • FIG. 9 is a diagram showing a baseline length, a distance between an imaging target and a camera, and a convergence angle used for three-dimensional displacement measurement in the embodiment.
  • FIG. 10 is a diagram showing
  • FIG. 15 is a flowchart showing the operation of the three-dimensional image processing unit when generating a free-viewpoint image.
  • FIG. 16 is a block diagram of a modified example of the three-dimensional image processing unit.
  • FIG. 17 is a flowchart showing the operation of a modified example of the three-dimensional image processing unit.
  • FIG. 18 is a diagram showing an outline of three-dimensional space recognition.
  • three-dimensional space recognition is realized by reconstructing (modeling) the three-dimensional shape of the subject or by using the reconstruction result.
  • the three-dimensional space reconstruction device that reconstructs the three-dimensional shape of the subject is an image data provided by an imaging system including a plurality of cameras that capture images of the same scene, and the position of each camera obtained by calibration. Modeling is performed using camera parameters that indicate the posture and the like (hereinafter referred to as "positional posture"). The position of the camera corresponds to the viewpoint.
  • the measurement target such as a structure (for example, a structure for infrastructure), a mechanical part, etc. in a three-dimensional space.
  • a structure for example, a structure for infrastructure
  • a mechanical part etc.
  • the reduction of the processing load for measuring the displacement of the subject over time in the three-dimensional space has been insufficient.
  • the three-dimensional displacement measurement method captures images from M (M is an integer larger than N) viewpoints different from each other by a plurality of cameras including N (N is an integer larger than 1) cameras.
  • M is an integer larger than N
  • N is an integer larger than 1
  • a camera calibration step that calculates the camera parameters of the N cameras based on the M images for calibration, and (1) imaging the subject with each of the N cameras at different times. It includes a displacement measurement step of measuring the temporal displacement of the subject in a three-dimensional space based on (2) the camera parameters.
  • the number of viewpoints M which is larger than the number of viewpoints N in the three-dimensional displacement measurement process, is used in the camera calibration process so as to improve the accuracy of the camera parameters.
  • the accuracy can be improved.
  • the displacement measurement step includes the first corresponding point of the subject included in the first image included in the N images and the N.
  • the displacement of the three-dimensional point over time is obtained based on the camera parameters, the first corresponding point, and the second corresponding point. Therefore, the displacement of the subject in the three-dimensional space can be obtained.
  • the N may be determined based on the camera parameters and the accuracy of the displacement measurement.
  • the three-dimensional displacement measuring method can accurately obtain the displacement of the subject in the three-dimensional space.
  • the inventors of the present application have found that the accuracy of camera calibration is improved by arranging the viewpoints used for camera calibration as follows rather than arranging them evenly.
  • the N cameras include a first camera and a second camera adjacent to each other, and the M viewpoints are arranged with the first camera.
  • the first midpoint between the first position and the second position where the second camera is arranged is included, and the midpoint between the first position and the first midpoint is defined as the second midpoint.
  • the viewpoints included in the M viewpoints and provided between the second midpoint and the first position are included in the M viewpoints and the first midpoint and the second midpoint are included. It may be more than the viewpoint provided between and.
  • the viewpoint used for camera calibration is set in a range closer to the second midpoint and more than a range farther than the second midpoint with respect to the position of the viewpoint used for three-dimensional displacement measurement. .. Due to such uneven arrangement, the accuracy of camera calibration is improved.
  • the three-dimensional displacement measuring device has M (M is an integer larger than N) viewpoints different from each other depending on a plurality of cameras including N (N is an integer larger than 1) cameras.
  • M is an integer larger than N
  • N is an integer larger than 1
  • the camera calibration unit that calculates the camera parameters of the N cameras based on the M images captured in (1) and the N cameras at different times. It includes N images to be generated, and a displacement measuring unit that measures the temporal displacement of the subject in a three-dimensional space based on (2) the camera parameters.
  • the three-dimensional displacement measuring device can exert the same effect as the three-dimensional displacement measuring method according to one aspect of the present disclosure.
  • the three-dimensional displacement measuring device can reconstruct a time-series three-dimensional model in which the coordinate axes match between times.
  • the three-dimensional reconstruction in the present disclosure is defined.
  • An image or image of a subject existing in a real space taken by a plurality of cameras from different viewpoints is called a multi-view image or a multi-view image. That is, the multi-viewpoint image includes a plurality of two-dimensional images of the same subject taken from different viewpoints.
  • multi-viewpoint images taken in time series are called multi-viewpoint images. Reconstructing a subject in a three-dimensional space using this multi-viewpoint image is called three-dimensional reconstruction.
  • the three-dimensional displacement measuring device acquires a three-dimensional model at each time by independently performing three-dimensional reconstruction for each time.
  • the three-dimensional displacement measuring device detects a stationary camera and a stationary object (stationary three-dimensional point), and uses the detected stationary camera and the stationary object to adjust the coordinates of the three-dimensional model between times to obtain the coordinate axes. Generate a matched time series 3D model.
  • the three-dimensional displacement measuring device can use the transition information in the time direction with high accuracy in the relative positional relationship between the subject and the camera at each time regardless of whether the camera is fixed / non-fixed or the subject is moving / stationary.
  • a possible time series 3D model can be generated.
  • FIG. 1 is a schematic view when the three-dimensional displacement measurement in the embodiment is performed.
  • the subject 1001 is photographed by the camera 100-1 and the camera 100-2.
  • the subject 1001 may be a compressor or the like fixed at a predetermined position.
  • the camera 100-1 and the camera 100-2 may be fixed cameras fixed in directions so as to include the subject 1001 in the shooting range at different positions. For example, when the same space or the same subject 1001 is photographed from multiple viewpoints by one or more calibrated cameras (for example, a fixed camera), the space or the subject 1001 to be photographed is three-dimensionally reconstructed. Can be done.
  • the position of the camera, the orientation of the camera, and the three-dimensional displacement or three-dimensional position of the subject 1001 can be determined. It is possible to estimate. That is, in the three-dimensional displacement measuring device, camera calibration and three-dimensional displacement measurement of the subject 1001 are performed using a plurality of two-dimensional images taken from multiple viewpoints. In the three-dimensional displacement measuring device, the subject 1001 may be further three-dimensionally reconstructed.
  • Three-dimensional displacement measurement is useful for, for example, infrastructure structure inspection systems.
  • a measuring device to measure the vibration of a wall such as a soundproof wall such as the Shinkansen in the depth direction of the wall.
  • a measuring device to measure the displacement of a device that generates vibration in the operation of a compressor or the like.
  • FIG. 2 is a diagram showing the principle of three-dimensional modeling in the embodiment.
  • the three-dimensional modeling in the present embodiment is defined.
  • the three-dimensional displacement measuring device reconstructs the subject in the three-dimensional space by using the two-dimensional multi-viewpoint images of the subject existing in the real space taken from different viewpoints by a plurality of cameras. This is called 3D modeling.
  • 3D modeling may also be referred to as 3D reconstruction.
  • each point on the image plane represented by the coordinate system on the image plane of the image taken by each camera is based on the principle of triangulation using this triangle.
  • the coordinate system on the image plane is, for example, the image plane coordinates 1 or the image plane coordinates 2 shown in FIG.
  • a three-dimensional model is a subject reconstructed (that is, three-dimensionally reconstructed) in a three-dimensional space represented using a world coordinate system.
  • the three-dimensional model represents the three-dimensional position of each of a plurality of points on a subject reflected in a plurality of two-dimensional images taken from multiple viewpoints.
  • the three-dimensional position is represented by, for example, ternary information including an X component, a Y component, and a Z component represented by a Cartesian coordinate system (XYZ axes) in a three-dimensional space.
  • the information corresponding to each point on the three-dimensional model includes not only the information representing the three-dimensional position of each point described above, but also the color of each point and the information representing the surface shape of each point and its surroundings. It may be.
  • the camera parameters of each camera may be acquired in advance, or may be estimated at the same time as the creation of the 3D model.
  • the camera parameter includes an internal parameter and an external parameter.
  • Internal parameters include the focal length of the camera, the center of the image, etc.
  • external parameters include the three-dimensional position of the camera, the orientation of the camera in three dimensions, and the like.
  • a typical pinhole camera model is used as an example.
  • camera lens distortion is not considered.
  • the position of each point in the coordinate system on the image plane of the image taken by each camera is normalized by the distortion model as the position of each point used in the 3D model. The corrected position is used.
  • FIG. 3 is a diagram showing the principle of three-dimensional displacement measurement in the embodiment.
  • the three-dimensional displacement measurement method of the present disclosure is defined. Calculation of the displacement of the subject 1001 in the three-dimensional space using a two-dimensional image taken from multiple viewpoints is called three-dimensional displacement measurement.
  • FIG. 3 shows the mechanism of three-dimensional displacement measurement.
  • a triangle as shown by the dotted line in FIG. 3 is formed. Based on the principle of triangulation in which this triangle is used, points on the image taken from a plurality of viewpoints and represented by the coordinate system on the image plane are reconstructed in the world coordinate system.
  • the coordinate system on the image plane is, for example, the image plane coordinates 1 or the image plane coordinates 2 shown in FIG.
  • the three-dimensional model of the subject 1001 is indicated by the three-dimensional positions of the plurality of points on the subject 1001 reflected in the plurality of two-dimensional images taken from multiple viewpoints.
  • the positions of the three-dimensional points (X t , Y t , Z t ) and (X t + i , Y t + i , Z t + i ) on the subject 1001 are re-established at the time t and the time t + i, respectively.
  • the displacements during time i (X t + i- X t , Y t + i- Y t , Z t + i- Z t ) are calculated.
  • the depth direction Z component
  • a so-called depth map may be created.
  • the displacement in any two components of the X component, the Y component, and the Z component may be measured.
  • the time difference i is the reciprocal of the frequency at which the resonance frequency of the vibrating body can be sampled.
  • the time difference i is not limited to this.
  • the time difference i is determined according to the durability required for the structure.
  • the positions of the three-dimensional points were reconstructed at time t and time t + i.
  • the number of times the three-dimensional point position is reconstructed is not limited to two, and may be three or more for each time difference i.
  • the three-dimensional displacement measurement of the present disclosure may calculate not only the displacement of the three-dimensional point but also the velocity and acceleration of the displacement. Good.
  • the matrix representing the internal parameters of the camera is K 1
  • the translation vector representing the translation of the external parameters is T 1
  • the rotation matrix representing the rotation is R 1
  • the matrix representing the internal parameters of the camera is K 2
  • the translation vector representing the translation of the external parameters is T 2
  • the rotation matrix representing the rotation is R 2 .
  • the corresponding point between the image plane coordinates 1 and the image plane coordinates 2 is calculated for a certain point on the subject 1001 by image matching.
  • block matching may be performed to search for the position where the difference value of the block, which is a rectangular region in the image, is minimized.
  • the degree of matching of the associated rectangular areas included in each image is the sum of the brightness differences (SAD: Sum of Absolute Evaluation) or the sum of squares of the brightness differences (SSD:).
  • SAD Sum of Absolute Evaluation
  • SSD sum of squares of the brightness differences
  • cross-correlation Cross Correlation
  • NCC Normalized Cross Correlation
  • image matching a feature amount that calculates a special quantity vector from a small area of an image such as SIFT (Scale-Invariant Features Transfer) and SURF (Speeded Up Robot Features), and searches for a position where the difference value is the minimum. Matching may be done.
  • SIFT Scale-Invariant Features Transfer
  • SURF Speeded Up Robot Features
  • the corresponding points a 1 and t in the image plane coordinates 1 are specific examples of the first corresponding points, and the corresponding points b 2 and t in the image plane coordinates 2 are specific examples of the second corresponding points.
  • the process described here corresponds to the corresponding point calculation step.
  • a three-dimensional point representing a three-dimensional position corresponding to each of a 1, t , b 2, t , a 1, t + i , and b 2, t + i is calculated.
  • a 1 a three-dimensional point corresponding to a 1, t at time t, t, 2 a three-dimensional point corresponding to b 2, t at time t B, t, the time t + i a 1, t + corresponding three-dimensional pointer i to Let the points be A 1, t + i , and the three-dimensional points corresponding to b 2, t + i at time t + i be B 2, t + i .
  • the three-dimensional points A 1, t and B 2, t corresponding to the corresponding points on the two-dimensional image are the same points in the world coordinate system, but the equations (1) and (2) are used.
  • the three-dimensional points A 1, t and B 2, t calculated as described above do not always match.
  • the three-dimensional points A 1, t + i and B 2, t + i are the same points in the world coordinate system, but the three-dimensional points A 1, t + i calculated using the equations (3) and (4).
  • B 2, t + i do not always match.
  • the midpoint between the three-dimensional point corresponding to the corresponding point on the two-dimensional image may be calculated.
  • the method of determining the three-dimensional points when the three-dimensional points corresponding to the corresponding points on the two-dimensional image do not match is not limited to this. Only one of the three-dimensional points A 1, t and B 2, t , or one of the three-dimensional points A 1, t + i and B 2, t + i , is placed on the two-dimensional image corresponding to each time. It may be used as a three-dimensional point corresponding to the corresponding point. Further, although the method of calculating the midpoint between the two three-dimensional points is shown in the formulas (5) and (6), other calculation methods may be used.
  • three-dimensional point in time t obtained by weighted averaging the three-dimensional point A 1, t and B 2, t, or by weighted averaging the three-dimensional point A 1, t + i and B 2, t + i
  • the obtained three-dimensional points at the time t + i may be used as the three-dimensional points corresponding to the respective times.
  • the process described here corresponds to the three-dimensional point calculation step.
  • the calculated three-dimensional point is not limited to a specific point of the subject.
  • the three-dimensional point is an arbitrary point of the subject as long as it can be obtained from the corresponding point.
  • V t, t + i (X t + i- X t , Y t + i- Y t , Z t + i- Z t ) Equation (7)
  • the above three-dimensional displacement measurement method is an example, and is not limited to this.
  • FIG. 4A is a diagram showing an example of the shooting cycle and the exposure time of the camera according to the embodiment.
  • FIG. 4B is a diagram showing another example of the shooting cycle and the exposure time of the camera according to the embodiment.
  • the lateral direction of FIGS. 4A and 4B indicates the time, and the time during which the rectangular signal is standing indicates that the camera 100-1 or the camera 100-2 is exposed.
  • the exposure time When an image is acquired by the camera 100-1 or the camera 100-2, the time when the shutter is open is called the exposure time.
  • a scene exposed to the image sensor through the lens can be obtained as an image.
  • the exposure times of the frames taken by the two cameras 100-1 and 100-2 having different viewpoints overlap.
  • the frames acquired by the two cameras 100-1 and 100-2 are determined to be synchronized frames containing the scenes at the same time.
  • FIG. 4B since the exposure times of the two cameras 100-1 and 100-2 do not overlap, the frames acquired by the two cameras 100-1 and 100-2 do not include the scenes at the same time. Judged as an asynchronous frame. Shooting a synchronized frame with a plurality of cameras as shown in FIG. 4A is called synchronous shooting.
  • FIG. 5 is a block diagram of the three-dimensional displacement measuring device according to the embodiment.
  • the three-dimensional reconstruction system 3 shown in FIG. 5 includes a plurality of cameras 100-1 to 100-n, a three-dimensional displacement measuring device 2000, and a control unit 300. Further, the three-dimensional displacement measuring device 2000 includes a receiving unit 210, a storage unit 220, an acquisition unit 230, a three-dimensional image processing unit 240, and a transmitting unit 250.
  • the plurality of cameras 100-1 to 100-n shoot the subject 1001 in the shooting space 1000, and transmit the multi-viewpoint video, which is a plurality of shot images, to the receiving unit 210.
  • the multi-viewpoint video may be transmitted via either a public communication network such as the Internet or a dedicated communication network.
  • the multi-viewpoint video may be once stored in an external storage device such as a hard disk drive (HDD) or a solid state drive (SSD), and input to the three-dimensional displacement measuring device 2000 when necessary.
  • the multi-viewpoint video is once transmitted to an external storage device such as a cloud server via a network and stored. Then, it may be transmitted to the three-dimensional displacement measuring device 2000 when necessary.
  • each of the n cameras 100-1 to 100-n is a fixed camera such as a surveillance camera or a non-fixed camera that is not fixed. That is, the n cameras 100-1 to 100-n may be, for example, fixed cameras fixed at different positions and in different postures, or a video camera, a smartphone, a wearable camera, or the like. It may be a mobile camera of the above, or a mobile camera such as a drone with a shooting function. Note that n is an integer of 2 or more.
  • camera-specific information such as a camera ID that identifies the camera that captured the image may be added to the multi-viewpoint video as header information of the video or frame.
  • Synchronous shooting may be performed by using a plurality of cameras 100-1 to 100-n to shoot a subject at the same time in each frame.
  • the time of the clocks built into the plurality of cameras 100-1 to 100-n may be adjusted, and the shooting time information may be added to the shot image for each video or frame without synchronous shooting.
  • An index number indicating the shooting order may be added.
  • Information indicating whether the images were shot synchronously or asynchronously may be added as header information for each video set, video, or frame of the multi-view video.
  • control unit 300 is realized by using, for example, a CPU (Central Processing Unit), and reads and executes a program stored in the storage unit 220.
  • the control unit 300 is used by the 3D reconstruction system 3 to perform one or more of camera calibration, 3D modeling, and 3D displacement measurement in the 3D image processing unit 240.
  • the number of viewpoints is the number of positions for capturing a two-dimensional image used in one or more processes of camera calibration, three-dimensional modeling, and three-dimensional displacement measurement.
  • the camera It may be the number of points where the image is taken by the fixed camera and the moving camera.
  • the viewpoint position is a position for capturing a two-dimensional image used in one or more processes of camera calibration, three-dimensional modeling, and three-dimensional displacement measurement, and specifically, the position of the camera. Is.
  • the receiving unit 210 receives the multi-viewpoint video captured by the cameras 100-1 to 100-n, and inputs the received multi-viewpoint video to the storage unit 220.
  • the storage unit 220 stores the input multi-viewpoint video. Further, in the storage unit 220, video data taken by the shooting space 1000, meta information such as a time stamp attached to the video data, camera parameters of cameras 100-1 to 100-n, and a frame being applied. Shooting settings such as rate or resolution are temporarily stored. Further, the storage unit 220 stores a program read and executed by the control unit 300.
  • the storage unit 220 may be a volatile memory such as RAM, DRAM, or SRAM. Further, the storage unit 220 may be realized by a non-volatile memory such as a ROM or a flash memory.
  • the acquisition unit 230 selects a frame from a plurality of multi-viewpoint images stored in the storage unit 220, and inputs the selected frame as a multi-viewpoint frame set to the three-dimensional image processing unit 240.
  • the three-dimensional image processing unit 240 performs camera calibration using the input multi-view frame set, and performs three-dimensional modeling of the subject or three-dimensional displacement measurement of the subject using the result of the camera calibration.
  • the three-dimensional image processing unit 240 calculates the camera parameters as a result of performing the camera calibration and outputs the camera parameters to the transmission unit 250.
  • the transmission unit 250 transmits at least one of the camera parameters calculated by the three-dimensional image processing unit 240, the three-dimensional model of the subject, and the three-dimensional displacement of the subject to the outside of the three-dimensional displacement measuring device 2000.
  • the receiving unit 210, the acquiring unit 230, the three-dimensional image processing unit 240, and the transmitting unit 250 may be realized by a processor and a memory, respectively. Further, the receiving unit 210, the acquiring unit 230, the three-dimensional image processing unit 240, and the transmitting unit 250 may be realized by a dedicated circuit.
  • FIG. 6 is a flowchart of the three-dimensional reconstruction system according to the embodiment.
  • control unit 300 determines the number of viewpoints and the viewpoint position used by the three-dimensional reconstruction system 3 to perform one or more processes of camera calibration, three-dimensional modeling, or three-dimensional displacement measurement. (Step S2100). Details of determining the number of viewpoints and the position of viewpoints will be described later.
  • the cameras 100-1 to 100-n capture a multi-viewpoint image of the subject based on the number of viewpoints and the viewpoint position determined by the control unit 300 (step S2200).
  • the receiving unit 210 receives the multi-viewpoint video captured by a plurality of cameras 100-1 to 100-n and inputs it to the storage unit 220 (step S2300).
  • the storage unit 220 stores the input multi-viewpoint video (step S2400).
  • the acquisition unit 230 selects a frame from the stored multi-viewpoint video based on the number of viewpoints and the viewpoint position determined by the control unit 300, and uses the selected frame as the multi-viewpoint frame set as the three-dimensional image processing unit 240. Is input to (step S2500).
  • the multi-view frame set may be composed of a plurality of frames selected by one frame from all the images captured by the cameras 100-1 to 100-n. Further, the multi-view frame set may be composed of a plurality of frames selected by at least one frame from all the images captured by the cameras 100-1 to 100-n. Further, the multi-view frame set may be composed of a plurality of frames in which two or more images are selected from the multi-view images and one frame is selected from each selected image. Further, the multi-view frame set may be composed of a plurality of frames in which two or more images are selected from the multi-view images and at least one frame is selected from each of the selected images.
  • the camera specific information may be added individually to the header information of each frame, or the camera specific information may be collectively added to the header information of the multi-view frame set. The camera specific information may be added.
  • the index number indicating the shooting time or shooting order may be individually added to the header information of each frame. In the same case, an index number indicating the shooting time or shooting order may be collectively added to the header information of the multi-view frame set.
  • the three-dimensional image processing unit 240 performs camera calibration using the input multi-viewpoint frame set, and performs three-dimensional modeling or three-dimensional displacement measurement using the result of the camera calibration (step S2600). Details of 3D modeling or 3D displacement measurement will be described later.
  • steps S2500 and S2600 are repeated for each multi-view frame set.
  • the camera calibration may be performed only once.
  • the result of the camera calibration performed once at the beginning may be repeatedly used in the repeated steps S2500 and S2600.
  • the transmission unit 250 transmits the camera parameters, the three-dimensional model of the subject, or the three-dimensional displacement of the subject to the outside of the three-dimensional displacement measuring device 2000 (step S2700).
  • the three-dimensional reconstruction system 3 ends the process.
  • the camera parameters calculated in step S2600, the three-dimensional model of the subject, or the three-dimensional displacement of the subject may be used in the processing after step S2600.
  • the camera parameter, the three-dimensional model of the subject, or the three-dimensional displacement of the subject output in step S2700 may be stored in the storage unit 220.
  • FIG. 7 is a block diagram of the control unit according to the embodiment.
  • the control unit 300 includes a measurement determination unit 310 and a camera calibration determination unit 320.
  • the measurement determination unit 310 and the camera calibration determination unit 320 may be realized by a processor and a memory, or may be realized by a dedicated logic circuit.
  • the measurement determination unit 310 determines the number of viewpoints, the viewpoint position, and the like for capturing a two-dimensional image used by the three-dimensional reconstruction system 3 when measuring the three-dimensional displacement of the subject 1001.
  • the camera calibration determination unit 320 determines the number of viewpoints, the viewpoint position, and the like of the viewpoints for capturing the two-dimensional image used by the three-dimensional reconstruction system 3 when performing camera calibration.
  • the number of viewpoints used by the three-dimensional reconstruction system 3 for performing camera calibration, three-dimensional modeling, and three-dimensional displacement measurement may be different.
  • the control unit 300 may determine only the viewpoint position. That is, the measurement determination unit 310 may determine only the viewpoint position of the viewpoint for capturing the two-dimensional image used when the three-dimensional reconstruction system 3 performs the three-dimensional displacement measurement of the subject 1001. Further, the camera calibration determination unit 320 may determine only the viewpoint position of the viewpoint for capturing the two-dimensional image used when the three-dimensional reconstruction system 3 performs camera calibration.
  • control unit 300 may determine the number of viewpoints and the viewpoint position of the camera before the camera is introduced into the three-dimensional displacement measuring device 2000. Further, the camera may be introduced according to the number of viewpoints and the viewpoint position of the camera determined by the control unit 300.
  • the control unit 300 determines the number of viewpoints N used for three-dimensional displacement measurement and the number of viewpoints M used for camera calibration so as to satisfy the relationship of M> N. Further, the control unit 300 has a viewpoint used for the three-dimensional displacement measurement so that the viewpoint used for the three-dimensional displacement measurement is included in the viewpoint used for the camera calibration (that is, the two viewpoints match). Determine the viewpoint used for camera calibration.
  • the cameras 100-1 to 100-n, or the acquisition unit 230 use N fixed cameras as N viewpoints used for three-dimensional displacement measurement. In other words, because of the three-dimensional displacement measurement, images are not taken by the camera from MN viewpoints.
  • (MN) fixed cameras are used as (MN) viewpoints other than the N viewpoints used for three-dimensional displacement measurement.
  • a mobile camera of (MN) or less may be used. When using a mobile camera with (MN) units or less, the mobile camera with (MN) units or less can shoot from (MN) or more viewpoints by shooting while moving. Realize.
  • (MN) viewpoints other than the N viewpoints used for three-dimensional displacement measurement may be realized by combining a fixed camera and a moving camera. Good.
  • FIG. 8 is a flowchart showing the operation of the control unit according to the embodiment.
  • the measurement determination unit 310 determines the number of viewpoints and the viewpoint position for three-dimensional displacement measurement (step S3100).
  • the camera calibration determination unit 320 determines the number of viewpoints and the viewpoint position for camera calibration (step S3200). At this time, the camera calibration determination unit 320 is set so that the number of viewpoints M for camera calibration is larger than the number of viewpoints N for three-dimensional displacement measurement, and the viewpoint for three-dimensional displacement measurement is changed to the viewpoint for camera calibration. Set to be included.
  • the control unit 300 ends the operation.
  • the measurement determination unit 310 may determine the number of viewpoints and the viewpoint position for three-dimensional modeling.
  • FIG. 9 is a diagram showing a baseline length, a distance between an imaging target and a camera, and a convergence angle used for three-dimensional displacement measurement in the embodiment.
  • FIG. 9 the positional relationship between the subject 1001 to be imaged and the two cameras 100-1 and 100-2 out of the n cameras 100-1 to 100-n is shown.
  • the control unit 300 sets the number of viewpoints used for 3D displacement measurement as N, the distance between the subject 1001 and the cameras 100-1 and 100-2 as d, and the accuracy of 3D modeling or 3D displacement measurement as ⁇ z. These values are arbitrary.
  • the number of viewpoints N may be the number of cameras that can be used for three-dimensional displacement measurement, or may be the maximum number within a settable range. In the following description, it is assumed that the number of viewpoints N is equal to the number n of cameras 100. However, the number of viewpoints N does not have to be equal to the number n of cameras 100.
  • the distance d between the subject 1001 and the cameras 100-1 and 100-2 is the size of the shooting space 1000, how the subject 1001 is reflected in the two-dimensional image captured by the cameras 100-1 and 100-2, and the like. It may be determined from at least one of them.
  • the distance d is the shortest distance from the line passing through the positions of the two cameras 100-1 and 100-2 to the subject 1001.
  • the distance d is not limited to the example shown in FIG.
  • the distance d may be the average value of the distances from the subject 1001 to the two cameras 100-1 and 100-2.
  • the control unit 300 uses the accuracy ⁇ z of the three-dimensional modeling or the three-dimensional displacement measurement, the focal length f of the camera, the element spacing ⁇ H of the camera, and the distance d between the subject 1001 and the two cameras 100-1 and 100-2.
  • the focal length L which is the distance between the two cameras 100-1 and 100-2, is calculated by the equation (8).
  • control unit 300 uses the baseline length L and the distance d between the subject and the camera as information for determining the viewpoint positions of the two cameras.
  • the control unit 300 When the number of viewpoints N used for the three-dimensional displacement measurement is larger than 2, the control unit 300 performs the following calculation. First, the control unit 300 sets the convergence angle of the optical axis of each of the two cameras as ⁇ , and calculates ⁇ by the equation (9) using the baseline length L and the distance d.
  • control unit 300 calculates the value P by the equation (10).
  • the value P is a value indicating the number of cameras used in the three-dimensional reconstruction system 3, which is determined according to the convergence angle ⁇ .
  • control unit 300 calculates N by the equation (11).
  • control unit 300 determines the number of viewpoints N based on the camera parameters and the required accuracy ⁇ z.
  • the baseline length L, the distance d, and the convergence angle ⁇ may be used as information for determining the viewpoint position of the camera.
  • the positional relationship between the subject 1001 and the two cameras 100-1 and 100-2 described above is not limited to the two cameras 100-1 and 100-2, and the n cameras 100-1 to 100-2. The same applies to any combination of any two of 100-n.
  • FIG. 10 is a diagram showing a method of determining a viewpoint position used for camera calibration in the embodiment.
  • FIG. 10 is a view of a plurality of cameras viewed along the vertical direction.
  • the control unit 300 arbitrarily sets the convergence angle ⁇ between the viewpoints of the viewpoints used for camera calibration.
  • ⁇ > 2 ⁇ (N-1) ⁇ ⁇ the control unit 300 calculates the number of viewpoints M used for camera calibration by the equation (12).
  • floor ( ⁇ / ⁇ ) viewpoints are added between each of the N viewpoints used in the three-dimensional displacement measurement.
  • the additional floor ( ⁇ / ⁇ ) viewpoints are added between each of the N viewpoints used in the three-dimensional displacement measurement, as shown in FIG.
  • the control unit 300 adds one or two viewpoints to the intermediate position c between the viewpoint position p1 and the viewpoint position p2.
  • the control unit 300 sets the position bisected between the viewpoint position p1 and the intermediate position c as the position p3, and the position bisected between the viewpoint position p2 and the intermediate position c as the position p4. ..
  • the control unit 300 arranges the number of viewpoints arranged at the intermediate position c between the viewpoint positions p1 and the position p3 by subtracting the number of viewpoints arranged from the floor ( ⁇ / ⁇ ) viewpoints. Similarly, the control unit 300 arranges the number of viewpoints arranged at the intermediate position c between the viewpoint positions p2 and the position p4 by subtracting the number of viewpoints arranged from the floor ( ⁇ / ⁇ ) viewpoints.
  • the majority of the viewpoints to be added are more than the midpoints of the cameras arranged at the positions corresponding to the viewpoints used in the three-dimensional displacement measurement. It is placed closer to the camera placed at the position corresponding to the viewpoint used in the three-dimensional displacement measurement.
  • the number of calibration viewpoints arranged between the position p3 (p4) and the viewpoint position p1 (p2) is larger than the number of calibration viewpoints arranged between the intermediate position c and the position p3 (p4). .. Further, in another arrangement example, a viewpoint for calibration is not provided between the position p3 (p4) and the intermediate position c.
  • the viewpoint used for camera calibration is closer to the position p3 (p4) and farther than the position p3 (p4) with respect to the position of the viewpoint used for the three-dimensional displacement measurement. Set more than the range. Due to such uneven arrangement, the accuracy of camera calibration is improved.
  • FIG. 11 is a diagram showing an example of a multi-view frame set in the embodiment.
  • the acquisition unit 230 determines the multi-viewpoint frame set by selecting one frame from each of the five cameras 100-1 to 100-5.
  • each frame camera IDs that identify the photographed camera are assigned as 100-1 to 100-5, respectively. Further, frame numbers 001 to Q indicating the shooting order in each camera are given to the header information of each frame, and the frame having the same frame number between the cameras shot the subject 1001 at the same time. Is shown.
  • the acquisition unit 230 sequentially outputs the multi-view frame sets 200-1 to 200-Q to the three-dimensional image processing unit 240.
  • the three-dimensional image processing unit 240 sequentially performs three-dimensional modeling or three-dimensional displacement measurement using the multi-view frame sets 200-1 to 200-Q by iterative processing.
  • the multi-view frame set 200-1 includes frame number 001 of camera 100-1, frame number 001 of camera 100-2, frame number 001 of camera 100-3, frame number 001 of camera 100-4, and camera 100-5. It is composed of five frames with frame number 001. As a set of the first frames of the multi-viewpoint video, these five frames are used in the iterative process 1 of the three-dimensional image processing unit 240. That is, in the iterative process 1, the three-dimensional model at the time when the frame number 001 is photographed is reconstructed, or the three-dimensional displacement is measured.
  • the multi-view frame set 200-2 includes a frame number 002 of the camera 100-1, a frame number 002 of the camera 100-2, a frame number 002 of the camera 100-3, a frame number 002 of the camera 100-4, and a frame number 002 of the camera 100-5. It is composed of five frames with frame number 002. These five frames are used in the iterative process 2 of the three-dimensional image processing unit 240. That is, in the iterative process 2, the three-dimensional model at the time when the frame number 002 is photographed is reconstructed, or the three-dimensional displacement is measured.
  • the three-dimensional reconstruction system 3 can reconstruct the three-dimensional model of the subject 1001 at each time, and can measure the three-dimensional displacement of the subject 1001 at each time.
  • the 3D reconstruction is performed independently at each time, the coordinate axes and scales of the reconstructed 3D models do not always match. That is, in order to acquire a three-dimensional model of the moving subject 1001, it is necessary to match the coordinate axes and scales of each time.
  • a shooting time is assigned to each frame, and the acquisition unit 230 creates a multi-view frame set that combines a synchronous frame and an asynchronous frame based on the shooting time.
  • the acquisition unit 230 creates a multi-view frame set that combines a synchronous frame and an asynchronous frame based on the shooting time.
  • the shooting time of the frame selected from the camera 100-1 is T1
  • the shooting time of the frame selected from the camera 100-2 is T2
  • the exposure time of the camera 100-1 is TE1
  • the exposure time of the camera 100-2 is TE2.
  • the shooting times T1 and T2 refer to the time when the exposure is started in the examples of FIGS. 4A and 4B, that is, the time when the rectangular signal rises.
  • the exposure end time of the camera 100-1 is T1 + TE1.
  • the equation (13) or the equation (14) is satisfied, it means that the two cameras 100-1 and 100-2 are shooting the subject 1001 at the same time, and the two frames. Is determined to be a synchronous frame.
  • the acquisition unit 230 When the cameras 100-1 to 100-n shoot a multi-viewpoint image based on the number of viewpoints and the viewpoint position determined by the control unit 300, the acquisition unit 230 has the same number as the number of input multi-viewpoint images.
  • the frame may be a multi-view frame set.
  • the acquisition unit 230 determines the number of viewpoints or viewpoints determined by the control unit 300.
  • a multi-view frame set may be created by selecting an image corresponding to the position.
  • the three-dimensional displacement measuring device 2000 sets N images captured at different times as one set.
  • the three-dimensional displacement measuring device 2000 measures the three-dimensional position of the subject 1001 at the time of the set by using the N images included in the set and the camera parameters for each set of the plurality of sets.
  • FIG. 12 is a block diagram of the three-dimensional image processing unit according to the embodiment.
  • the three-dimensional image processing unit 240 includes a camera calibration unit 410 and a three-dimensional processing unit 420.
  • the three-dimensional processing unit 420 is a specific example of the displacement measurement unit.
  • the camera calibration unit 410 calibrates the camera. Specifically, the camera calibration unit 410 calculates the internal parameters, external parameters, and lens distortion coefficients of the plurality of cameras 100-1 to 100-n as camera parameters.
  • the internal parameters indicate the characteristics of the optical system such as the focal length, aberration, and image center of the camera.
  • the external parameters indicate the position and orientation of the camera in three-dimensional space.
  • the camera calibration unit 410 uses M images for calibration, which are M frames obtained by photographing the black and white intersections of the checker boards by a plurality of cameras 100-1 to 100-n, to display internal parameters and external parameters.
  • the parameters and the lens distortion coefficient may be calculated separately, or the internal parameters, the external parameters, and the lens distortion coefficient are collectively calculated using the corresponding points between the M frames such as the Structure from Motion, and the overall optimum is achieved. It may be converted. In the latter case, the M frames do not have to be the image captured by the checker board.
  • the three-dimensional processing unit 420 performs three-dimensional processing such as three-dimensional modeling of the subject 1001 and three-dimensional displacement measurement of the subject 1001 by using the multi-view frame set and camera parameters. Specifically, the three-dimensional processing unit 420 generates a model of the subject 1001 from three-dimensional points as three-dimensional modeling using the results of the multi-view frame set and camera calibration. Instead of the three-dimensional modeling, the three-dimensional processing unit 420 may measure the three-dimensional displacement of the subject 1001 by using the result of the multi-view frame set and the camera calibration as the three-dimensional displacement measurement. When measuring the three-dimensional displacement, the three-dimensional processing unit 420 may use the data of the three-dimensional points of the three-dimensional model of the subject 1001 generated by the three-dimensional modeling.
  • the three-dimensional point used in the three-dimensional displacement measurement may be an arbitrary point of the subject 1001.
  • the three-dimensional image processing unit 240, and the camera calibration unit 410 and the three-dimensional processing unit 420 that constitute the three-dimensional image processing unit 240 are realized by a processor and a memory. Further, these may be realized by a dedicated logic circuit.
  • FIG. 13 is a flowchart showing the operation of the three-dimensional image processing unit according to the embodiment. In the process shown in FIG. 13, a multi-view frame set having a number of viewpoints determined by the control unit 300 is used.
  • the camera calibration unit 410 calculates the camera parameters of the cameras existing at each viewpoint in which the image was taken, using the multi-view frame set corresponding to the number of viewpoints M and the viewpoint position determined by the control unit 300 (step). S4100).
  • the control unit 300 may calculate the camera parameters only for at least N cameras. Note that step S4100 is an example of a camera calibration step.
  • the three-dimensional processing unit 420 uses the multi-viewpoint frame set corresponding to the number of viewpoints N and the viewpoint position determined by the control unit 300 and the camera parameters calculated by the camera calibration unit 410 to three-dimensionalize the subject 1001. Generate a model or measure the three-dimensional displacement of the subject 1001 (step S4200).
  • the three-dimensional processing unit 420 is a specific example of the displacement measurement unit.
  • step S4200 is an example of a displacement measurement step.
  • the number of viewpoints used as a reference is the number of viewpoints N used for three-dimensional modeling or three-dimensional displacement measurement.
  • the number of viewpoints N is equal to the number of cameras constantly installed in the shooting space 1000.
  • step S4100 a viewpoint number M larger than the viewpoint number N in step S4200 is used.
  • the number of cameras does not necessarily have to be M.
  • Camera calibration may be performed by shooting from a plurality of viewpoints with a moving camera of less than M units and using a stationary region in the shot image.
  • the stationary area is an area in the image in which a stationary object is reflected, excluding a moving area in which an animal body is reflected.
  • the three-dimensional displacement measuring device 2000 may perform three-dimensional modeling in addition to the three-dimensional displacement measurement in the three-dimensional image processing unit 240, and may further generate a free-viewpoint image. Good.
  • the three-dimensional displacement measuring device 2000 can reconstruct a time-series three-dimensional model in which the coordinate axes match between times. Specifically, first, the three-dimensional displacement measuring device 2000 acquires a three-dimensional model at each time by independently performing three-dimensional reconstruction for each time. Next, the three-dimensional displacement measuring device 2000 detects a stationary camera and a stationary object (stationary three-dimensional point), and uses the detected stationary camera and the stationary object to adjust the coordinates of the three-dimensional model between times, and coordinates the axes. Generate a matching time-series 3D model of.
  • FIG. 14 is a flowchart showing an operation when the three-dimensional displacement measuring device 2000 according to the present embodiment generates a free viewpoint image.
  • the receiving unit 210 receives the multi-viewpoint video captured by the plurality of cameras 100-1 to 100-n (S101).
  • the storage unit 220 stores the received multi-viewpoint video (S102).
  • the acquisition unit 230 selects a frame from the multi-viewpoint video and outputs it as a multi-viewpoint frame set to the three-dimensional image processing unit 240 (S103).
  • the multi-view frame set selected here may have the same configuration as the multi-view frame set selected in step S2500 of FIG.
  • the three-dimensional image processing unit 240 generates a free-viewpoint image by executing camera calibration processing, three-dimensional modeling processing, and free-viewpoint image composition processing using the multi-viewpoint frameset (S104).
  • steps S103 and S104 are repeated for each multi-view frame set.
  • the transmission unit 250 transmits at least one of the camera parameters, the three-dimensional model of the subject, and the free viewpoint image to the external device (S105).
  • the control unit 300 determines the optimum number of viewpoints for each process in the camera calibration unit 410 and the three-dimensional processing unit 420, as described with reference to FIGS. 7 to 10.
  • the number of viewpoints determined here indicates the number of viewpoints different from each other.
  • the control unit 300 sets the number of viewpoints M, which is larger than the number of viewpoints N in the three-dimensional modeling process, so as to improve the accuracy of the camera parameters so as not to reduce the accuracy in the three-dimensional modeling process and the free viewpoint image composition process. It is determined as the number of viewpoints of the multi-view frame set used in the camera calibration process. That is, the control unit 300 has n frames imaged by n cameras 100-1 to 100-n and k (k is an integer greater than or equal to a) frames imaged by a non-fixed camera.
  • the camera calibration unit 410 is made to execute the camera calibration process using the M frames to which the above is added.
  • the number of non-fixed cameras a does not necessarily have to be k, and the k frames (images) obtained as a result of imaging from the k viewpoint by moving the non-fixed cameras a. There may be.
  • the control unit 300 determines the number of viewpoints l, which is smaller than the number of viewpoints N in the three-dimensional modeling process, as the number of viewpoints of the multi-view frame set used in the free-viewpoint video composition process.
  • FIG. 15 is a flowchart showing the operation of the three-dimensional image processing unit 240 when generating a free-viewpoint image.
  • a plurality of camera calibration units 410 are used, using M images for calibration taken from different M viewpoints by n cameras 100-1 to 100-n arranged at different positions.
  • the camera parameters of the cameras 100-1 to 100-n of the above are calculated (S310).
  • the M viewpoint here is based on the number of viewpoints determined by the control unit 300.
  • the camera calibration unit 410 performs a camera calibration process using M calibration images obtained by n cameras 100-1 to 100-n including a fixed camera and a non-fixed camera.
  • the camera calibration process the larger the number of cameras, the closer the distance between the cameras, and the closer the fields of view of multiple cameras with short distances, so it is easy to associate multiple images obtained from multiple cameras with short distances. Become. Therefore, when performing camera calibration, the camera calibration unit 410 increases the number of viewpoints by using a non-fixed camera in addition to the fixed camera that is always installed in the shooting space 1000.
  • the non-fixed camera may be at least one mobile camera, and when the mobile camera is used as the non-fixed camera, images taken at different timings are included. That is, the M calibration images used in the camera calibration process include images captured at different timings.
  • the M viewpoint multi-view frame set composed of M calibration images includes the frames obtained by asynchronous shooting. Therefore, the camera calibration unit 410 performs the camera calibration process by utilizing the corresponding points between the images of the feature points obtained from the stationary region, which is the region in which the stationary object is reflected in the M calibration images. .. Therefore, the camera calibration unit 410 calculates the camera parameters corresponding to the stationary region.
  • the stationary region is an region of the M calibration images excluding the moving region in which the animal body is reflected.
  • the moving area reflected in the frame is detected, for example, by calculating the difference from the past frame, calculating the difference from the background image, or automatically detecting the area of the animal body by machine learning.
  • the camera calibration unit 410 does not have to always perform the camera calibration process in step S310 in the free viewpoint image generation process in the three-dimensional image processing unit 240, and may perform it once every predetermined number of times.
  • the three-dimensional processing unit 420 uses N images for three-dimensional processing captured by each of the n cameras 100-1 to 100-n, and camera parameters obtained in the camera calibration process. To reconstruct the three-dimensional model (S320). That is, the three-dimensional processing unit 420 reconstructs the three-dimensional model using N images for three-dimensional processing captured at the N viewpoints based on the number of viewpoints N determined by the control unit 300. As a result, the three-dimensional processing unit 420 reconstructs the subject in the N three-dimensional processing images as three-dimensional points.
  • the N images for 3D processing used in the 3D modeling process are images captured by each of n cameras 100-1 to 100-n at arbitrary timings.
  • the N viewpoint multi-view frame set composed of N three-dimensional processing images is a multi-view frame set obtained by synchronous shooting. Therefore, the three-dimensional processing unit 420 performs the three-dimensional modeling process using the region (that is, all the regions) including the stationary object and the animal body among the N images for three-dimensional processing.
  • the three-dimensional processing unit 420 may use the measurement result of the position of the subject in the three-dimensional space by using the laser scan, or may use the corresponding points of a plurality of stereo images as in the multi-view stereo method. The position of the subject in the three-dimensional space may be calculated.
  • the three-dimensional processing unit 420 includes l third images captured by each of the l cameras out of the n cameras 100-1 to 100-n, and the camera calculated in the camera calibration process.
  • a free-viewpoint image is synthesized using the parameters and the 3D model reconstructed in the 3D modeling process (S330). That is, the three-dimensional processing unit 420 synthesizes a free viewpoint image using one third image captured at one viewpoint based on the number of viewpoints l determined by the control unit 300.
  • the three-dimensional processing unit 420 uses the texture information of the real camera to obtain the image of the virtual viewpoint based on the corresponding position between the image of the real camera and the image of the virtual viewpoint obtained by the camera parameters and the three-dimensional model. By calculating the texture information, the free-viewpoint image is synthesized.
  • the number of viewpoints M which is larger than the number of viewpoints N in the three-dimensional displacement measurement process, is used for the camera calibration process so as to improve the accuracy of the camera parameters.
  • the accuracy in the original displacement measurement process can be improved.
  • the image matching method and the principle of triangulation are used based on the highly accurate camera parameters, the first corresponding point and the second corresponding point, and are tertiary. Find the displacement of the original point over time. Therefore, the displacement of the subject in the three-dimensional space can be effectively obtained.
  • the number of viewpoints N of the camera can be determined so as to satisfy the accuracy of the measurement performed in the displacement measurement step. Therefore, in the three-dimensional displacement measuring method, the displacement of the subject in the three-dimensional space can be obtained with high accuracy.
  • the three-dimensional displacement measuring device 2000 considering that the accuracy of the camera parameters calculated in the camera calibration process has a great influence on the accuracy in the three-dimensional modeling process and the free-viewpoint image composition process.
  • the number of viewpoints M which is larger than the number of viewpoints N in the three-dimensional modeling process, is determined as the number of viewpoints of the multi-view frame set used in the camera calibration process. Therefore, the accuracy in the three-dimensional modeling process and the free-viewpoint video composition process can be improved.
  • the number of viewpoints l which is smaller than the number of viewpoints N in the three-dimensional modeling process, is determined as the number of viewpoints of the multi-view frame set used in the free viewpoint video composition processing. By doing so, the processing load required to generate the free-viewpoint video can be reduced.
  • FIG. 16 is a block diagram of a modified example of the three-dimensional image processing unit.
  • the three-dimensional image processing unit 240A is composed of a camera calibration unit 510 and a three-dimensional processing unit 520.
  • the camera calibration unit 510 has the same function, configuration, etc. as the camera calibration unit 410, but differs in the following points.
  • the camera calibration unit 510 receives the output of the camera calibration unit 510 as a feedback input, and performs processing again based on the input.
  • the three-dimensional processing unit 520 has the same function and configuration as the three-dimensional processing unit 420.
  • the three-dimensional processing unit 520 performs processing based on the data finally output from the camera calibration unit 510.
  • the final output from the camera calibration unit is, for example, an output excluding the first output from the camera calibration unit.
  • FIG. 17 is a flowchart showing the operation of a modified example of the three-dimensional image processing unit.
  • the camera calibration unit 510 calculates the camera parameters of the cameras existing at each viewpoint where shooting was performed by using the multi-view frame set corresponding to the number of viewpoints and the viewpoint position determined by the control unit 300 (step S5100). ).
  • the camera calibration unit 510 uses the value of the result of the camera calibration performed in step S5100 as an initial value, and uses a multi-viewpoint frame set corresponding to the number of viewpoints and the viewpoint position determined by the control unit 300 to perform photographing.
  • the camera parameters of the cameras existing at each viewpoint are calculated (step S5200).
  • the three-dimensional processing unit 520 uses the multi-viewpoint frame set corresponding to the number of viewpoints and the viewpoint position determined by the control unit 300 and the camera parameters calculated by the camera calibration unit 510 to perform three-dimensional modeling of the subject 1001. Alternatively, three-dimensional displacement measurement of the subject is performed (step S5300).
  • the greater the number of cameras used for camera calibration the closer the distance between the cameras. Since the areas of the shooting space 1000 within the field of view of the cameras are close to each other between cameras that are close to each other, it is easy to associate the images shot by each camera with each other. Therefore, when performing camera calibration, the number of viewpoints used for camera calibration is calculated from the number of viewpoints realized by the fixed camera by using a non-fixed camera in addition to the fixed camera that is always installed in the shooting space 1000. Also increase.
  • the non-fixed camera may be at least one mobile camera. However, when a mobile camera is used as the non-fixed camera, the shooting times of the images taken from different viewpoints are different, so that the camera parameters corresponding only to the still area of the shot images are calculated.
  • the calculated camera parameters do not always correspond to the moving area in the image taken by the fixed camera.
  • the moving region is an region in the image in which the animal body is reflected.
  • overall optimization of camera parameters is performed. Therefore, when this method is used, the camera parameters are not always optimized when focusing only on the fixed camera.
  • the camera calibration unit 510 uses a multi-viewpoint in which B viewpoints are used, including images taken by at least one camera other than the A camera always installed in the shooting space 1000. Use the frameset to perform a rough camera calibration.
  • the camera calibration unit 510 uses only the images taken by the cameras A that are always installed in the shooting space 1000, and sets the camera parameters calculated by the camera calibration unit 510 to the cameras of the A units.
  • Optimize for the environment means that the three-dimensional points obtained secondarily during the calculation of the camera parameters are reprojected on the captured image, and the reprojected points and the feature points detected on the image are used. It may point to perform the process of minimizing the evaluation value by using the error (reprojection error) of the above as the evaluation value.
  • the number of viewpoints used in the second stage camera calibration may be larger than the number of cameras A always installed in the shooting space 1000. Further, the number of viewpoints used in the second stage camera calibration may be larger than the number of viewpoints used in three-dimensional image processing (for example, three-dimensional displacement measurement). However, the number of viewpoints used in the second stage camera calibration may be equal to or smaller than the number of viewpoints used in three-dimensional image processing (for example, three-dimensional displacement measurement).
  • the n cameras 100-1 to 100-n may include a stereo camera having two cameras.
  • the stereo camera may have two cameras that image in substantially the same direction as each other, that is, a first camera and a second camera, and the distance between the two cameras may be a predetermined distance or less.
  • n cameras 100-1 to 100-n are configured by stereo cameras, they are composed of n / 2 first cameras and n / 2 second cameras.
  • the two cameras included in the stereo camera may be integrated or may be separate.
  • first camera and the second camera constituting the stereo camera may image with different sensitivities.
  • the first camera is a camera that captures images with the first sensitivity.
  • the second camera is a camera that takes an image with a second sensitivity different from the first sensitivity.
  • the first camera and the second camera are cameras having different color sensitivities.
  • the three-dimensional processing unit reconstructs a three-dimensional model using n images for three-dimensional processing obtained by being imaged by all of n cameras 100-1 to 100-n. Since the 3D processing unit uses the luminance information in the 3D modeling process, it is possible to calculate the 3D model with high accuracy by using all n cameras regardless of the difference in color sensitivity.
  • the three-dimensional processing unit is composed of n / 2 third images, which are a plurality of images obtained by being imaged by n / 2 first cameras or n / 2 second cameras, and a camera calibration unit.
  • a free-viewpoint image is synthesized using the calculated camera parameters and the 3D model reconstructed by the 3D processing unit.
  • the three-dimensional processing unit can accurately use n / 2 images from either the n / 2 first camera or the n / 2 second camera. The effect is small. Therefore, the three-dimensional processing unit performs free viewpoint composition using n / 2 images captured by one of the first camera and the second camera according to the situation of the photographing space 1000.
  • the n / 2 first camera is a camera having a high red color sensitivity and the n / 2 second camera is a camera having a high blue color sensitivity.
  • the three-dimensional processing unit uses the image captured by the first camera having a high red color sensitivity if the subject has a red color, and the blue color sensitivity if the subject has a blue color.
  • the image to be used is switched so that the free-viewpoint image composition process is executed using the image captured by the second camera having a high color.
  • free viewpoint image composition is performed using one of two types of images obtained from two types of cameras having different sensitivities depending on the situation of the shooting space. Therefore, it is possible to generate a free viewpoint image with high accuracy.
  • the first camera and the second camera are not limited to having different color sensitivities, and may be cameras having different luminance sensitivities.
  • the three-dimensional processing unit can switch the camera according to the conditions such as daytime and nighttime, fine weather and cloudy weather.
  • the n cameras are not limited to the n / 2 first cameras and the n / 2 second cameras, but the i first cameras and the j first cameras. It may be composed of two cameras.
  • the plurality of cameras 100-1 to 100-n are assumed to be composed of a fixed camera and a non-fixed camera, but the present invention is not limited to this, and all the plurality of cameras may be composed of fixed cameras. Further, although the n images used in the three-dimensional modeling are assumed to be images captured by a fixed camera, images captured by a non-fixed camera may be included.
  • FIG. 18 is a diagram showing an outline of three-dimensional space recognition.
  • the space to be photographed can be three-dimensionally reconstructed by photographing the same space from multiple viewpoints using a calibrated camera (for example, a fixed camera) (three-dimensional space reconstruction). Using this three-dimensionally reconstructed data, three-dimensional displacement measurement and the like can be performed. This makes it possible to realize a next-generation wide area monitoring system and a three-dimensional displacement measurement system. Specifically, at an intersection or the like on a roadway, a passing vehicle 10 or the like can be photographed by using a fixed camera 100-a or a non-fixed camera 100-b to measure a three-dimensional displacement of the vehicle 10 or the like.
  • the three-dimensional displacement of the subject may be measured, and then the three-dimensional modeling of the subject may be performed.
  • the present invention is not limited to this, and the three-dimensional displacement of the subject is measured. You may perform three-dimensional modeling of the subject without doing so.
  • the present disclosure can be applied to a three-dimensional displacement measuring method and a three-dimensional displacement measuring device, and can be applied to, for example, an infrastructure inspection system, an industrial machine monitoring system, a three-dimensional space recognition system, a next-generation monitoring system, and the like.
  • 3 3D reconstruction system 10 Vehicle 100-1 to 100-n Camera 100-a Fixed camera 100-b Non-fixed camera 200-1 to 200-Q Multi-view frame set 210 Receiver 220 Storage 230 Acquisition 240, 240A 3D image processing unit 250 Transmission unit 300 Control unit 310 Measurement determination unit 320 Camera calibration determination unit 410, 510 Camera calibration unit 420, 520 3D processing unit 1000 Shooting space 1001 Subject 2000 3D displacement measuring device

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Image Analysis (AREA)
PCT/JP2020/026801 2019-07-30 2020-07-09 三次元変位計測方法及び三次元変位計測装置 Ceased WO2021020062A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
JP2021536879A JP7442072B2 (ja) 2019-07-30 2020-07-09 三次元変位計測方法及び三次元変位計測装置
US17/577,733 US20220138971A1 (en) 2019-07-30 2022-01-18 Three-dimensional displacement measuring method and three-dimensional displacement measuring device
JP2024016929A JP7607229B2 (ja) 2019-07-30 2024-02-07 三次元変位計測方法及び三次元変位計測装置

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2019-139581 2019-07-30
JP2019139581 2019-07-30

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/577,733 Continuation US20220138971A1 (en) 2019-07-30 2022-01-18 Three-dimensional displacement measuring method and three-dimensional displacement measuring device

Publications (1)

Publication Number Publication Date
WO2021020062A1 true WO2021020062A1 (ja) 2021-02-04

Family

ID=74230583

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/026801 Ceased WO2021020062A1 (ja) 2019-07-30 2020-07-09 三次元変位計測方法及び三次元変位計測装置

Country Status (3)

Country Link
US (1) US20220138971A1 (https=)
JP (2) JP7442072B2 (https=)
WO (1) WO2021020062A1 (https=)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11830223B2 (en) * 2019-04-08 2023-11-28 Nec Corporation Camera calibration apparatus, camera calibration method, and nontransitory computer readable medium storing program
US11941794B2 (en) 2019-08-19 2024-03-26 Current Lighting Solutions, Llc Commissioning of lighting system aided by augmented reality
JP2022126206A (ja) * 2021-02-18 2022-08-30 キヤノン株式会社 画像処理装置、画像処理方法及びプログラム

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011030771A1 (ja) * 2009-09-08 2011-03-17 株式会社パスコ 変位計測方法、変位計測装置及び変位計測プログラム
CN102778207A (zh) * 2012-07-10 2012-11-14 中联重科股份有限公司 一种结构件应力应变的测量方法、装置及系统
KR20160082659A (ko) * 2014-12-30 2016-07-08 순천대학교 산학협력단 다채널 연속 촬영 영상을 이용한 구조물의 입체적인 진동 자동 측정 방법
WO2018135510A1 (ja) * 2017-01-19 2018-07-26 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ 三次元再構成方法及び三次元再構成装置
JP2019074991A (ja) * 2017-10-18 2019-05-16 株式会社富士通アドバンストエンジニアリング 画像処理プログラム、画像処理装置および画像処理方法

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9262840B2 (en) * 2012-06-08 2016-02-16 Correlated Solutions, Inc. Optical non-contacting apparatus for shape and deformation measurement of vibrating objects using image analysis methodology

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011030771A1 (ja) * 2009-09-08 2011-03-17 株式会社パスコ 変位計測方法、変位計測装置及び変位計測プログラム
CN102778207A (zh) * 2012-07-10 2012-11-14 中联重科股份有限公司 一种结构件应力应变的测量方法、装置及系统
KR20160082659A (ko) * 2014-12-30 2016-07-08 순천대학교 산학협력단 다채널 연속 촬영 영상을 이용한 구조물의 입체적인 진동 자동 측정 방법
WO2018135510A1 (ja) * 2017-01-19 2018-07-26 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ 三次元再構成方法及び三次元再構成装置
JP2019074991A (ja) * 2017-10-18 2019-05-16 株式会社富士通アドバンストエンジニアリング 画像処理プログラム、画像処理装置および画像処理方法

Also Published As

Publication number Publication date
JP7607229B2 (ja) 2024-12-27
US20220138971A1 (en) 2022-05-05
JPWO2021020062A1 (https=) 2021-02-04
JP2024052755A (ja) 2024-04-12
JP7442072B2 (ja) 2024-03-04

Similar Documents

Publication Publication Date Title
JP7607229B2 (ja) 三次元変位計測方法及び三次元変位計測装置
CN108765498B (zh) 单目视觉跟踪方法、装置及存储介质
JP6754992B2 (ja) 三次元再構成方法
JP7227969B2 (ja) 三次元再構成方法および三次元再構成装置
CN106289106B (zh) 一种线阵相机和面阵相机相结合的立体视觉传感器及标定方法
JP7143225B2 (ja) 三次元再構成方法及び三次元再構成装置
WO2021140886A1 (ja) 三次元モデル生成方法、情報処理装置およびプログラム
CN106127745B (zh) 结构光3d视觉系统与线阵相机的联合标定方法及装置
JP5725953B2 (ja) 撮像装置及びその制御方法、並びに情報処理装置
JP6304244B2 (ja) ３次元形状計測装置、３次元形状計測方法及び３次元形状計測プログラム
US11803982B2 (en) Image processing device and three-dimensional measuring system
Dias et al. Registration and fusion of intensity and range data for 3D modelling of real world scenes
JPWO2019211970A1 (ja) 三次元再構成方法及び三次元再構成装置
JP7170224B2 (ja) 三次元生成方法および三次元生成装置
JP2013254097A (ja) 画像処理装置及びその制御方法、並びにプログラム
CN106225676A (zh) 三维测量方法、装置及系统
CN107038753A (zh) 立体视觉三维重建系统及方法
CN105430298A (zh) 通过立体摄像系统同时曝光合成hdr图像的方法
JP7692175B2 (ja) 三次元モデル生成方法及び三次元モデル生成装置
CN110068308A (zh) 一种基于多目相机的测距方法及测距系统
JP5925109B2 (ja) 画像処理装置、その制御方法、および制御プログラム
KR101857977B1 (ko) 플래놉틱 카메라와 깊이 카메라를 결합한 영상 장치 및 영상 처리 방법
JPH1153549A (ja) 画像処理装置、画像処理方法、および、伝送媒体
KR101846390B1 (ko) 소형 보정 패턴을 이용한 카메라 네트웍 보정 방법 및 장치
Wierzbicki et al. Application of stereo cameras with wide-angle lenses for the indoor mapping

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20845914

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021536879

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20845914

Country of ref document: EP

Kind code of ref document: A1