WO2020105697A1 - Système de caméra à capture de mouvement et procédé d'étalonnage - Google Patents

Système de caméra à capture de mouvement et procédé d'étalonnage

Info

Publication number
WO2020105697A1
WO2020105697A1 PCT/JP2019/045552 JP2019045552W WO2020105697A1 WO 2020105697 A1 WO2020105697 A1 WO 2020105697A1 JP 2019045552 W JP2019045552 W JP 2019045552W WO 2020105697 A1 WO2020105697 A1 WO 2020105697A1
Authority
WO
WIPO (PCT)
Prior art keywords
camera
coordinate system
mounter
motion capture
global coordinate
Prior art date
Application number
PCT/JP2019/045552
Other languages
English (en)
Japanese (ja)
Inventor
中村　仁彦
洋介 池上
文香 山田
大輝 小原
智行 堀川
亮矢 鈴木
Original Assignee
国立大学法人東京大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 国立大学法人東京大学 filed Critical 国立大学法人東京大学
Priority to JP2020557612A priority Critical patent/JP7395189B2/ja
Publication of WO2020105697A1 publication Critical patent/WO2020105697A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules

Definitions

  • the present invention relates to a camera system for motion capture.
  • Motion capture technology is widely used for various motion measurements. However, in team sports, it is not performed to capture the motion of each player during the game and analyze the motion. For example, moving images of sports such as soccer, rugby, baseball, volleyball, and handball are not acquired, and motion analysis of each player is not performed based on the acquired moving images.
  • motion capture does not reduce the performance of the player
  • motion capture is a supporting operation (TV broadcast, team Staff, spectators, etc.)
  • large storage capacity TV broadcast, team Staff, spectators, etc.
  • synchronization between high-speed cameras over a long distance synchronization between high-speed cameras over a long distance
  • e ability to receive large amounts of data in real time
  • f long time It is necessary to satisfy the conditions such as stable system with operable performance, and (g) camera calibration for accurate 3D reconstruction.
  • Optical motion capture and motion capture using an inertial sensor are known as conventional typical motion capture techniques. These motion captures have the drawback that it takes time to prepare a measurement for attaching a reflective marker or an inertial sensor to a target body, and further, the marker or the sensor may hinder the smooth movement of the target. Cannot be used to acquire the action of each player during a match. Moreover, these motion captures have a limited measurement space. In optical motion capture using an infrared light source, it is necessary to perform measurement in a space surrounded by a camera that is not affected by external light, and when receiving sensor data wirelessly, wireless signals are limited. It was necessary to measure in space.
  • Non-Patent Document 1 performs motion capture from the images of a plurality of RGB cameras in a completely unconstrained manner. From the indoor living space to the wide outdoor sports field principle Specifically, this is a technology that enables motion measurement if images can be acquired.
  • the camera parameters include optical parameters such as lens distortion, internal parameters such as focal length and optical center, and external parameters representing the position / orientation of the camera in the space where the camera is installed.
  • Conventional camera calibration is performed by calibrating all of these parameters by photographing a calibration instrument (checker board, calibration wand, etc.) of known shape and dimensions with one or more cameras. ..
  • the measurement of a person or robot in a sports field or outdoors has a feature that the measurement area becomes a large space. It is possible in principle to calibrate using a calibration device (checker board, calibration wand, etc.) as in the past, but since the accuracy of calibration depends on the number of pixels of the image sensor image, If the distance to the object to be photographed is several tens of meters, the calibration accuracy will decrease and it is not suitable for three-dimensional reconstruction in motion capture.
  • the present invention has an object to easily and accurately calibrate a camera even if the imaging space for motion capture is a large space.
  • the motion capture camera system is A plurality of camera units consisting of one or a plurality of cameras, Multiple camera mounters equipped with each camera unit, Calibration means for acquiring camera parameters for three-dimensionally reconstructing each camera image;
  • One or more surveying instruments with angle measuring function, Equipped with The camera parameters include external parameters that represent the position and orientation of each camera in the global coordinate system,
  • the calibration means obtains the external parameter using the position and orientation of each camera mounter in the global coordinate system, and the position and orientation of the camera mounter in the global coordinate system is the one or more surveying points. It is acquired using the machine.
  • one or more surveying instruments of the one or more surveying instruments have a distance measuring function. Examples of the surveying instrument having the angle measuring function include a theodolite and a total station. A total station can be exemplified as the surveying instrument having the angle measuring function and the distance measuring function.
  • a part or all of the plurality of camera mounters is the surveying instrument.
  • one camera mounter is composed of total stations and the remaining camera mounters are composed of theodolites. The surveying instruments that make up the camera mounter are leveled.
  • a part or all of the plurality of camera mounters includes a plurality of markers (characteristic points) that can be measured by the surveying instrument.
  • all camera mounters are camera mounters with markers.
  • the camera mounter has a plane with three or more markers (eg, four survey markers).
  • the calibration means includes position information (three-dimensional position information) in a global coordinate system of a plurality of feature points located in the imaging space of each camera, and position information of each feature point in each camera image. (Two-dimensional position information) and Position information of the plurality of feature points in the global coordinate system is acquired using the surveying instrument.
  • the feature point is a fixed point located in the imaging space of each camera, and includes, for example, an intersection (including a corner) of a line on the court, an intersection, a point on the goal post (a post and a post). The intersection of the crossbar or the bottom of the post) can be used as a feature point.
  • the three-dimensional position information and the two-dimensional position information of the feature points are used for optimization calculation when acquiring an external parameter.
  • the calibration means uses the position / orientation of each camera mounted on the camera mounter in each camera mounter coordinate system (for example, when the position / orientation of the camera with respect to the camera mount is known).
  • the calibration means includes means for acquiring the position / orientation of each camera mounted on the camera mounter in each camera mounter coordinate system.
  • the camera mounter comprises the surveying instrument, The surveying instrument is used in acquiring the position and orientation of the camera in the camera mounter coordinate system.
  • the position / orientation of each camera mounted on the camera mounter in each camera mounter coordinate system is known or previously acquired.
  • the calibration means acquires the extrinsic parameter by an optimization calculation.
  • the position and orientation of the camera mounter in the global coordinate system, and the three-dimensional position information and the two-dimensional position information of the feature points are used in the optimization calculation.
  • position information of each camera image corresponding to one or more feature points whose position information in the global coordinate system is unknown is used.
  • the camera parameter includes an optical parameter and an internal parameter
  • the calibration unit acquires a part or all of the external parameter, the optical parameter, and the internal parameter by an optimization calculation.
  • the optical parameters and internal parameters may be obtained by a conventional method (using a calibration instrument). For example, when the camera mounter is composed of the surveying instrument, the surveying instrument may be used in acquiring the optical parameter and the internal parameter. In the acquisition of external parameters, some or all of the acquired optical parameters and internal parameters may be used.
  • the present invention includes a plurality of camera units including one or a plurality of cameras, Multiple camera mounters equipped with each camera unit,
  • a motion-capture camera system comprising: a calibration method for acquiring camera parameters for three-dimensionally reconstructing the plurality of camera images,
  • the camera parameters include external parameters that represent the position and orientation of each camera in the global coordinate system, Obtaining the position and orientation of each camera mounter in the global coordinate system using one or more surveying instruments equipped with an angle measuring function, Obtaining the extrinsic parameters using the position and orientation of each camera mounter in the global coordinate system, including.
  • a part or all of the plurality of camera mounters is the surveying instrument, and the position and orientation of each camera mounter in the global coordinate system is acquired using surveying data of a plurality of measurement points (feature points).
  • the plurality of measurement points are two characteristic points A and B that can be surveyed from each surveying instrument (the intervals are known, for example, points A and B on a vertical surveying pole) and each camera. And a feature point on the mounter (for example, the lower end of the pendulum weight).
  • the horizontal axis having the same horizontal angle as that of the vector is the z axis and the vertical axis is the x axis.
  • the angle between the mounter coordinate systems can be acquired.
  • each camera is trigonometrically Obtain the position and orientation of the mounter in the global coordinate system.
  • a part or all of the plurality of camera mounters includes a plurality of markers (feature points) that can be surveyed by the surveying instrument, and camera in a global coordinate system is used by using surveying data of the plurality of markers. Get the position and orientation of the mounter.
  • the camera mounter has a plane with three or more markers (eg, four survey markers).
  • the distances between the three or more markers are known, and these markers are provided on a vertical plane, and are measured by a surveying instrument (for example, having an angle measuring function and a distance measuring function).
  • position information in a global coordinate system of a plurality of feature points located in the imaging space of each camera and position information of each feature point in each camera image are used in the acquisition of the external parameter.
  • Position information of the plurality of feature points in the global coordinate system is acquired using the surveying instrument.
  • the feature point is a fixed point located in the imaging space of each camera, and includes, for example, an intersection (including a corner) of a line on the court, an intersection, a point on the goal post (a post and a post). The intersection of the crossbar or the bottom of the post) can be used as a feature point.
  • the three-dimensional position information and the two-dimensional position information of the feature points are used for optimization calculation.
  • the position / orientation of each camera mounted on the camera mounter in each camera mounter coordinate system is used in the acquisition of the external parameter.
  • the calibration means includes means for acquiring the position / orientation of each camera mounted on the camera mounter in each camera mounter coordinate system.
  • the camera mounter comprises the surveying instrument, The position and orientation of the camera in the camera mounter coordinate system are acquired using the surveying instrument.
  • a plurality of feature points whose positions in the camera mounter coordinate system are known are set using the surveying instrument (see FIG.
  • the position and orientation of the camera in the camera mounter coordinate system is acquired using the position information of the feature point in the camera mounter coordinate system and the position information in the camera image.
  • the position / orientation of each camera mounted on the camera mounter in each camera mounter coordinate system is known or previously acquired.
  • the extrinsic parameters are obtained by an optimization calculation.
  • the position and orientation of the camera mounter in the global coordinate system, and the three-dimensional position information and two-dimensional position information of the feature points are used in the optimization calculation.
  • position information of each camera image corresponding to one or more feature points whose position information in the global coordinate system is unknown is used.
  • the cameras are synchronized, each camera captures images of calibration instruments in different positions and orientations, and the position information of each of the plurality of feature points on the calibration instrument in each camera image is used.
  • the camera parameter includes an optical parameter and an internal parameter
  • the calibration unit acquires a part or all of the external parameter, the optical parameter, and the internal parameter by an optimization calculation.
  • the optical parameters and internal parameters may be obtained by a conventional method (using a calibration instrument). For example, when the camera mounter is composed of the surveying instrument, the surveying instrument may be used in acquiring the optical parameter and the internal parameter. In the acquisition of external parameters, some or all of the acquired optical parameters and internal parameters may be used.
  • the present invention can easily and highly accurately calibrate a camera by using highly accurate survey data measured by a surveying instrument in calibrating the camera even if the imaging space for motion capture is a large space. it can. Therefore, by shooting sports games that are played in indoor / outdoor sports fields, indoor / outdoor courts, arenas, domes, gymnasiums, etc., the motion of each player during the game is reconstructed in three dimensions to perform motion capture, and motion analysis is performed. It can be performed.
  • Motion capture / camera system [A-1] Overall configuration
  • the motion capture / camera system captures a team sport performed in a large space such as a sports field or an arena with a plurality of cameras.
  • the motion capture of each player is performed by using the captured image acquired by each camera.
  • the motion capture used in this embodiment is a video motion capture technique (Non-Patent Document 1), and its specific content will be described later.
  • the motion capture camera system includes a plurality of camera units including one or a plurality of cameras, and a plurality of camera mounters in which the camera units are mounted.
  • FIG. 1 is a conceptual diagram of a motion capture camera system that captures a soccer match, in which four camera units including four cameras are arranged so as to surround a soccer court. The distance from each camera to the target athlete is several tens of meters.
  • the motion capture camera system shown in FIG. 1 includes four camera units CU 1 , CU 2 , CU 3 , and CU 4 , and the camera unit CU 1 has four cameras C 11 , C 12 , C 13 , C 14 , and cameras.
  • the unit CU 2 has four cameras C 21 , C 22 , C 23 , C 24
  • the camera unit CU 3 has four cameras C 31 , C 32 , C 33 , C 34
  • the camera unit CU 4 has four cameras C 41 , It is equipped with C 42 , C 43 , and C 44 .
  • the camera unit CU 1 is mounted on the camera mounter CM 1
  • the camera unit CU 2 is mounted on the camera mounter CM 2
  • the camera unit CU 3 is mounted on the camera mounter CM 3
  • the camera unit CU 4 is mounted on the camera unit CU 4. It is mounted on the camera mounter CM 4 .
  • four camera units are shown for convenience, but in reality, more camera units could be arranged. Note that the number of cameras installed in one camera unit is not limited.
  • FIG. 2 is a conceptual diagram of a motion capture camera system for shooting a volleyball game, in which four camera units having one camera are arranged so as to surround a volleyball court.
  • the motion capture camera system includes four camera units CU 1 , CU 2 , CU 3 , and CU 4 , and the camera unit CU 1 is one camera C 11 and the camera unit CU 2 is one.
  • the camera C 21 and the camera unit CU 3 are equipped with one camera C 31
  • the camera unit CU 4 is equipped with one camera C 41 .
  • the camera unit CU 1 is mounted on the camera mounter CM 1
  • the camera unit CU 2 is mounted on the camera mounter CM 2
  • the camera unit CU 3 is mounted on the camera mounter CM 3
  • the camera unit CU 4 is mounted on the camera unit CU 4. It is mounted on the camera mounter CM 4 . Note that the number of cameras installed in one camera unit is not limited.
  • each including one camera are arranged so as to surround the volleyball court.
  • the volleyball court has a rectangular size of 18 m ⁇ 9 m, and a free zone having a width of 3 m is provided around the rectangle.
  • Each camera is arranged so that it can photograph a space having a height of 3 m in an area including at least the free zone.
  • Each camera is arranged by mounting it on a tripod leveled at a predetermined position or by fixing it to a handrail.
  • Each camera is calibrated in advance with the position and orientation fixed, and the position and orientation are maintained during shooting.
  • the camera network of the motion capture camera system shown in FIG. 7 is shown in FIG. It should be noted that although 12 cameras can be connected to the camera network, 10 cameras are used in the embodiment.
  • the camera network according to this embodiment includes a plurality of cameras, a transmission box, a camera hub, one or a plurality of computers, a power supply, and a waveform generator that generates a synchronization signal. Each camera is electrically connected to the transmission box, the transmission box and the camera hub are electrically connected by hybrid copper / fiber cable, and the camera hub and the computer are electrically connected. The cameras are synchronized.
  • Each camera acquires an image with a resolution of 1920x1200 at 120Hz.
  • Each camera has a USB3.0 interface, and USB3 to Fiber Optic Extender is used for image signal transmission.
  • the transmitter box is equipped with a USB3toFiberOpticExtender Transmitter
  • the camera hub is equipped with a USB3toFiberOpticExtender Receiver
  • image signals are sent to the computer via the USB3toFiberOpticExtender. ..
  • the computer includes a data receiving unit, a processing unit, and a storage unit. The image data received by the data receiving unit is appropriately compressed by the processing unit and stored in the storage unit.
  • Each camera is mounted on a camera mounter, and the position and orientation of each camera is fixed with respect to the camera mounter.
  • the internal parameters, external parameters, and distortion coefficient of the camera are obtained by the calibration, and these camera parameters are stored in the storage unit of the computer.
  • the camera parameters are used for three-dimensional reconstruction of the motion of the object using each camera image.
  • the motion capture camera system further includes a calibration means and one or a plurality of surveying instruments having an angle measuring function.
  • the calibration means is composed of a computer (including an input unit, a calculation unit, a storage unit, an output unit, etc.).
  • the calibration means acquires camera parameters for three-dimensionally reconstructing each camera image.
  • the calibration according to the present embodiment is characterized in that the survey data of the survey instrument is used in the calibration calculation. More specifically, the calibration means acquires external parameters by using the position and orientation of each camera mounter in the global coordinate system, and the position and orientation of the camera mounter in the global coordinate system is one. Alternatively, it is acquired using a plurality of surveying instruments. In one aspect, the position / orientation of the camera is fixed and the surveying instrument is not used during the motion capture.
  • the calibration means acquires the camera parameters by optimization calculation, and the calibration means includes an optimization calculation unit that executes optimization calculation.
  • Theodolite and total station are known as high-precision surveying technology in a large outdoor space.
  • Theodolite is an angle measuring instrument with a typical angle measuring function. It has a movable part (with a telescope) that can rotate horizontally and vertically, and can measure the horizontal angle and altitude angle with high accuracy. ..
  • a total station is a device that has a theodolite equipped with an angle measuring function and a distance measuring function (laser measurement).
  • the total station used in the experiment described later has a pan angle (horizontal angle) and tilt angle (vertical angle) accuracy of 20 seconds, and a laser range finder accuracy of ⁇ 1 mm at 28.8 m.
  • the present invention relates to a motion capture camera system used for measurement in a large space, such as a surveying instrument, specifically, a theodolite or a total station, and, if necessary, a surveying instrument (reflection prism or reflection marker used for these measurements. It is possible to easily and accurately calibrate camera parameters even in a large space by configuring a camera system including
  • the camera mounter When the camera mounter is composed of a surveying instrument
  • the camera mounter of each camera unit consisting of one or a plurality of cameras is comprised of a theodolite or a total station surveying instrument.
  • the camera mounters CM 1 , CM 2 , CM 3 and CM 4 are composed of surveying instruments.
  • Each camera unit is mounted on the movable part of the surveying instrument, and can rotate in the horizontal and vertical directions integrally with the movable part when measuring the angle.
  • a camera stand equipped with one or a plurality of cameras is fixedly installed on a theodolite or a total station.
  • the theodolite and the total station are mounted, for example, on a leveled tripod, and the camera unit is mounted thereon (see FIG. 4).
  • the attitude / position of the camera mounter in the global coordinate system can be acquired by using the surveying technology.
  • Each camera unit, a camera stand, and a plurality of cameras mounted on the camera stand are mounted on the surveying instrument.
  • four cameras are radially arranged at equal angles so that the optical axes of the cameras are located in the same plane and the optical axes of all the cameras intersect at one point.
  • each camera is located on the side of a regular polygonal pyramid and the optical axes of all cameras intersect at one point on the axis of the regular polygonal pyramid.
  • the number of cameras in each camera unit and the arrangement mode are not limited.
  • a plurality of camera mounters are configured by surveying instruments
  • all the camera mounters are configured by surveying instruments.
  • a portion of the plurality of camera mounters comprises a surveying instrument.
  • at least one camera mounter may be a total station.
  • a theodolite equipped with a camera is disclosed in, for example, Patent Documents 1-3 and Non-Patent Document 2, and is also known as a video theodolite.
  • the camera in the conventional video theodolite is in a rotatable state during normal use, and the camera also rotates together with the rotation of the angle measuring unit, and uses the camera image information together with the angle measuring information.
  • the angle measuring function of the surveying instrument is used only in the calibration of the camera, and the position of the camera is fixed during use (during motion capture).
  • the theodolite angle measurement function is not used.
  • use of the angle measuring function of the surveying instrument at the time of motion capture is not excluded, and the angle measuring function may be used in the dynamic calibration described later.
  • the camera mounter is provided with a marker that can be measured by a surveying instrument
  • the camera mounter is provided with a plurality of markers (characteristic points), and the marker is used by the surveying instrument.
  • the camera mounter includes a plate (for example, a square of 30 cm ⁇ 30 cm to 40 cm ⁇ 40 cm) having a flat surface of a predetermined shape and size, and the flat surface portion of the plate is a reflection that serves as a target for a theodolite or a total station.
  • Three or more (for example, four) markers are provided.
  • the camera mounters CM 1 , CM 2 , CM 3 and CM 4 are composed of a plate having a flat surface having a plurality of reflection markers.
  • a camera whose relative position is fixed is mounted on the flat plate.
  • each camera unit includes one camera, and the camera is fixed at the center of the width direction in the lower portion of the square flat plate.
  • a flat plate that has three or more reflection markers and is fixed in its relative position to the camera is attached to the camera base (head) together with the camera, and the camera base is fixed to a tripod or the like.
  • a plurality of cameras may be mounted on the camera mounter including a plate having a plurality of markers.
  • the posture / position of the camera mounter (flat plate) in the global coordinate system can be acquired.
  • a surveying instrument typically a total station
  • a surveying instrument typically a total station having an angle measuring function and a distance measuring function for the measurement of the marker. ..
  • surveying instruments other than theodolite and total station
  • examples of surveying instruments other than theodolite and total station include a tripod, aluminum stuff, aluminum rod, pole, level, auto level, reflecting prism, and prism pole.
  • [B] Camera Calibration Camera calibration for three-dimensional reconstruction in large space motion capture will be described.
  • the three methods of the first method, the second method, and the third method will be described below.
  • the first method is a known method, but the third method is a combination of the first method and the second method. Therefore, the first method will be referred to together with the description of the camera parameters.
  • [B-1] General bundle adjustment using a feature point group whose position is unknown (first method) Calibration of each camera of the camera system used in this embodiment will be described.
  • the bundle adjustment means that the bundle of rays from the three-dimensional feature point to the camera is optimally adjusted for both the position and orientation of the three-dimensional feature point and the camera. ..
  • This optimization is generally solved as a non-linear optimization problem that minimizes reprojection error of feature points.
  • the perspective projection transformation matrix based on the pinhole camera model which is the simplest camera model, is used for reprojection of the feature points.
  • the matrices i A and i B that represent the internal and external parameters of the camera can be expressed as follows.
  • i f x and i f y are the focal lengths in pixel units of the x and y axes on the image plane
  • i c x and i c y are the principal points on the image plane (the intersection of the image plane and the optical axis).
  • position of) the matrix i R is the attitude of the global coordinate system in the camera coordinate system
  • i t represents the position of the origin of the global coordinate system in the camera coordinate system.
  • FIG. 11 shows a projection from a three-dimensional space onto an image plane along coordinate axes commonly used in computer vision.
  • the internal parameter matrix A of the camera is defined by the following formula.
  • the matrix i R can be treated as i R ( i ⁇ ⁇ , i ⁇ ⁇ , i ⁇ ⁇ ) as three variables when expressed in Euler angles.
  • the nonlinear least squares problem that minimizes the reprojection error of feature points in multiple cameras can be expressed as follows.
  • the focal lengths i f x , i f y which are optimization variables, and the camera position i t have a relationship such that if one is fixed, the other is not determined. Therefore, either one must be fixed, or the upper and lower limits must be set narrower for each.
  • the data obtained for multi-camera calibration is the pixel position in the image plane of the tie points (a group of feature points whose positions in the global coordinate system are unknown, such as the locus of the center position of the color ball) for multiple cameras.
  • the position X tie of the feature point group in the global coordinate system is determined depending on the positions and orientations of the plurality of cameras.
  • This bundle adjustment can be expressed as:
  • the above system of nonlinear equations can be solved using the Levenberg-Marquardt method.
  • This bundle adjustment can be expressed as:
  • the above nonlinear equation can be solved using the Levenberg-Marquardt method or the Trust-Region-Reflective method.
  • the Trust-Region-Reflective method for optimization variables, You can set a range constraint such as. Based on the obtained camera mounter position / orientation i B m , by setting the initial values of the optimization variables and narrow range constraints, the above-mentioned focal lengths i f x , i f y and camera position i t You can add surveyingly relevant constraints to the relationship.
  • the distortion coefficient of the camera is an internal parameter using the OpenCV function cv :: calibrateCamera by using multiple images taken at various distances and angles for a checkerboard with known distances between grid points. Ask with.
  • the internal parameters are also optimized as variables for bundle adjustment.
  • optimization solver optimization was performed using the function lsqnonlin that solves the nonlinear least squares problem in the Optimization Toolbox of MATLAB.
  • FIGS. 5 and 6 The characteristic points on the court and the position / orientation of the camera mounter were obtained using a surveying instrument, and the bundle position adjustment was performed for each camera to obtain the camera position / orientation.
  • the camera mounter with a survey marker and the survey instrument used in the experiment are shown in FIGS. 5 and 6, respectively.
  • the surveying instrument is a total station (angle surveyer / laser rangefinder), the camera mounter is a 40 cm ⁇ 40 cm flat plate, and surveying markers are provided at the four corners.
  • FIG. 9 shows 18 on-court feature points (white line intersections). The position information of these feature points in the global coordinate system is acquired using a surveying instrument.
  • the feature point number 0 on the court is the origin
  • the X axis is the direction from the origin to the feature point number 9 on the court is positive
  • the Z axis is the vertical upward direction determined by the leveling of the total station. I am trying.
  • the axis of the coordinate system obtained as a result of optimization is rotated, and the three-dimensional position obtained by surveying is also translated / rotated.
  • the on-court feature point whose position in the global coordinate system is known is used as the control point.
  • the position / orientation of the camera mounter with surveying markers in the global coordinate system is converted into the local coordinate system of the camera mounter, which is used as the initial value of the camera external parameter matrix.
  • Calibration is performed by solving the equation (5). Further, although optimization is performed for each camera based on direct measurement by the surveying instrument, bundle adjustment of a plurality of cameras may be performed based on direct measurement by the surveying instrument. This is the evaluation function Can be treated as a minimization problem.
  • Non-Patent Document 1 the camera calibration data and the video motion capture system (see Non-Patent Document 1) were used to reconstruct the whole-body motions of 12 players during a volleyball game.
  • the motion capture camera system according to the present embodiment can be applied to any motion measurement without being limited to the type and location of sports.
  • target sports include soccer, futsal, rugby, baseball, volleyball, handball, tennis, and gymnastics.
  • Examples of the shooting location include an outdoor sports field, an indoor sports field, an outdoor court, an indoor court, an arena, a dome, and a gymnasium.
  • the calibration according to the present embodiment should be applied to video motion capture from a plurality of camera images for broadcasting / recording in which a cameraman manually manipulates the focal length, pan / tilt, and the like.
  • These parameters are calculated by using fixed feature points (for example, points on the line or points on the goal) on elements that form a sports field or court, and feature points of unknown positions (for example, balls in ball games). All parameters of time are dynamically obtained by optimization calculation.
  • position information in a global coordinate system of a plurality of immovable feature points is acquired and stored by a surveying instrument having a angle measuring function (which may have a distance measuring function).
  • Fixed feature points are points on elements that form a sports field or court, such as intersections (including corners) of lines on the field or court, intersections, and points on goal posts (intersection between post and crossbar). The lower end of a part or a post) can be illustrated.
  • a ball in a ball game can be exemplified as the feature point whose position information in the global coordinate system is unknown. Moreover, you may use the mark of the specific position on the uniform which the player is wearing, or the specific point of the shoes as the feature point whose position is unknown.
  • the variation width of some parameters may be set small or the change speed of some parameters may be set small by using an empirical rule.
  • the camera mounter may be configured by a surveying instrument having an angle measuring function, and the posture information of the camera mounter acquired by using the surveying instrument may be used for the optimization calculation.
  • the time series information of the measurement value (angle) of the surveying instrument is acquired and stored, or the optimization calculator receives the measurement value of the surveying instrument in real time and The optimization calculation may be performed with.
  • the point T i is the origin of the local coordinate system of the i-th angle measuring instrument
  • the point Hi is the foot of the perpendicular from T i on the pole
  • the axes X Ti , Y Ti , and X Ti are of the local coordinate system of the angle measuring instrument. Refers to each axis.
  • the angle ⁇ i A is the angle
  • the angle ⁇ i B is the angle
  • Each line segment is as follows. Therefore, the vector Is as follows. If there are actual measured values for the line segments T i A and T i B using the laser rangefinder, consider the values and optimize the vector. Ask.
  • the external parameter matrix i B m representing the position / orientation can be defined as follows.
  • angles i ⁇ horizontalj and i ⁇ verticalj indicate the direction angle and the altitude angle to the marker measured by the angle range finder.
  • the external parameter i B m is defined as follows, as in the previous section.
  • the focal length of the camera and the relative extrinsic parameter matrix between the camera and the mounter it is preferable to obtain the focal length of the camera and the relative extrinsic parameter matrix between the camera and the mounter by optimization. If the focal length is obtained in advance and fixed, a proper camera position can be obtained in the field. At this time, if an angle measuring instrument is used as the camera mounter, the focal length of the camera can be obtained more accurately.
  • the procedure is specifically described. First, for a leveled surveying pole, a level coordinate system is used to define the global coordinate system as described above. Next, the surveying pole is photographed by moving it at regular intervals (1 ° step, ⁇ 10 ° range). This allows control points to be defined on the survey pole (see Figure 15). You can get the position of the control point guaranteed to the accuracy of theodolite.
  • the camera system includes a camera unit including a plurality of (for example, 4 or 2) cameras, and each camera unit is mounted on a surveying instrument (theodolite or total station). Each camera unit is mounted on the movable part of the surveying instrument, and can rotate in the horizontal and vertical directions integrally with the movable part when measuring the angle. In one aspect, one camera unit is mounted on the total station and another camera unit is mounted on the theodolite. Of course, a plurality of camera units may be mounted in a plurality of total stations.
  • i indicates the theodolite serial number
  • j indicates the j-th camera mounted on the i-th theodolite. Note that in section [C] above, i indicates the camera serial number, and the camera mounter (including the theodolite) is assigned the camera serial number.
  • [D-1] Theodolite coordinate system in the world coordinate system
  • the acquisition of the homogeneous conversion matrix from the world coordinate system to the theodolite coordinate system will be described.
  • the theodolite and total station are mounted on a tripod.
  • Set each tripod so that the field of view of each camera includes the measurement range.
  • the surveying pole is a well-known pole generally used in surveying, and for example, red and white stripes are formed with a width of 10 cm.
  • the upper measurement point on the pole is A, and the lower measurement point is B.
  • the coordinate system of the total station is the coordinate system ⁇ 0 ⁇ . Counterclockwise with the theodolite coordinate system as the coordinate system ⁇ 1 ⁇ , ⁇ 2 ⁇ , ..., ⁇ M ⁇ .
  • M is the theodolite number.
  • the Yaw angle between the camera mounters is measured using the theodolites or each total station.
  • the Yaw angle from the coordinate system ⁇ 0 ⁇ to the coordinate system ⁇ 1 ⁇ is represented by ⁇ 01 .
  • the tip of the pendulum weight of the theodolite or total station that is the target is the target (characteristic point).
  • Each camera unit is equipped with four fixed cameras on a horizontal plane that can rotate integrally with the Yaw rotation of the theodolite or total station.
  • the camera is fan-shaped with a sector angle of 22.5 °.
  • the axis is symmetric with respect to the zx plane. Each axis Rotate around 9.65 °.
  • the camera is squeezed open and focused on ⁇ .
  • the amount of light is controlled by the shutter speed. Show the checkerboard to the camera in 15 different positions and orientations.
  • the distortion-free image I ij * is generated from I ij using P ij in the following format,
  • the distortion-free image is the three-dimensional position of the camera coordinate system. To Indicate.
  • An optimization calculation is performed so as to satisfy the above two equations, and a homogeneous transformation matrix from the theodolite coordinate system to the camera coordinate system. To get. Similarly, the simultaneous conversion matrix is acquired for the next camera j, and the same conversion matrix is acquired for the next theodolite to acquire the simultaneous conversion matrix.
  • the result of the calibration according to this embodiment is represented by the following three formulas.
  • the motion capture system is a so-called video motion capture system (see Non-Patent Document 1), and three-dimensional reconstruction is performed from joint positions estimated using deep learning from images of a plurality of cameras acquired by the motion capture camera system.
  • the target does not need to attach any marker or sensor, and the measurement space is not limited.
  • Motion capture is performed completely unconstrained from the images of multiple RGB cameras. In principle, if images can be acquired from indoor living spaces to large outdoor sports fields, motion measurement is possible. It is a technology.
  • the motion capture system described below is an example, and the motion capture used with the motion capture camera system according to the present invention is not limited, and other methods may be adopted.
  • each image includes multiple people.
  • the joint positions of one person selected from the plurality of persons or an arbitrary number of a plurality of persons are acquired.
  • one image contains multiple people, for example, PAF and PCM (Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields.InInProceedings By using IEEE Conference on Computer Vision and Pattern Recognition.CVPR 2017, 2017.), joint position can be acquired for each person at the same time.
  • the object has a link structure or an articulated structure.
  • the object is typically a human with a skeletal structure, but the object may be a robot.
  • the video motion capture system uses the motion capture camera system that captures the motion of the target and the degree of certainty of the position of the feature points (Keypoints) including joint positions in color intensity based on the images captured by the camera system.
  • the heat map acquisition unit that acquires the heat map information to be displayed, the joint position acquisition unit that acquires the target joint position using the heat map information acquired by the heat map acquisition unit, and the joint that is acquired by the joint position acquisition unit
  • a smoothing processing unit that smoothes the position
  • a storage unit that stores the skeletal structure of the target body, time series data of images acquired by the video system, time series data of the joint positions acquired by the joint position acquisition unit, and the like.
  • the motion capture / camera system includes a plurality of cameras, and a plurality of synchronized cameras capture the motion of the subject, and each camera outputs an RGB image at a predetermined frame rate.
  • a plurality of camera images acquired at the same time are transmitted to the heat map acquisition unit.
  • the heat map acquisition unit generates a heat map based on the RGB image.
  • the heat map represents the spatial distribution of the likelihood of the position of the feature points on the body.
  • the heat map acquisition unit generates a two-dimensional or three-dimensional spatial distribution of the likelihood of the likelihood of the position of the feature points (keypoints) on the body including each joint position based on the input image, and calculates the likelihood of the likelihood. Display the spatial distribution in heat map format.
  • the heat map acquisition unit typically uses a convolutional neural network (CNN) to calculate the position (typically a joint position) of a feature point on the target body from a single input image as a heat map. Estimate as.
  • CNN convolutional neural network
  • the generated heat map information is transmitted to the joint position acquisition unit, and the joint position acquisition unit acquires the joint position.
  • the joint position acquisition unit estimates the joint position candidate using the heat map information acquired from the heat map acquisition unit, and executes the optimization calculation based on the inverse kinematics using the joint position candidate to calculate the skeleton model. Update the joint angle and joint position.
  • the joint position acquisition unit estimates the joint position candidate based on the heat map data, and the joint position candidate inverse motion that performs the optimization calculation based on the inverse kinematics using the joint position candidate to calculate the joint angle.
  • a forward kinematics calculation unit that performs forward kinematics calculation using the calculated joint angle to calculate the joint position.
  • the acquired joint position data is stored in the storage unit as time series data of the joint position.
  • the acquired joint position data is transmitted to the smoothing processing unit, and the smoothed joint position and the joint angle are acquired.
  • the smoothed joint position acquisition unit of the smoothing processing unit performs smoothing processing on the acquired joint position using the joint position in the past frame to smooth the temporal movement of the joint position.
  • the target pose is determined based on the smoothed joint position or joint angle and the skeletal structure of the target body, and the target motion made up of time series data of the pose is displayed on the display.
  • FIG. 17 exemplifies the processing steps of motion analysis using motion capture.
  • the target motion is acquired by the motion capture according to this embodiment.
  • Time series data of the acquired joint angle and joint position is acquired.
  • the joint torque is obtained by inverse dynamics calculation, and the joint torque is used to optimize the wire tension in the musculoskeletal model including the wire imitating a muscle (quadratic programming or linear programming).
  • Method calculate the muscle activity using the wire tension, generate a musculoskeletal image with colors assigned according to the degree of muscle activity, and visualize the musculoskeletal with muscle activity.
  • the image is output at a predetermined frame rate and displayed on the display as a moving image. In this way, it is possible to automatically and efficiently perform the process of capturing the motion of the target, the acquisition of the three-dimensional pose of the target during the motion, and the estimation and visualization of the muscle activity required for the motion.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Studio Devices (AREA)
  • Image Analysis (AREA)

Abstract

La présente invention réalise simplement un étalonnage sur une caméra avec une grande précision même dans un grand espace de capture d'image pour une capture de mouvement. L'invention concerne un procédé d'étalonnage pour acquérir des paramètres de caméra pour reconfigurer en trois dimensions une pluralité d'images de caméra, les paramètres de caméra comprenant des paramètres externes indiquant la position et l'orientation de chaque caméra dans un système de coordonnées global, le procédé d'étalonnage comprenant: l'acquisition de la position et de l'orientation de chaque dispositif de montage de caméra dans le système de coordonnées global en utilisant un ou plusieurs instruments de mesure ayant une fonction de mesure d'angle; et l'acquisition des paramètres externes en utilisant la position et l'orientation de chaque dispositif de montage de caméra dans le système de coordonnées global.
PCT/JP2019/045552 2018-11-22 2019-11-21 Système de caméra à capture de mouvement et procédé d'étalonnage WO2020105697A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2020557612A JP7395189B2 (ja) 2018-11-22 2019-11-21 モーションキャプチャ・カメラシステム及びキャリブレーション方法

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201862770766P 2018-11-22 2018-11-22
US62/770,766 2018-11-22

Publications (1)

Publication Number Publication Date
WO2020105697A1 true WO2020105697A1 (fr) 2020-05-28

Family

ID=70774209

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/045552 WO2020105697A1 (fr) 2018-11-22 2019-11-21 Système de caméra à capture de mouvement et procédé d'étalonnage

Country Status (2)

Country Link
JP (1) JP7395189B2 (fr)
WO (1) WO2020105697A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113487726A (zh) * 2021-07-12 2021-10-08 北京未来天远科技开发有限公司 动作捕捉系统和方法
CN115861592A (zh) * 2023-02-16 2023-03-28 之江实验室 基于神经网络的动作捕捉系统的速度精度优化方法及系统
JP7489349B2 (ja) 2021-03-30 2024-05-23 Kddi株式会社 複数カメラ校正装置、方法およびプログラム

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH11118425A (ja) * 1997-10-17 1999-04-30 Sony Corp キャリブレーション方法及び装置、並びにキャリブレーション用データ生成方法
JP2016126425A (ja) * 2014-12-26 2016-07-11 Kddi株式会社 自由視点画像生成装置、方法およびプログラム

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3597832B2 (ja) 2002-07-19 2004-12-08 ジェイアール西日本コンサルタンツ株式会社 軌道狂い計測方法およびその方法に用いられる軌道狂い計測システム
JP2004163292A (ja) 2002-11-13 2004-06-10 Topcon Corp 測量装置と電子的記憶媒体
JP5366258B2 (ja) 2010-02-17 2013-12-11 Ｋｄｄｉ株式会社 大空間カメラ配置における幾何情報に基づく仮想視点画像生成方法およびプログラム
EP2383696A1 (fr) 2010-04-30 2011-11-02 LiberoVision AG Procédé pour l'estimation d'une pose d'un modèle d'objet articulé

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH11118425A (ja) * 1997-10-17 1999-04-30 Sony Corp キャリブレーション方法及び装置、並びにキャリブレーション用データ生成方法
JP2016126425A (ja) * 2014-12-26 2016-07-11 Kddi株式会社 自由視点画像生成装置、方法およびプログラム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
KITAHARA, ITARU ET AL.: "Recording of Multiple Videos in a Large-scale Space for Large-scale Virtualized Reality", THE JOURNAL OF THE INSTITUTE OF IMAGE INFORMATION AND TELEVISION ENGINEERS, vol. 56, no. 8, 1 August 2002 (2002-08-01), pages 120, XP007015542 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7489349B2 (ja) 2021-03-30 2024-05-23 Kddi株式会社 複数カメラ校正装置、方法およびプログラム
CN113487726A (zh) * 2021-07-12 2021-10-08 北京未来天远科技开发有限公司 动作捕捉系统和方法
CN113487726B (zh) * 2021-07-12 2024-05-14 未来元宇数字科技(北京)有限公司 动作捕捉系统和方法
CN115861592A (zh) * 2023-02-16 2023-03-28 之江实验室 基于神经网络的动作捕捉系统的速度精度优化方法及系统

Also Published As

Publication number Publication date
JPWO2020105697A1 (ja) 2021-10-07
JP7395189B2 (ja) 2023-12-11

Similar Documents

Publication Publication Date Title
CN108961395B (zh) 一种基于拍照重建三维空间场景的方法
JP7395189B2 (ja) モーションキャプチャ・カメラシステム及びキャリブレーション方法
US8335345B2 (en) Tracking an object with multiple asynchronous cameras
WO2020054442A1 (fr) Procédé et dispositif d'acquisition de position d'articulation, et procédé et dispositif d'acquisition de mouvement
US6774932B1 (en) System for enhancing the televised broadcast of a golf game
US8223208B2 (en) Device and method for calibrating an imaging device for generating three dimensional surface models of moving objects
WO1994029670A1 (fr) Appareil pour mesurer la position d'un objet en mouvement
US20070098250A1 (en) Man-machine interface based on 3-D positions of the human body
CN113160325B (zh) 基于进化算法的多摄像机高精度自动标定方法
US20150097937A1 (en) Single-camera motion capture system
JP7427188B2 (ja) ３ｄポーズ取得方法及び装置
US11657578B2 (en) Registration for augmented reality system for viewing an event
US20230118280A1 (en) Use of multiple registrations for augmented reality system for viewing an event
JP2004132831A (ja) 球体の回転特性と飛行特性の測定方法及び球体の回転特性と飛行特性の測定装置
CN113884081A (zh) 测定定位点三维坐标的方法及设备
JP4205004B2 (ja) キャリブレーション方法
US11645819B2 (en) Augmented reality system for viewing an event with mode based on crowd sourced images
US11527047B2 (en) Augmented reality system for viewing an event with distributed computing
Teo Video-based point cloud generation using multiple action cameras
Rau et al. Systematic calibration for a backpacked spherical photogrammetry imaging system
Nocerino et al. Low-cost human motion capture system for postural analysis onboard ships
WO2022192064A1 (fr) Système de réalité augmentée pour visualiser un événement avec de multiples systèmes de coordonnées et modèle généré automatiquement
Chong et al. A photogrammetric application in virtual sport training
CN113506210A (zh) 一种篮球比赛自动生成运动员点位图的方法和视频拍摄装置
CN113274733A (zh) 一种高尔夫球置顶式检测方法、系统及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19887575

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020557612

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19887575

Country of ref document: EP

Kind code of ref document: A1