Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N17/00—Diagnosis, testing or measuring for television systems or their details
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules

Definitions

• This disclosure relates to a camera calibration method, a reference point setting method, a program, and a camera calibration device.
• Patent Document 1 In applications of digital cameras such as vehicle-mounted cameras, it may be necessary to calibrate the camera (see, for example, Patent Document 1).
• a stereo camera captures an image of a road surface with marks, and the captured image is processed to calibrate the installation parameters.
• the purpose of this disclosure is to simplify camera calibration.
• a first aspect is a method for calibrating a plurality of cameras arranged at different positions, the method comprising the steps of: A first step of capturing an image of a calibration subject and acquiring captured image information; A second step of calibrating the cameras based on the imaging information;
• the calibration subject satisfies the following requirements: Requirement 1: The calibration object has at least six reference points that are distinguishable from one another on the images captured by each of the multiple cameras.
• Requirement 2 At least six of the reference points have a non-coplanar relationship.
• the imaging information is obtained by capturing an image with each of the multiple cameras
• the calibration is performed using imaging information of at least six reference points that satisfy all of the requirements.
• At least six common reference points can be captured by all cameras.
• multiple captures are not required to obtain information on the reference points. Therefore, in this aspect, the task of moving the reference points for each capture is not required.
• parameters related to the camera position and camera attitude required for camera calibration can be obtained with at least one capture. This leads to simplification of camera calibration.
• a second aspect is the camera calibration method of the first aspect, In the second step, the reference points are distinguished from one another based on a mechanism or arrangement that enables distinction between at least six of the reference points within one captured image.
• the reference points are distinguished from one another based on a mechanism or arrangement that allows for the distinction of at least six reference points within one captured image.
• the reference points 201 can be reliably distinguished from one another.
• a third aspect is the camera calibration method of the first or second aspect
• the first step is a camera calibration method using the calibration subject in which the positional relationship between the reference points is fixed in a manner that does not impede visibility of the reference points from any angle.
• the reference point is visible from all angles.
• the calibration subject can be easily positioned.
• a fourth aspect is a method for setting the reference point in the camera calibration method of the first aspect, comprising the steps of: An initialization step of selecting three points having different positions as the group of reference points; a setting step of adding a point D that satisfies a predetermined setting condition for all combinations of three points selected from the point group to the point group; Including, The set conditions are: The three reference points in the combination under consideration are point A, point B, and point C.
• the vector passing through points A and D is V AD .
• the vector passing through points A and B is V AD .
• the setting step is repeated until a desired number of the reference points are obtained.
• a fifth aspect is a program for performing a calibration process for a plurality of cameras arranged at different positions, comprising: The program causes a computer to: a first processing unit that captures an image of a calibration subject and acquires imaging information; a second processing unit that calibrates the plurality of cameras based on the imaging information; The calibration subject satisfies the following requirements: Requirement 1: The calibration object has at least six reference points that are distinguishable from one another on the images captured by each of the multiple cameras. Requirement 2: At least six of the reference points have a non-coplanar relationship.
• the first processing unit obtains the imaging information by capturing an image with each of the multiple cameras
• the second processing unit is a program that distinguishes between the reference points based on a mechanism or arrangement that enables distinguishing at least six of the reference points within one captured image, and extracts the reference points that satisfy requirement 2 to perform the calibration.
• all cameras can capture at least six common reference points.
• a sixth aspect of the present invention is a method for controlling a computer-readable storage medium according to the fifth aspect of the present invention.
• camera calibration is simplified in a multiple camera calibration device.
• This disclosure allows for simplified camera calibration.
• FIG. 2 is a block diagram showing a configuration of a calibration device.
• FIG. 2 is a perspective view of a calibration subject.
• FIG. 1 is a diagram showing a schematic diagram of a calibration site.
• 13 is a flowchart showing a process performed by a calibration program.
• 13 is a flowchart showing a setting process performed by a calibration unit.
• FIG. 13 is a block diagram showing a configuration of a calibration device according to a modified example of the embodiment.
• a camera calibration device 10 will be described.
• the calibration device 10 simultaneously calibrates a plurality of cameras (described later). These cameras are installed in, for example, a factory.
• FIG. 1 is a block diagram showing the configuration of a calibration device 10.
• a plurality of cameras 300 also called imaging units
• the number of cameras 300 is not limited to the example in FIG. 1, as long as it is two or more.
• the cameras 300 are digital cameras. Each camera 300 is connected to the calibration device 10 by a serial bus B.
• a serial bus B For example, a Universal Serial Bus (abbreviated as USB) can be used as the serial bus B.
• USB Universal Serial Bus
• FIG. 1 the image transmission line is represented by the serial bus B, but a parallel bus may be used instead of the serial bus.
• a signal instructing camera 300 to capture an image (hereinafter referred to as an imaging instruction signal) can be sent to camera 300 via serial bus B.
• Camera 300 has the function of capturing a still image (in this example, a color image) when it receives an imaging instruction signal.
• Camera 300 outputs digital data of the captured image (hereinafter referred to as imaging information) to calibration device 10 via serial bus B.
• the calibration device 10 When calibrating each camera 300, the calibration device 10 causes each camera 300 to capture an image of the calibration subject 200. First, the calibration subject 200 will be described.
• Calibration subject 200 2 is a perspective view of a calibration subject 200.
• the calibration subject 200 has a plurality of reference points 201 and a plurality of connecting portions 202 connecting the reference points 201.
• Each of the reference points 201 is an object to be imaged by a camera 300.
• the calibration subject 200 satisfies the following two requirements.
• the calibration subject 200 has at least six reference points 201 that can be distinguished from one another on each of the images captured by the multiple cameras 300.
• Requirement 2 At least six of the reference points 201 have a non-coplanar relationship.
• a non-coplanar relationship means that four or more reference points 201 do not exist on the same plane.
• the reference point 201 and the connecting portion 202 are configured as follows.
• Each reference point 201 is configured with a light-emitting element.
• a light-emitting diode LED
• LED light-emitting diode
• each light-emitting diode (reference point 201) emits light with a different wavelength.
• each reference point 201 emits light with a different color. Therefore, in an image captured of the reference points 201 emitting light, the reference points 201 can be distinguished from one another by the difference in color. Therefore, the calibration subject 200 satisfies requirement 1.
• Each connecting part 202 is a triangular plate member.
• the connecting parts 202 are made of a transparent material.
• the connecting parts 202 can be made of a transparent acrylic plate, for example. In the calibration subject 200, these connecting parts 202 are combined to form an octahedron.
• reference points 201 are fixed by eight connecting parts 202.
• the reference points 201 i.e., LEDs
• the reference points 201 are placed on each vertex of an octahedron formed by the eight connecting parts 202.
• One reference point 201 corresponds to one vertex of the octahedron.
• the calibration subject 200 has a mechanism or arrangement that allows for the distinction of at least six reference points 201 within one captured image.
• the calibration device 10 can be configured, for example, by installing a predetermined program (hereinafter, referred to as a calibration program) in a personal computer, an embedded computer, etc.
• the calibration device 10 may be configured using an FPGA (field-programmable gate array), or may be configured as a circuit (computer) within the camera 300.
• the calibration device 10 includes an input interface 11, a calculation processing execution unit 12, a memory unit 13, and an output interface 14.
• the input interface 11 is an interface that exchanges data between the camera 300 and the calibration device 10. Multiple cameras 300 can be connected to the input interface 11.
• the memory unit 13 is composed of storage devices such as ROM (Read Only Memory) and RAM (Random access memory). Various programs including a calibration program are stored in the memory unit 13. The memory unit 13 may also store data such as external parameters (described below), internal parameters (described below), captured images, etc.
• the calculation processing execution unit 12 (also known as the control unit) is equipped with a processor (not shown) that executes a calibration program.
• the calibration program is a program that performs calibration processing for multiple cameras that are placed in different positions. By executing the calibration program, the calculation processing execution unit 12 functions as a calibration control unit 121, an information acquisition unit 122, and a calibration unit 123.
• the calibration control unit 121 controls the calibration of the multiple cameras 300. Calibration is to estimate the external parameters and internal parameters of the camera. In particular, the calibration in this embodiment is to estimate the external parameters and internal parameters for each of the multiple cameras 300.
• the external parameters are parameters that indicate the position and orientation of the camera in three-dimensional space.
• the camera position is expressed in the world coordinate system.
• the camera orientation is expressed by the camera's rotation angle with respect to each coordinate axis.
• Internal parameters are parameters that indicate the focal length of a camera's lens, the optical center of a camera's image sensor, etc. In what follows, both external and internal parameters may be collectively referred to simply as parameters.
• the calibration control unit 121 has a user interface (e.g., Graphical User Interface, abbreviated as GUI) function in order to receive various instructions from the user.
• GUI Graphical User Interface
• the calibration control unit 121 controls the operation (calibration process) of the information acquisition unit 122 and the calibration unit 123 by sending a predetermined control signal to the information acquisition unit 122 and the calibration unit 123.
• the information acquisition unit 122 (also referred to as the first processing unit) causes the camera 300 to capture an image of the calibration subject 200 and acquires imaging information. Specifically, the information acquisition unit 122 sends an imaging instruction signal to each camera 300 via the input interface 11 in response to an instruction (control) from the calibration control unit 121.
• Each camera 300 that receives the imaging instruction signal performs imaging and transmits the imaging information to the information acquisition unit 122.
• the information acquisition unit 122 stores the received imaging information in the memory unit 13 together with information that identifies the camera 300 (e.g., a number assigned to each camera 300).
• the calibration unit 123 (also referred to as the second processing unit) calibrates each camera 300 using imaging information of at least six reference points 201 that satisfy all of requirements 1 and 2.
• the calibration unit 123 starts the calibration in response to an instruction (control) from the calibration control unit 121.
• the calibration unit 123 has a function (hereinafter referred to as function 1) of distinguishing between reference points 201 on the captured image of the calibration subject 200. Furthermore, the calibration unit 123 has a function (hereinafter referred to as function 2) of extracting at least six reference points 201 that satisfy requirement 2 from the distinguished reference points 201.
• the calibration unit 123 performs the following processing when performing function 1.
• the calibration unit 123 executes a calculation to identify the center of gravity of the extracted region (center of gravity calculation process). Furthermore, the calibration unit 123 performs labeling so that the identified center of gravity positions can be distinguished (labeling process).
• the calibration unit 123 performs the area identification process, the center of gravity calculation process, and the labeling process for all the hues of the reference points 201 in the calibration subject 200. As a result, for a given image, pairs of reference points 201 and their position information are obtained in the same number as the number of extracted reference points 201.
• the calibration unit 123 performs the following processing when function 2 is performed.
• the calibration unit 123 extracts four reference points 201 in a certain captured image (assumed to be image 1).
• the four extracted reference points 201 are designated as O, P1, P2, and P3. Note that O is an alphabet (not a number).
• equation 1 can be constructed based on O, P1, P2, and P3.
• OP1 s ⁇ OP2+t ⁇ OP3 ... Equation 1
• OP1, OP2, and OP3 are vectors.
• the calibration unit 123 also constructs Equation 1 for another captured image (Image 2) using the same four points.
• Image 2 is an image captured by a camera 300 other than the camera 300 that captured Image 1. If the pair of s, t for Image 1 and the pair of s, t for Image 2 have the same values, the calibration unit 123 excludes P3 from the processing target. The calibration unit 123 repeats this operation (selection and exclusion of reference points 201) until at least the number of reference points 201 required for the calibration calculation (described below) is obtained.
• the calibration unit 123 has a function of performing calculations for calibration (hereinafter referred to as calibration calculations).
• the calibration unit 123 uses at least six reference points 201 in the calibration calculations. At least six reference points 201 are required in the calibration calculations because at least six different pieces of information regarding the position and orientation are required to identify the position and orientation of the camera in three-dimensional space.
• the calibration unit 123 uses a known method. In this embodiment, the calibration unit 123 finds the internal and external parameters of the camera 300 by a nonlinear optimization process called bundle adjustment.
• the calibration unit 123 may also store each parameter in the memory unit 13.
• the output interface 14 outputs each parameter to its intended use. There are no limitations on the output destination (use). For example, the output interface 14 outputs each parameter to an image processing device or the like. The image processing device can use these parameters to correct lens distortion, measure the size of an object (image subject) in the world coordinate system, determine the camera position within the scene, and so on.
• Fig. 3 is a diagram showing a schematic diagram of a calibration site.
• five cameras 300 are present. These cameras 300 are installed in advance at the place of use (e.g., a factory). These cameras 300 are the objects of calibration.
• a user control device is connected to the calibration device 10 (see Fig. 3).
• the user control device is a device that can transmit a predetermined instruction (such as a command to start calibration) to the calibration device 10 and display the output and held data (such as captured images) of the calibration device 10.
• the calibration subject 200 is placed in a position where six or more reference points 201 can be captured by all cameras 300 to be calibrated. Each reference point 201 is set to an illuminated state. It is also possible to indicate in a software menu of a user control device connected to the calibration device 10 that the calibration subject 200 explicitly has six or more reference points.
• the user control device may be configured so that the user can instruct calibration with a calibration execution button in the software (GUI).
• GUI calibration execution button in the software
• the calibration device 10 may also automatically determine that there are six or more reference points before performing calibration.
• the software of the user control device may notify the user or the calibration device 10 of completion.
• FIG. 4 is a flowchart showing the processing by the calibration program. This processing is started when a command to start calibration is given by the user. The processing in this flowchart is managed by the calibration control unit 121.
• step St01 (also referred to as the first step) in FIG. 4, imaging information is acquired.
• the calibration control unit 121 instructs the information acquisition unit 122 to acquire imaging information.
• the information acquisition unit 122 sends an imaging instruction signal to each camera 300 via the input interface 11.
• each camera 300 captures an image of the calibration subject 200 and sends the image information to the calibration device 10 (information acquisition unit 122) via serial bus B.
• the information acquisition unit 122 stores the received image information in the memory unit 13. Note that in this example, the information acquisition unit 122 causes each camera 300 to capture an image at least once (one image).
• step St02 a reference point 201 is extracted from within the captured image.
• the calibration control unit 121 instructs the calibration unit 123 to extract the reference point 201.
• the calibration unit 123 performs function 1.
• the reference point 201 is extracted from within the captured image.
• step St03 reference points 201 that are non-coplanar with each other are selected. Specifically, in the processing of step St03, the calibration unit 123 performs function 2. In the calibration subject 200 of this embodiment, all of the reference points 201 (six points) satisfy requirement 2. Therefore, when the calibration unit 123 performs function 2, all of the reference points 201 on the calibration subject 200 are extracted.
• step St04 each parameter is estimated.
• the calibration control unit 121 instructs the calibration unit 123 to estimate the parameters.
• the calibration unit 123 performs bundle adjustment to estimate the external parameters and internal parameters so that the positions of the reference points in the images captured by each camera 300 substantially coincide in the world coordinate system.
• step St02 the series of steps from step St02 to step St04 is a step (called the second step) in which the calibration is performed using imaging information of at least six reference points that satisfy both requirements 1 and 2.
• step St05 the output interface 14 outputs each parameter to the destination where the parameter will be used.
• the calibration subject 200 that satisfies requirements 1 and 2 is used.
• at least six common reference points 201 can be captured by all of the cameras 300 to be calibrated.
• the calibration device 10 does not need to input information such as the distance between multiple cameras or the angle between them (convergence angle). Therefore, in this embodiment, it is possible to simplify camera calibration.
• the calibration device 10 can obtain information on at least six common reference points 201 from one image.
• the necessary information on the reference points 201 can be obtained with one image capture without moving the calibration subject 200. From this perspective, this embodiment also makes it possible to simplify camera calibration.
• the calibration device 10 can calibrate multiple cameras simultaneously. From this perspective, this embodiment also makes it possible to simplify camera calibration.
• step St02 in Fig. 4 the reference points 201 in the captured image are extracted.
• the calibration unit 123 extracts (sets) six reference points.
• a setting method (setting process) for setting the six reference points 201 by the calibration unit 123 will be described.
• FIG. 5 is a flowchart showing the setting process by the calibration unit. The process in FIG. 5 is performed in step St02.
• step St11 initialization step in FIG. 5
• three reference points 201 are set. Specifically, in the processing of step St11, the calibration unit 123 sets three reference points 201 at different positions in space from each other from the captured image. For ease of explanation, the three set reference points 201 are referred to as point A, point B, and point C.
• step St12 (setting step), a point D is set that has a non-coplanar relationship with points A, B, and C.
• the calibration unit 123 sets point D so that there are no real numbers s and t that satisfy the following formula 2.
• V AD s ⁇ V AB + t ⁇ V AC Equation 2
• V AD is a vector passing through points A and D
• V AB is a vector passing through points A and B
• V AC is a vector passing through points A and C.
• step St12 sets four non-coplanar reference points.
• step St13 point E is set to have a non-coplanar relationship with points A, B, C, and D.
• the calibration unit 123 selects three reference points 201 from points A, B, C, and D.
• the calibration unit 123 sets point E so that it has a non-coplanar relationship with the three selected reference points 201.
• point D in equation 2 becomes point E, and points A, B, and C become the three selected reference points 201.
• step St13 there are four possible combinations for selecting three reference points 201 from the four reference points 201, points A, B, C, and D. Therefore, in step St13, the calibration unit 123 sets point E so that it is in a non-coplanar relationship with the three selected reference points 201 for each of the four patterns.
• step St13 sets five non-coplanar reference points.
• step St14 point F is set to have a non-coplanar relationship with points A, B, C, D, and E.
• the calibration unit 123 selects three reference points 201 from points A, B, C, D, and E.
• the calibration unit 123 sets point F so that it has a non-coplanar relationship with the three selected reference points 201.
• point D in equation 2 becomes point F
• points A, B, and C become the three selected reference points 201.
• step St14 there are 20 possible combinations for selecting three reference points 201 from the five reference points 201, points A, B, C, D, and E. Therefore, in step St13, the calibration unit 123 sets point F so that it is in a non-coplanar relationship with the three selected reference points 201 for each of the 20 patterns.
• the calibration unit 123 can extract (set) six reference points.
• k reference points 201 (k is a number equal to or greater than 7) are set, it is possible to set k reference points by selecting three reference points 201 from the k non-coplanar points and extracting (setting) the non-coplanar points for each pattern.
• Modifications of the embodiment 6 is a block diagram showing the configuration of a calibration device according to a modification of the embodiment.
• the calibration device 10 of this modification includes a network interface 15 instead of the input interface 11 (interface with the camera 300) in the calibration device 10 of the embodiment.
• the network interface 15 communicates with the outside of the calibration device 10 via a network.
• the network interface 15 has a function for connecting to a serial bus.
• the network interface 15 may also communicate with a parallel bus.
• a network hub 310 is connected to the network interface 15.
• the network hub 310 is a device for connecting multiple cameras 300 to a network.
• multiple cameras 300 are connected to the network hub 310. This configuration enables the calibration device 10 to exchange data with the multiple cameras 300.
• the only difference between the calibration device 10 of this modified example and the calibration device 10 of the above embodiment is the interface for connecting the camera 300 and the calibration device 10. Therefore, the camera calibration of this modified example can achieve the same effect as the above embodiment.
• the reference points 201 may be distinguished from one another by the blinking interval of a light-emitting element that is provided as the reference point 201 and blinks.
• a video is captured by the camera 300.
• the period of video capture may be determined taking into consideration the blinking cycle of the reference points 201. In the case of video capture, the period from the start to the end of video capture is "one capture", regardless of the number of frames of the capture.
• the reference points 201 do not necessarily need to have the ability to emit light, so long as the components used as the reference points 201 have different colors.
• the reference points 201 may be distinguished from one another based on the shape of the member provided as the reference point 201.
• the reference point 201 may be configured as a polyhedron.
• the shape can be distinguished, for example, by AI (artificial intelligence).
• a polyhedron made of only support rods may be used as the calibration subject 200, and the vertices of the polyhedron may be used as the reference points 201.
• the reference points 201 may be distinguished from one another by a combination of multiple techniques (e.g., by color, by shape, etc.).
• the number of reference points 201 (six) described as an embodiment is merely an example.
• the number of reference points 201 may be six or more as long as requirements 1 and 2 are met.
• the shape of the connecting part 202 described as an embodiment is merely an example.
• the connecting part 202 may be a rod-shaped member.
• the connecting part 202 is configured from a rod-shaped member, it is not necessarily required to configure the connecting part 202 from a transparent member.
• a transparent rod-shaped member or a transparent curved member may be used as the connecting part 202.
• the configuration of the connecting part 202 is arbitrary as long as the positional relationship between the reference points is fixed in a manner that does not substantially impede the visibility of the reference points 201 from any angle.
• the connecting part 202 is transparent or has a thin shape, even if the visibility of a part of the reference points 201 is obstructed, the information from the reference points 201 can be confirmed sufficiently, so in this case, it can be said that the visibility is not substantially obstructed.
• the calibration calculation method in the calibration unit 123 is not limited to bundle adjustment.
• the installation location (application) of the camera 300 is not limited to a factory.
• the camera 300 to be calibrated may be a camera for in-vehicle use, business use, medical use, consumer use, etc.
• the present disclosure is useful as a camera calibration method, program, and camera calibration device.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Health & Medical Sciences (AREA)
• Biomedical Technology (AREA)
• General Health & Medical Sciences (AREA)
• Studio Devices (AREA)
• Length Measuring Devices By Optical Means (AREA)

Priority Applications (1)

Applications Claiming Priority (2)

Publications (1)

Family

ID=91485510

Family Applications (1)

Country Status (2)

Citations (3)

• 2023
  • 2023-11-02 JP JP2024564207A patent/JPWO2024127846A1/ja active Pending
  • 2023-11-02 WO PCT/JP2023/039698 patent/WO2024127846A1/ja not_active Ceased

Patent Citations (3)

Also Published As

Similar Documents

Legal Events