WO2023241372A1 - 相机内参标定方法及相关设备 - Google Patents

相机内参标定方法及相关设备 Download PDF

Info

Publication number
WO2023241372A1
WO2023241372A1 PCT/CN2023/097798 CN2023097798W WO2023241372A1 WO 2023241372 A1 WO2023241372 A1 WO 2023241372A1 CN 2023097798 W CN2023097798 W CN 2023097798W WO 2023241372 A1 WO2023241372 A1 WO 2023241372A1
Authority
WO
WIPO (PCT)
Prior art keywords
internal parameter
internal
image
target terminal
parameter calibration
Prior art date
Application number
PCT/CN2023/097798
Other languages
English (en)
French (fr)
Inventor
王润之
徐其超
冯思淇
李江伟
时天欣
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2023241372A1 publication Critical patent/WO2023241372A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Definitions

  • Embodiments of the present application relate to the field of image processing, and in particular, to a camera internal parameter calibration method and related equipment.
  • In-camera parameters are parameters related to the camera's own characteristics, such as the camera's focal length, pixel size, etc.
  • Camera internal parameter calibration technology is one of the key research contents in the fields of photogrammetry and computer vision. It is the basis of technologies in visual positioning, robot visual navigation mapping, industrial control and other fields.
  • AR augmented reality
  • Algorithms in AR applications usually require a priori camera parameters as input.
  • the corresponding in-camera parameters are also different.
  • the calibration board can be used to complete the internal parameter calibration of the terminal camera. This method requires the internal parameters of each model to be obtained and stored through offline calibration in advance, and then the internal parameters of the camera can be read and called in the AR application algorithm.
  • this calibration method requires a lot of manpower and material resources, and it cannot obtain real-time camera internal parameters for subsequent calculations of AR applications.
  • This application provides a camera internal parameter calibration method and related equipment, which can calibrate the camera's internal parameters in real time, effectively reducing the workload required for camera internal parameter calibration.
  • this application provides a camera internal parameter calibration method, which can be applied to electronic devices.
  • the above-mentioned electronic device is, for example, a server or a smart terminal.
  • the electronic device acquires the first image and the identification of the target terminal.
  • the first image is an image captured by the target terminal using its own camera at a first position, and the first position is located within a preset area.
  • the electronic device determines the first internal parameter calibration data of the camera based on the first image and the high-precision three-dimensional map corresponding to the preset area; and determines the historical internal parameter calibration data of the target terminal based on the identification of the target terminal and the historical internal parameter calibration database.
  • the historical internal parameter calibration database includes historical internal parameter calibration data of one or more identified terminals.
  • the electronic device finally determines the calibrated internal parameters of the camera based on the first internal parameter calibration data and the historical internal parameter calibration data of the target terminal.
  • This solution first determines the first internal parameter calibration data of the camera based on the first image and high-precision three-dimensional map acquired by the target terminal, and determines the historical internal parameter calibration data of the target terminal based on the target terminal's identification and the historical internal parameter calibration database. Then, the calibrated internal parameters of the camera can be determined based on the first internal parameter calibration data and the historical internal parameter calibration data.
  • This method can calibrate the camera internal parameters of the target terminal in real time, improve the calibration efficiency, and effectively reduce the workload required for camera internal parameter calibration.
  • the high-precision three-dimensional map includes multiple scene images, two-dimensional feature point coordinates of the scene images, feature descriptors of the two-dimensional feature points, and three-dimensional space point coordinates corresponding to the two-dimensional feature points.
  • multiple refers to more than or equal to two.
  • the above method determines the first internal parameter calibration data of the camera based on the first image and the high-precision three-dimensional map corresponding to the preset area, including:
  • N N is an integer greater than zero.
  • M M is an integer greater than zero.
  • the coordinate pair includes two-dimensional feature point coordinates and three-dimensional space point coordinates, and X is less than or equal to M.
  • the first internal parameter calibration data is determined based on the coordinate pairs of the X image feature points.
  • the feature descriptors of the image feature points are used to match in a high-precision three-dimensional map to determine the The coordinate pairs corresponding to the image feature points, and then the first internal parameter calibration data can be determined based on the coordinate pairs of the X image feature points.
  • determining the first internal parameter calibration data based on the coordinate pairs of X image feature points includes:
  • the coordinate pairs of X image feature points are determined as the first internal parameter calibration data.
  • determining the calibrated internal parameters of the camera based on the first internal parameter calibration data and the historical internal parameter calibration data of the target terminal includes:
  • the first internal parameters of the camera are determined based on the coordinate pairs of X image feature points.
  • the second internal parameters of the camera are determined based on the coordinate pairs of the X image feature points and the historical internal parameter calibration data of the target terminal. The one with the smallest reprojection error among the first internal parameter and the second internal parameter is determined as the calibration internal parameter.
  • the first internal parameter and the second internal parameter are determined respectively, and then the one with the smallest reprojection error between the two is determined as the calibrated internal parameter of the camera.
  • Using reprojection error comparison can effectively improve the accuracy of the calibrated internal parameter.
  • determining the first internal parameter calibration data based on the coordinate pairs of X image feature points includes:
  • the first internal parameters of the camera are determined based on the coordinate pairs of X image feature points.
  • the first internal parameter is determined as the first internal parameter calibration data.
  • determining the calibrated internal parameters of the camera based on the first internal parameter calibration data and the historical internal parameter calibration data of the target terminal includes:
  • a weighted calculation is performed based on the first internal parameter and its first weight, each historical internal parameter calibration data corresponding to the target terminal and its second weight to determine the second internal parameter of the camera. The one with the smallest reprojection error among the first internal parameter and the second internal parameter is determined as the calibration internal parameter.
  • the first internal parameter and the historical internal parameter calibration data are combined to determine the second internal parameter of the camera using weighted calculation, and then one of the first internal parameter and the second internal parameter is determined as the calibration internal parameter based on the reprojection error. parameter.
  • the first weight is the reciprocal of the reprojection error corresponding to the first internal parameter; the second weight is the reciprocal of the reprojection error corresponding to the historical internal parameter calibration data.
  • the above camera internal parameter calibration method also includes:
  • the electronic device updates the local internal parameters of the target terminal according to the calibrated internal parameters, and saves the first internal parameter calibration data to the historical internal parameter calibration database.
  • the electronic device updates the local internal parameters of the target terminal according to the preset internal parameters.
  • the internal parameter validity conditions are the standards that the preset internal parameters need to meet, and the specific standards can be set according to the actual situation.
  • the electronic device at this time is the above-mentioned target terminal, that is, the method steps of obtaining the calibration internal parameters are executed on the target terminal, and then the validity of the internal parameters is judged.
  • the target terminal Calibrating internal parameters updates its local internal parameters, and saves the first internal parameter calibration data to the historical internal parameter calibration database.
  • the target terminal updates its local internal parameters according to the preset internal parameters.
  • the above camera internal parameter calibration method also includes:
  • the electronic device saves the first internal parameter calibration data into the historical internal parameter calibration database, and sends the calibrated internal parameters to the target terminal, so that the target terminal updates the local location of the target terminal according to the calibrated internal parameters. internal parameters.
  • the electronic device sends the preset internal parameters to the target terminal so that the target terminal updates the local internal parameters of the target terminal.
  • the above-mentioned acquisition of the first image and the identity of the target terminal includes:
  • a first image and an identification of the target terminal are received from the target terminal.
  • this application also provides a camera internal parameter calibration device, which includes an acquisition module, a determination module and a calibration module.
  • the acquisition module is used to acquire the first image and the identification of the target terminal.
  • the first image is an image captured by the target terminal using its own camera at a first position, and the first position is located within a preset area.
  • the determination module is configured to determine the first internal parameter calibration data of the camera based on the first image and the high-precision three-dimensional map corresponding to the preset area; and determine the historical internal parameter calibration data of the target terminal based on the identification of the target terminal and the historical internal parameter calibration database.
  • the historical internal parameter calibration database includes historical internal parameter calibration data of one or more identified terminals.
  • the calibration module is configured to determine the calibration internal parameters of the camera based on the first internal parameter calibration data and the historical internal parameter calibration data of the target terminal.
  • the high-precision three-dimensional map includes multiple scene images, two-dimensional feature point coordinates of the scene images, feature descriptors of the two-dimensional feature points, and three-dimensional space point coordinates corresponding to the two-dimensional feature points.
  • the above determination module is specifically used for:
  • N N is an integer greater than zero.
  • M M is an integer greater than zero.
  • the coordinate pair includes two-dimensional feature point coordinates and three-dimensional space point coordinates, and X is less than or equal to M.
  • the first internal parameter calibration data is determined based on the coordinate pairs of the X image feature points.
  • the determination module is specifically used to determine the first internal parameter calibration data based on the coordinate pairs of X image feature points:
  • the coordinate pairs of X image feature points are determined as the first internal parameter calibration data.
  • Calibration module specifically used for:
  • the first internal parameters of the camera are determined based on the coordinate pairs of X image feature points.
  • the second internal parameters of the camera are determined based on the coordinate pairs of the X image feature points and the historical internal parameter calibration data of the target terminal. The one with the smallest reprojection error among the first internal parameter and the second internal parameter is determined as the calibration internal parameter.
  • the determination module is specifically used to determine the first internal parameter calibration data based on the coordinate pairs of X image feature points:
  • the first internal parameters of the camera are determined based on the coordinate pairs of X image feature points.
  • the first internal parameter is determined as the first internal parameter calibration data.
  • the above calibration module is specifically used for:
  • a weighted calculation is performed based on the first internal parameter and its first weight, each historical internal parameter calibration data corresponding to the target terminal and its second weight to determine the second internal parameter of the camera. The one with the smallest reprojection error among the first internal parameter and the second internal parameter is determined as the calibration internal parameter.
  • the first weight is the reciprocal of the reprojection error corresponding to the first internal parameter; the second weight is the reciprocal of the reprojection error corresponding to the historical internal parameter calibration data.
  • the camera internal parameter calibration device further includes:
  • the processing module is used to update the local internal parameters of the target terminal according to the calibrated internal parameters when the calibrated internal parameters meet the internal parameter validity conditions, and save the first internal parameter calibration data to the historical internal parameter calibration database. And, when the calibrated internal parameters do not meet the internal parameter validity conditions, the local internal parameters of the target terminal are updated according to the preset internal parameters.
  • the camera internal parameter calibration device further includes:
  • the processing module is used to save the first internal parameter calibration data to the historical internal parameter calibration database when the calibration internal parameters meet the internal parameter validity conditions, and send the calibration internal parameters to the target terminal, so that the target terminal updates the target according to the calibration internal parameters. Local intrinsic parameters of the terminal. And, when the calibrated internal parameters do not meet the internal parameter validity conditions, the preset internal parameters are sent to the target terminal, so that the target terminal updates the local internal parameters of the target terminal.
  • the acquisition module is specifically used for:
  • a first image and an identification of the target terminal are received from the target terminal.
  • this application also provides a camera internal parameter calibration device, including a processor and a memory, wherein the processor is connected to the memory, wherein the memory is used to store program code, and the processor is used to call the program code to execute the first The camera internal parameter calibration method described in the aspect.
  • the present application also provides a computer-readable storage medium that stores a computer program, and the computer program is executed by a processor to implement the camera internal parameter calibration method as described in the first aspect.
  • the present application also provides a computer program product containing instructions, which when the computer program product is run on a computer, causes the computer to execute the camera intrinsic parameter calibration method described in the first aspect.
  • the present application also provides a chip.
  • the chip includes a processor and a data interface.
  • the processor reads instructions stored in the memory through the data interface and executes the camera internal parameter calibration method described in the first aspect. .
  • the chip may further include a memory, in which instructions are stored, and the processor is configured to execute the instructions stored in the memory.
  • the processor is configured to execute the camera internal parameter calibration method described in the first aspect.
  • Figure 1 is a schematic diagram of a system architecture provided by an embodiment of the present application.
  • Figure 2 is a schematic structural diagram of a convolutional neural network provided by an embodiment of the present application.
  • Figure 4 is a schematic diagram of a camera internal parameter calibration method provided by an embodiment of the present application.
  • Figure 5a is a schematic flowchart of determining the first internal parameter calibration data provided by an embodiment of the present application.
  • Figure 5b is a schematic flowchart of determining calibration internal parameters provided by an embodiment of the present application.
  • Figure 7 is a schematic structural diagram of a camera internal parameter calibration device provided by an embodiment of the present application.
  • At least one mentioned in the embodiments of this application means one or more, and “multiple” means two or more. “At least one of the following” or similar expressions thereof refers to any combination of these items, including any combination of a single item (items) or a plurality of items (items).
  • at least one of a, b, or c can represent: a, b, c, (a and b), (a and c), (b and c), or (a and b and c), where a, b, c can be single or multiple.
  • Machine learning is an artificial intelligence technology. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. Deep learning algorithms can be implemented by multi-layer neural networks (NN). Such multi-layer neural networks are also called “deep learning models”.
  • s 1, 2,...n, n is a natural number greater than 1
  • W s is the weight of x s
  • b is the bias of the neural unit.
  • f is the activation function of the neural unit (Activation Functions), which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer.
  • the activation function can be a Sigmoid function.
  • a neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit.
  • the input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field.
  • the local receptive field can be an area composed of several neural units.
  • Deep Neural Network also known as multi-layer neural network
  • DNN Deep Neural Network
  • the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer.
  • the first layer is the input layer
  • the last layer is the output layer
  • the layers in between are hidden layers.
  • the layers are fully connected, that is to say, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.
  • Convolutional Neural Network is a deep neural network with a convolutional structure.
  • the convolutional neural network contains a feature extractor composed of convolutional layers and subsampling layers.
  • the feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or convolution feature plane (Feature Map).
  • Feature Map convolution feature plane
  • the convolutional layer refers to the neuron layer in the convolutional neural network that convolves the input signal.
  • a neuron can be connected to only some of the neighboring layer neurons.
  • a convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units.
  • the camera imaging system contains a total of four coordinate systems: world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system.
  • the camera coordinate system is centered on the camera, describes the position of the object from the camera perspective, and serves as a bridge between the pixel coordinate system and the world coordinate system.
  • the pixel coordinate system is a digital coordinate system introduced to describe the position of an object in a digital image on a photo.
  • the process of camera imaging is essentially the conversion of the above-mentioned coordinate systems. Specifically, a point in space is converted from the world coordinate system to the camera coordinate system, and then projected onto the imaging plane (that is, converted from the camera coordinate system to the image coordinate system), and finally convert the data on the imaging plane to the image plane (that is, convert from the image coordinate system to the pixel coordinate system).
  • camera calibration is the process of converting from the world coordinate system to the pixel coordinate system.
  • the purpose of camera calibration includes obtaining the intrinsic and extrinsic parameters of the camera.
  • the external parameters of the camera are used to convert coordinates from the world coordinate system to the camera coordinate system.
  • the external parameters are parameters in the world coordinate system, such as the camera's position, rotation direction, etc.
  • the intrinsic parameters of the camera are used to convert the coordinates from the camera coordinate system to the pixel coordinate system.
  • the intrinsic parameters are parameters related to the camera's own characteristics, such as the focal length of the camera (such as X-axis focal length, Y-axis focal length), image principal point coordinates (such as the coordinates of the main image point on the X-axis, the coordinates of the main image point on the Y-axis), etc.
  • the internal parameters of the camera are generally set before the camera leaves the factory, and the specific internal parameters can be calculated through camera calibration.
  • the camera internal parameter calibration method provided by the embodiment of this application can be applied in machine vision scenes such as augmented reality (Augmented Reality, AR) scenes and visual positioning and navigation scenes.
  • machine vision scenes machine vision applications use cameras to obtain images, and then according to The acquired images complete related machine vision tasks.
  • the camera parameters need to be calibrated first, including calibrating the camera's internal parameters.
  • embodiments of the present application provide a camera internal parameter calibration method, which can perform real-time calibration of camera internal parameters, effectively reduce the workload of camera internal parameter calibration, and save the manpower and material resources required for camera internal parameter calibration.
  • an embodiment of the present application provides a system architecture 100.
  • the data collection device 160 is used to collect training data.
  • the training data includes training images; and the training data is stored in the database 130.
  • the training device 120 is based on the training data maintained in the database 130.
  • the target model 101 is obtained through training.
  • the target model 101 here can be a feature detection model, an image retrieval model, etc. that are trained in the training stage.
  • the training data maintained in the database 130 may not all be collected from the data collection device 160, and may also be received from other devices.
  • the training device 120 may not necessarily train the target model 101 based entirely on the training data maintained by the database 130. It may also obtain training data from the cloud or other places for model training.
  • the above description should not be used as a reference to the embodiments of the present application. restrictions.
  • the target model 101 trained according to the training device 120 can be applied to different systems or devices, such as the execution device 110 shown in Figure 1 .
  • the execution device 110 can be a terminal, such as a mobile phone terminal, a tablet computer, or a notebook computer. , AR/VR, vehicle-mounted terminal, etc., it can also be a server or cloud, etc.
  • the execution device 110 is configured with an I/O interface 112 for data interaction with external devices.
  • the user can input data to the I/O interface 112 through the client device 140.
  • the input data is used in the embodiment of the present application. may include the first image, that is to say, the camera internal parameter calibration method of the present application is executed on the execution device 110, and the first image is processed to obtain the calibrated internal parameters of the camera.
  • the database 130 can also input training data into the I/O interface 112 as needed, for example, when using the training image as the first image.
  • the preprocessing module 113 is used to perform preprocessing according to the input data (such as the first image) received by the I/O interface 112.
  • the preprocessing module 113 can be used to perform image restoration processing on the input data or Preprocessing such as image enhancement processing to make the input data meet preset standards.
  • image restoration processing is to correct the image degradation caused by various reasons, so that the reconstructed or estimated image is as close as possible to the ideal image field without degradation.
  • the principle of image restoration processing is: first, the entire process of image degradation must be appropriately estimated, and on this basis, an approximate mathematical model of degradation must be established. Afterwards, the model must be appropriately modified to compensate for the distortion that occurs during the degradation process. , to ensure that the image obtained after restoration is close to the original image and to achieve image optimization.
  • Image enhancement processing is to selectively strengthen and suppress the information in the image to improve the visual effect of the image, or to transform the image into a form more suitable for machine processing to facilitate data extraction or recognition.
  • an image enhancement system can highlight the contours of an image through a high-pass filter, allowing the machine to measure the shape and perimeter of the contours.
  • Contrast broadening, logarithmic transformation, density layering and histogram equalization can be used to change the gray tone of the image and highlight the details.
  • the execution device 110 When the execution device 110 preprocesses input data, or when the calculation module 111 of the execution device 110 performs calculations and other related processes, the execution device 110 can call data, codes, etc. in the data storage system 150 for corresponding processing. , the data, instructions, etc. obtained by corresponding processing can also be stored in the data storage system 150 .
  • the data storage system 150 may specifically be the historical internal reference calibration database in the embodiment of the present application.
  • the I/O interface 112 returns the processing results, such as the calibrated internal parameters of the camera obtained above, to the client device 140, thereby providing them to the user.
  • the user can manually set the input data, and the manual setting can be operated through the interface provided by the I/O interface 112 .
  • the client device 140 can automatically send input data to the I/O interface 112. If requiring the client device 140 to automatically send input data requires the user's authorization, the user can set corresponding permissions in the client device 140.
  • the user can view the results output by the execution device 110 on the client device 140, and the specific presentation form may be display, sound, action, etc.
  • the client device 140 can also serve as a data collection end, collecting the input data input to the I/O interface 112 as shown in FIG. 1 and the output results of the I/O interface 112 as new sample data, and storing them in the database 130 .
  • the I/O interface 112 directly uses the input data input to the I/O interface 112 as shown in Figure 1 and the output results of the I/O interface 112 as a new sample.
  • the data is stored in database 130.
  • Figure 1 is only a schematic diagram of a system architecture provided by an embodiment of the present application.
  • the positional relationship between the devices, devices, modules, etc. shown in the figure does not constitute any limitation.
  • the data storage The system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110.
  • the target model 101 is obtained by training according to the training device 120.
  • the target model 101 can be a feature detection model, an image retrieval model, etc.
  • the feature detection model is used to extract feature points of the image and compare the features. Point to perform feature description to obtain the feature descriptor.
  • Feature descriptors include but are not limited to Scale-Invariant Feature Transform (SIFT) feature descriptor, Speeded Up Robust Features (SURF) descriptor, Histogram of Oriented Gradient (HOG) ) feature descriptor, local binary pattern (Local Binary Pattern, LBP) feature descriptor and Haar feature descriptor.
  • SIFT Scale-Invariant Feature Transform
  • SURF Speeded Up Robust Features
  • HOG Histogram of Oriented Gradient
  • LBP Local Binary Pattern
  • Haar feature descriptor Haar feature descriptor.
  • Image retrieval models are used to match images that are similar to a target image.
  • a convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture.
  • the deep learning architecture refers to the algorithm of machine learning. Multiple levels of learning at different levels of abstraction.
  • CNN is a feed-forward artificial neural network. Each neuron in the feed-forward artificial neural network can respond to the image input into it.
  • FIG. 2 is a schematic structural diagram of a convolutional neural network provided by an embodiment of the present application.
  • the convolutional neural network (CNN) 200 may include an input layer 210, a convolutional/pooling layer 220 (where the pooling layer is optional), and a fully connected layer 230.
  • the image to be processed is input into the convolutional neural network 200 through the input layer 210 for processing. in:
  • the convolution layer/pooling layer 220 may include layers 221-226 as shown in the example in Figure 2.
  • layer 221 is a convolution layer
  • layer 222 is a pooling layer.
  • Layer 223 is a convolution layer
  • layer 224 is a pooling layer
  • layer 225 is a convolution layer
  • layer 226 is a pooling layer.
  • 221 and 222 are convolution layers
  • 223 is a pooling layer
  • 224 and 225 are convolution layers
  • 226 is a pooling layer. That is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or can be used as the input of another convolutional layer to continue the convolution operation.
  • convolutional layer 221 As an example to introduce the internal working principle of a convolutional layer.
  • the convolution layer 221 can include many convolution operators.
  • the convolution operator is also called a kernel. Its role in image processing is equivalent to a filter that extracts specific information from the input image matrix.
  • the convolution operator is essentially It can be a weight matrix. This weight matrix is usually predefined. During the convolution operation on the image, the weight matrix is usually one pixel after one pixel (or two pixels after two pixels) along the horizontal direction on the input image. ...This depends on the value of the step size (Stride), so as to complete the work of extracting specific features from the image.
  • the size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image.
  • the weight matrix will extend to Enter the entire depth of the image. Therefore, convolution with a single weight matrix will produce a convolved output with a single depth dimension, but in most cases, instead of using a single weight matrix, multiple weight matrices of the same size (rows ⁇ columns) are applied, That is, multiple matrices of the same type.
  • the output of each weight matrix is stacked to form the depth dimension of the convolution image.
  • the dimension here can be understood as being determined by the "multiple" mentioned above.
  • Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract edge information of the image, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to remove unnecessary noise in the image.
  • the multiple weight matrices have the same size (row ⁇ column), and the feature maps extracted by the multiple weight matrices with the same size are also the same size. The extracted multiple feature maps with the same size are then merged to form a convolution operation. output.
  • weight values in these weight matrices require a large amount of training in practical applications.
  • Each weight matrix formed by the weight values obtained through training can be used to extract information from the input image, thereby allowing the convolutional neural network 200 to make correct predictions. .
  • the initial convolutional layer for example, 221
  • the features extracted by subsequent convolutional layers for example, 226) become more and more complex, such as high-level semantic features.
  • each layer 221-226 as shown at 220 in Figure 2 there can be a layer of convolutional layer followed by a layer of
  • the pooling layer can also be a multi-layer convolution layer followed by one or more pooling layers.
  • the only purpose of the pooling layer is to reduce the spatial size of the image.
  • the pooling layer may include an average pooling operator and/or a maximum pooling operator for sampling the input image to obtain a smaller size image.
  • the average pooling operator can calculate the pixel values in the image within a specific range to generate an average value as the result of average pooling.
  • the max pooling operator can take the pixel with the largest value in a specific range as the result of max pooling.
  • the size of the weight matrix used in the convolutional layer should be related to the image size
  • the operation in the pooling layer Symbols should also be related to the size of the image.
  • the size of the image output after processing by the pooling layer can be smaller than the size of the image input to the pooling layer.
  • Each pixel in the image output by the pooling layer represents the average or maximum value of the corresponding sub-region of the image input to the pooling layer.
  • the convolutional neural network 200 After being processed by the convolutional layer/pooling layer 220, the convolutional neural network 200 is not enough to output the required output information. Because as mentioned above, the convolutional layer/pooling layer 220 will only extract features and reduce the parameters brought by the input image. However, in order to generate the final output information (required class information or other related information), the convolutional neural network 200 needs to use the fully connected layer 230 to generate the output of one or a set of required number of classes. Therefore, the fully connected layer 230 may include multiple hidden layers (231, 232 to 23n as shown in Figure 2) and an output layer 240. The parameters contained in the multiple hidden layers may be based on the specific task type. Related training data are pre-trained. For example, the task type can include image feature detection, image retrieval, etc.
  • the output layer 240 has a loss function similar to categorical cross entropy and is specifically used to calculate the prediction error.
  • the convolutional neural network 200 shown in Figure 2 is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models.
  • the camera internal parameter calibration method is explained in detail below.
  • the electronic device may be a smartphone, a server, a smart TV, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an Ultra-Mobile Personal Computer (UMPC), Netbooks, as well as cellular phones, Personal Digital Assistants (PDAs), Augmented Reality (AR) devices, Virtual Reality (VR) devices, Artificial Intelligence (AI) devices, wearables devices, vehicle-mounted devices, smart home devices and/or smart city devices, etc.
  • PDAs Personal Digital Assistants
  • AR Augmented Reality
  • VR Virtual Reality
  • AI Artificial Intelligence
  • wearables devices wearables devices
  • vehicle-mounted devices smart home devices and/or smart city devices, etc.
  • smart home devices smart home devices and/or smart city devices
  • Figure 3 is a schematic flowchart of a camera internal parameter calibration method provided by an embodiment of the present application
  • Figure 4 is a schematic diagram of a camera internal parameter calibration method provided by an embodiment of the present application.
  • the camera intrinsic parameter calibration method 300 includes step 301, step 302 and step 303.
  • the first image is an image captured by the target terminal using its own camera at a first location, the first location is within a preset area, and the scene captured by the first image is within the preset area.
  • the first image may be one or multiple images.
  • the specific range of the above preset area can be set according to actual needs and is not specifically limited.
  • the default area is park A, square B or shopping mall C, etc.
  • the identification of the target terminal may be the unique label information of the target terminal or the model information of the target terminal or other information that can identify the terminal.
  • the unique label of the target terminal may be composed of at least one of numbers, letters, special characters, etc., for example, the target terminal
  • the terminal is uniquely marked BJ22-2598.
  • the model number of the target terminal may be composed of at least one of numbers, letters, special characters, etc., for example, the model number of the target terminal is XH1-123.
  • the target terminal can be the above-mentioned electronic device itself.
  • the user can use the target terminal to obtain the first image in the preset area.
  • the target terminal and the electronic device are two different devices.
  • the target terminal obtains the first image and its own identification, sends the first image and identification to the electronic device, and the electronic device receives the first image from the target terminal and The identification of the target terminal, and the processing of the method steps shown in Figure 3 is performed according to the first image and the identification of the target terminal.
  • the high-precision three-dimensional map is the three-dimensional map corresponding to the above-mentioned preset area.
  • the historical internal parameter calibration database includes historical internal parameter calibration data of one or more identified terminals.
  • the historical internal parameter calibration database can be set in the electronic device.
  • the electronic device is a server in the cloud
  • the server executes the method shown in Figure 3 and is set with a historical internal parameter calibration database.
  • the historical internal parameter calibration database can be set on other devices, and the electronic device establishes a connection with the historical internal parameter calibration database by accessing other devices.
  • the first internal parameter calibration data is data related to the calibration parameters of the target terminal.
  • the high-precision three-dimensional map is an electronic map with higher precision.
  • the high-precision three-dimensional map corresponding to the preset area includes multiple scene images in the preset area, the two-dimensional feature point coordinates of the above scene images, the feature descriptors of the two-dimensional feature points, and the three-dimensional space point coordinates corresponding to the two-dimensional feature points.
  • multiple pictures refer to more than or equal to two pictures
  • the two-dimensional feature point coordinates refer to the coordinates of the two-dimensional feature points of the scene image in the scene image, that is, the coordinates of the two-dimensional feature point coordinates in the pixel coordinate system
  • the three-dimensional space Point coordinates refer to the coordinates of the two-dimensional feature points of the scene image in the world coordinate system.
  • the accuracy of the two-dimensional feature point coordinates of the high-precision three-dimensional map is within 5 pixels.
  • the coordinate accuracy of high-precision three-dimensional maps is about 10-50 centimeters.
  • the accuracy of the three-dimensional space point coordinates of the high-precision three-dimensional map is 10 centimeters; when the preset area is an outdoor scene, the accuracy of the three-dimensional space point coordinates of the high-precision three-dimensional map is 20 centimeters.
  • the above-mentioned scene image is an image in a preset direction (such as up, down, left, right, etc.) acquired at several image acquisition positions in the preset area.
  • Feature points can be extracted using the scene image to determine the coordinates of the two-dimensional feature points and the feature descriptors corresponding to the two-dimensional feature points.
  • the coordinates of the three-dimensional space points corresponding to the two-dimensional feature points can be obtained, and a high-precision three-dimensional map can be established.
  • the above two-dimensional feature point coordinates and the corresponding three-dimensional space point coordinates can be understood as the coordinate correspondence between the two-dimensional feature points in the pixel coordinate system and the three-dimensional space points in the world coordinate system.
  • the preset area is Park A
  • the left field of view image of the image acquisition position can be acquired as the scene image.
  • the view image of the right, upper, front, etc. directions of the image acquisition position can also be acquired as the scene. image.
  • a high-precision three-dimensional map corresponding to the preset area can be established.
  • the embodiment of the present application uses a high-precision three-dimensional map and a historical internal parameter calibration database to first obtain the first internal parameter calibration data and the historical internal parameter calibration data of the target terminal, and then performs the process based on the first internal parameter calibration data and the historical internal parameter calibration data of the target terminal.
  • Internal parameter calibration determines the calibrated internal parameters of the camera.
  • the calibration method of the embodiment of the present application does not require the use of additional calibration equipment (such as a calibration board) for offline calibration, nor does it require the user to take specific actions to complete the internal parameter calibration.
  • the user only needs to use the camera of the target terminal to obtain the first image in the preset area to automatically complete the calibration. Therefore, the internal parameters of the camera of the target terminal can be calibrated in real time, effectively improving the calibration efficiency.
  • Manual offline calibration is required, which can effectively reduce the workload required for camera internal parameter calibration.
  • the machine vision application in the target terminal can perform other related processing according to the calibrated internal parameters to normally complete related machine vision tasks. Therefore, the calibration method of the embodiment of the present application will not block the process of machine vision applications. Even an uncalibrated target terminal can use the machine vision normally like a calibrated target terminal. Visual applications reduce user operations, bring greater convenience to users, and increase the popularity of machine vision applications in different target terminals.
  • the above-mentioned camera internal parameter calibration method also includes:
  • the electronic device updates the local internal parameters of the target terminal according to the calibrated internal parameters, and saves the first internal parameter calibration data to the historical internal parameter calibration database.
  • the electronic device updates the local internal parameters of the target terminal according to the preset internal parameters.
  • the internal parameter validity conditions are the standards that the internal parameters need to meet, and the specific standards can be set according to the actual situation.
  • the specific values of the preset internal parameters can be set according to actual needs and are not particularly limited.
  • the electronic device at this time is the above-mentioned target terminal, that is, the method steps of obtaining the calibration internal parameters are executed on the target terminal, and then the validity of the calibration internal parameters is judged.
  • the target terminal updates its local internal parameters according to the calibration internal parameters, saves the first internal parameter calibration data to the historical internal parameter calibration database, and updates the historical internal parameter calibration database.
  • the target terminal updates its local internal parameters according to the preset internal parameters.
  • the above camera internal parameter calibration method also includes:
  • the electronic device saves the first internal parameter calibration data into the historical internal parameter calibration database, and sends the calibrated internal parameters to the target terminal, so that the target terminal updates the local internal parameters according to the calibrated internal parameters.
  • the electronic device sends the preset internal parameters to the target terminal, so that the target terminal updates the local internal parameters of the target terminal.
  • the electronic device and the target terminal are different devices.
  • the electronic device sends the calibrated internal parameters to the target terminal, so that the target terminal updates its local internal parameters, that is, The local internal parameters are updated to the above-mentioned calibration internal parameters, and the first internal parameter calibration data is saved in the historical internal parameter calibration database.
  • the electronic device sends the preset internal parameters to the target terminal, so that the target terminal updates the local internal parameters to the received preset internal parameters.
  • Figure 5a is a schematic flowchart of determining the first internal parameter calibration data provided by an embodiment of the present application; the above-mentioned method determines the first internal parameter calibration data of the camera based on the first image and the high-precision three-dimensional map corresponding to the preset area, Including step 511, step 512, step 513 and step 514. in:
  • N Determine N scene images with the highest image similarity to the first image from the high-precision three-dimensional map, where N is an integer greater than zero.
  • the specific value of N can be set according to the actual situation and is not particularly limited.
  • the image retrieval model can be used to determine N scene images with the highest image similarity to the first image from the high-precision three-dimensional map, for example, ten scene images.
  • the coordinate pair includes two-dimensional feature point coordinates and three-dimensional space point coordinates, and X is less than or equal to M.
  • image feature points can be extracted from the first image to obtain the first image pair. corresponding M image feature points.
  • the feature detection model can also perform feature description on the image feature points to obtain feature descriptors corresponding to the M image feature points.
  • the specific types of feature descriptors are not particularly limited.
  • X images of the first image can be determined by matching the feature descriptors of the image feature points in a high-precision three-dimensional map Coordinate pairs corresponding to feature points. Specifically, matching is performed based on the feature descriptors of the M image feature points and the feature descriptors corresponding to the two-dimensional feature points of the N scene images. When the feature descriptors of the former and the latter are the same, the two-dimensional feature point coordinates and the three-dimensional space point coordinates corresponding to the matched feature descriptors are used as the coordinate pair of the image feature points.
  • the extrinsic parameters (known) of the camera are used to convert the coordinates from the world coordinate system to the camera coordinate system, and the intrinsic parameters of the camera are used to convert the coordinates from the camera coordinate system to the pixel coordinate system; and
  • the two-dimensional feature point coordinates are coordinates in the pixel coordinate system, and the three-dimensional space point coordinates are coordinates in the world coordinate system. Therefore, the first internal parameter calibration data can be determined by using the correspondence between the two-dimensional feature point coordinates and the three-dimensional space point coordinates. .
  • the coordinate pairs of the X image feature points may be directly determined as the first internal parameter calibration data.
  • the first internal parameter calibration data at this time is the coordinate pairs of X image feature points.
  • the first internal parameter of the camera can be determined. For example, the specific calculation process as follows:
  • the pose of the target terminal's camera in the world coordinate system is [R t]
  • R is the rotation matrix
  • t is the translation vector
  • the position of a three-dimensional space point in the world coordinate system is (X i Y i Z i ) T
  • the pixel coordinates (u i v i ) T can be obtained.
  • the corresponding transformation relationship is as follows:
  • the M matrix has a total of 12 unknowns.
  • the M matrix can be solved by at least 6 non-coplanar feature points. Further, based on the relationship between the M matrix and the camera's internal parameter matrix K, the camera's internal parameter matrix K can be calculated, which is the first internal parameter of the camera.
  • the historical internal parameter calibration data corresponding to the target terminal is the first internal parameter calibration data saved in the database by the target terminal in the past calibration process.
  • the historical internal parameter calibration data corresponding to the target terminal is the first internal parameter calibration data saved in the database during past calibration processes for terminals of the same model including the target terminal. Therefore, by combining the coordinate pairs of the X image feature points and the historical internal parameter calibration data of the target terminal, the second internal parameter of the camera can be determined.
  • the calculation method of the second internal parameter is similar to the calculation method of the first internal parameter.
  • the difference is that during the calculation process of the second internal parameter, it is necessary to calibrate the coordinate pairs of X image feature points and the historical internal parameter calibration data of the target terminal respectively. Give different weights and perform calculations based on the results of weighted operations.
  • the above weight may be the reciprocal of the depth distance of a three-dimensional space point or the response value of a two-dimensional feature point.
  • the weight of the i-th coordinate pair is set to w i , and the w i matrix is 2 ⁇ 2 dimension.
  • the unknown number X matrix can be calculated, which is the 12 unknown numbers of the M matrix, thus obtaining the M matrix.
  • the first internal parameter and the second internal parameter are determined respectively, and then the one with the smallest reprojection error between the two is determined as the calibration internal parameter of the camera.
  • the reprojection error comparison can be used to effectively improve the calibration. Accuracy of internal parameters.
  • the reprojection error of the first internal parameter is based on the coordinate pair set of the first internal parameter pair (the coordinate pair set here refers to the coordinate pair of X image feature points and the historical internal parameter calibration data of the target terminal).
  • the three-dimensional space points in the set) are reprojected to determine the coordinates of the two-dimensional feature points after reprojection.
  • the coordinates of the two-dimensional feature points in the set can be determined based on the two-dimensional feature point coordinates and coordinate pairs obtained after reprojection.
  • Reprojection error of the first intrinsic parameter for each coordinate pair, a reprojected two-dimensional feature point coordinate can be obtained, and accordingly a sub-reprojection error can be obtained.
  • the average value is calculated according to the multiple sub-reprojection errors corresponding to the coordinate pair set, and the average value is Reprojection error of the first intrinsic parameter. Or, calculate the root mean square error based on the coordinates of multiple two-dimensional feature points before reprojection and the two-dimensional feature point coordinates after reprojection corresponding to the coordinate pair set, and use the root mean square error as the reprojection error of the first internal parameter . In the same way, the reprojection error of the second internal parameter can be determined using the calculation method of the reprojection error of the first internal parameter.
  • the corresponding weights of the coordinate pairs of X image feature points can also be stored in the history.
  • the weight can be the reciprocal of the depth distance of the three-dimensional space point in the coordinate pair or the response value of the two-dimensional feature point. In this way, when calculating the second internal parameter for the next calibration, the weights can be directly taken from the historical internal parameter calibration database for weighted calculation.
  • Step 1 Turn on the terminal device, and the user enters the augmented reality application; where the augmented reality application contains the algorithm corresponding to the calibration method in the embodiment of the present application.
  • Step 2 The target terminal obtains data such as the terminal model and an image captured by the terminal's camera at the current location, and the target terminal uploads data such as the terminal model and the captured image to the cloud server.
  • Step 3 Complete image retrieval from the high-precision three-dimensional map based on the current location image to obtain the scene image, and obtain the coordinate pair through feature point matching between the current location image and the scene image.
  • Step 4 retrieve the historical valid coordinate pairs corresponding to the model from the historical internal parameter calibration database according to the terminal model.
  • Step 5 Calculate the internal parameter K 1 of the camera in real time based on the coordinate pair matched in step 3.
  • Step 6 Combine the coordinate pairs obtained by matching in step 3 and the effective coordinate pair data of the same model retrieved from the historical internal parameter calibration database in step 4 to jointly calculate the internal parameters of the camera. During calculation, the respective weights can be weighted according to the coordinates to obtain the internal parameters of the camera.
  • Step 7 Compare the internal camera parameters K 1 and Calculate the reprojection error of the three-dimensional space points respectively, and use the camera internal parameters with smaller reprojection errors as the internal parameter calibration results to output.
  • Step 8 Judge the validity of the internal parameter calibration results obtained in Step 7. If the internal parameter calibration result satisfies the validity judgment condition, the calibration result is considered valid, and the successful internal parameter calibration flag and valid camera internal parameters are returned. At the same time, the coordinate pair data obtained from the matching and the corresponding weight ( The weight can be the reciprocal of the depth distance of the three-dimensional space point in the coordinate pair or the response value of the two-dimensional feature point) and is saved as valid calibration data in the historical internal parameter calibration database. If any validity judgment condition is not met, the current internal parameter calibration result is considered invalid, and the internal parameter calibration failure flag and the default camera internal parameters are returned.
  • the weight can be the reciprocal of the depth distance of the three-dimensional space point in the coordinate pair or the response value of the two-dimensional feature point
  • Step 9 The terminal receives the flag bit and calibration result returned by the cloud server, and the terminal uses the returned camera internal parameters. for augmented reality applications. If it is a flag indicating that the internal parameter calibration is successful, the calculated calibration result will be used as the internal camera parameter during the use of this augmented reality application, and no calibration will be performed during the use of this application. If the internal parameter calibration fails, the default internal camera parameters will be used for the augmented reality application, which will not block the application process, and return to step 2 to request calibration again until the calibration is successful.
  • the first internal parameter of the camera when determining the first internal parameter calibration data based on the coordinate pairs of X image feature points, can be determined first based on the coordinate pairs of Calibration data for the first internal parameter.
  • the first internal parameter of the camera For determining the first internal parameter of the camera based on the coordinate pairs of the X image feature points, reference can be made to the above description, which will not be described again.
  • Figure 6a is a schematic flowchart of another method of determining calibration internal parameters provided by an embodiment of the present application; determining the calibration internal parameters of the camera based on the first internal parameter calibration data and the historical internal parameter calibration data of the target terminal, including step 601 and steps 602. in:
  • the form of the historical internal parameter calibration data is the internal parameters of the camera.
  • the historical internal parameter calibration data corresponding to the target terminal is the first internal parameter calibration data saved in the database by the target terminal in the past calibration process, that is, the first internal parameter.
  • the historical internal parameter calibration data corresponding to the target terminal is the first internal parameter calibration data saved in the database for terminals of the same model including the target terminal in the past calibration process, that is, the first internal parameter calibration data.
  • the historical internal parameter calibration data corresponding to the target terminal, that is, the number of first internal parameters saved in history is more than one.
  • the first internal parameter and the historical internal parameter calibration data are combined to determine the second internal parameter of the camera using weighted calculation, and then one of the first internal parameter and the second internal parameter is determined as the calibration internal parameter based on the reprojection error.
  • the relevant description of step 523 please refer to the relevant description of step 523 and will not be described again.
  • the first weight corresponding to the first internal parameter can also be used to calculate the first internal parameter.
  • the coordinate pairs of The coordinate pairs of X image feature points are used for reprojection error calculation in the next calibration.
  • the first weight is the reciprocal of the reprojection error corresponding to the first internal parameter
  • the second weight is the reciprocal of the reprojection error corresponding to the historical internal parameter calibration data.
  • Figure 6b is a schematic flowchart of another camera internal parameter calibration method provided by an embodiment of the present application.
  • the machine vision application takes the augmented reality application as an example
  • the identification of the target terminal takes the model as an example
  • the electronic device is a cloud server as an example, and the calibration method of the embodiment of the present application is described in detail.
  • Step 1 Open the terminal device and enter the augmented reality application; where the augmented reality application includes the algorithm corresponding to the calibration method in the embodiment of the present application.
  • Step 2 The target terminal obtains data such as the terminal model and an image captured by the terminal's camera at the current location, and the target terminal uploads data such as the terminal model and the captured image to the cloud server.
  • Step 3 Complete image retrieval from the high-precision three-dimensional map based on the image of the current location to obtain the scene image, and obtain the coordinate pair through feature point matching between the current location image and the scene image.
  • Step 4 According to the terminal model, retrieve multiple sets of historical effective camera internal parameters corresponding to the model from the historical internal parameter calibration database.
  • Step 5 Calculate the internal parameter K 1 of the camera based on the coordinate pair matched in step 3.
  • Step 6 Combining the camera internal parameter K 1 calculated based on the coordinate pair obtained in step 5 and the multiple sets of historical effective camera internal parameters of the same model retrieved from the historical calibration database in step 4, jointly calculate the camera internal parameters During calculation, the respective weights can be weighted according to the coordinates to obtain the internal parameters of the camera.
  • Step 7 Compare the internal camera parameters K 1 and Calculate the reprojection error size of the two three-dimensional space points, and use the internal camera parameters with smaller reprojection errors as the calibration result output.
  • Step 8 Judge the validity of the internal parameter calibration results obtained in Step 7. If the internal parameter calibration result meets the validity judgment condition, the calibration result is considered valid, and the flag bit of successful internal parameter calibration and the valid camera internal parameter are returned. At the same time, the camera internal parameter K 1 obtained in step 5 and the corresponding The weights are saved to the historical internal parameter calibration database. If any validity judgment condition is not met, the current internal parameter calibration result is considered invalid, and the internal parameter calibration failure flag and a default camera internal parameter are returned.
  • Step 9 The terminal receives the flag bits and calibration results returned by the cloud calibration algorithm, and the terminal uses the returned camera parameters for augmented reality applications. If it is a flag indicating that the internal parameter calibration is successful, the calculated calibration result will be used as the internal camera parameter during the use of this augmented reality application, and no calibration will be performed during the use of this application. If the internal parameter calibration fails, the default internal camera parameters will be used for the augmented reality application, which will not block the application process, and return to step 2 to request calibration again until the calibration is successful.
  • the intrinsic parameters of the camera include the X-axis focal length fx, the Y-axis focal length fy, the image principal point coordinates cx on the X axis, and the image principal point coordinates cy on the Y axis.
  • the intrinsic parameter validity condition Can be:
  • the result of the camera's calibration internal parameters needs to be greater than 0, that is, fx, fy, cx and cy are all greater than 0;
  • a is 50;
  • b The difference between cx and half of the image width does not exceed b pixels.
  • the specific value of b can be set according to the actual situation and is not specifically limited. For example, b is 15;
  • c is 15;
  • the reprojection error of the set of coordinates calculated based on the internal parameters of the camera does not exceed d pixels.
  • the specific value of d can be set according to the actual situation and is not particularly limited.
  • d is 15.
  • the three-dimensional space point coordinates in the set are re-projected according to the internal parameters and coordinates of the camera, and the re-projected two-dimensional feature point coordinates can be obtained.
  • the two-dimensional feature points in the set are re-projected according to the re-projected two-dimensional feature point coordinates and coordinates.
  • Feature point coordinates can determine the reprojection error.
  • the above coordinate pair set may be a set consisting of the coordinate pairs of the X image feature points and historical internal parameter calibration data of the target terminal.
  • the internal parameters of the camera satisfy all the above internal parameter validity conditions, the internal parameters of the camera can be confirmed to be valid internal parameters. Otherwise, confirm that the intrinsic parameters of the camera are invalid intrinsic parameters.
  • FIG. 7 is a schematic structural diagram of a camera internal parameter calibration device provided by an embodiment of the present application.
  • the camera intrinsic parameter calibration device 700 includes an acquisition module 701 , a determination module 702 and a calibration module 703 .
  • the acquisition module 701 is used to acquire the first image and the identification of the target terminal.
  • the first image is an image captured by the target terminal using its own camera at a first position, and the first position is located within a preset area.
  • the determination module 702 is configured to determine the first internal parameter calibration data of the camera based on the first image and the high-precision three-dimensional map corresponding to the preset area; and determine the historical internal parameter calibration data of the target terminal based on the identification of the target terminal and the historical internal parameter calibration database.
  • the historical internal parameter calibration database includes historical internal parameter calibration data of one or more identified terminals.
  • the calibration module 703 is configured to determine the calibration internal parameters of the camera based on the first internal parameter calibration data and the historical internal parameter calibration data of the target terminal.
  • the acquisition module 701 is specifically used to:
  • a first image and an identification of the target terminal are received from the target terminal.
  • the high-precision three-dimensional map includes multiple scene images, two-dimensional feature point coordinates of the scene images, feature descriptors of the two-dimensional feature points, and three-dimensional space point coordinates corresponding to the two-dimensional feature points.
  • the above-mentioned determination module 702 is specifically used for:
  • N N is an integer greater than zero.
  • M M is an integer greater than zero.
  • the coordinate pair includes two-dimensional feature point coordinates and three-dimensional space point coordinates, and X is less than or equal to M.
  • the first internal parameter calibration data is determined based on the coordinate pairs of the X image feature points.
  • the determination module 702 in determining the first internal parameter calibration data based on the coordinate pairs of the X image feature points, is specifically configured to: determine the coordinate pairs of the X image feature points as the first internal parameter calibration data.
  • the above-mentioned calibration module 703 is specifically used for:
  • the first internal parameters of the camera are determined based on the coordinate pairs of X image feature points.
  • the second internal parameters of the camera are determined based on the coordinate pairs of the X image feature points and the historical internal parameter calibration data of the target terminal. The one with the smallest reprojection error among the first internal parameter and the second internal parameter is determined as the calibration internal parameter.
  • the determination module 702 in determining the first internal parameter calibration data based on the coordinate pairs of the X image feature points, is specifically configured to determine the first internal parameter of the camera based on the coordinate pairs of the X image feature points.
  • the first internal parameter is determined as the first internal parameter calibration data.
  • the above-mentioned calibration module 703 is specifically used for:
  • a weighted calculation is performed based on the first internal parameter and its first weight, each historical internal parameter calibration data corresponding to the target terminal and its second weight to determine the second internal parameter of the camera. The one with the smallest reprojection error among the first internal parameter and the second internal parameter is determined as the calibration internal parameter.
  • the first weight is the reciprocal of the reprojection error corresponding to the first internal parameter; the second weight is the reciprocal of the reprojection error corresponding to the historical internal parameter calibration data.
  • the camera internal parameter calibration device 700 further includes:
  • the processing module is used to update the local internal parameters of the target terminal according to the calibrated internal parameters when the calibrated internal parameters meet the internal parameter validity conditions, and save the first internal parameter calibration data to the historical internal parameter calibration database. And, when the calibrated internal parameters do not meet the internal parameter validity conditions, the local internal parameters of the target terminal are updated according to the preset internal parameters.
  • the camera internal parameter calibration device 700 also includes:
  • the processing module is used to save the first internal parameter calibration data to the historical internal parameter calibration database when the calibration internal parameters meet the internal parameter validity conditions, and send the calibration internal parameters to the target terminal, so that the target terminal updates the target according to the calibration internal parameters. Local intrinsic parameters of the terminal. And, when the calibrated internal parameters do not meet the internal parameter validity conditions, the preset internal parameters are sent to the target terminal, so that the target terminal updates the local internal parameters of the target terminal.
  • FIG. 8 is a schematic structural diagram of a camera internal parameter calibration device provided by an embodiment of the present application.
  • the camera internal parameter calibration device 800 includes a memory 801, a processor 802, a communication interface 804 and a bus 803. Among them, the memory 801, the processor 802, and the communication interface 804 implement communication connections between each other through the bus 803.
  • the memory 801 may be a read-only memory (Read Only Memory, ROM), a static storage device, a dynamic storage device or a random access memory (Random Access Memory, RAM).
  • the memory 801 can store programs. When the program stored in the memory 801 is executed by the processor 802, the processor 802 is used to execute each step of the camera intrinsic parameter calibration method described in any of the above embodiments.
  • the processor 802 can be a general central processing unit (Central Processing Unit, CPU), a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a graphics processor (graphics processing unit, GPU) or one or more
  • the integrated circuit is used to execute relevant programs to implement the functions required to be performed by the units in the camera internal parameter calibration device described in any of the above embodiments, or to execute the camera internal parameter calibration method described in any of the above embodiments.
  • the processor 802 may also be an integrated circuit chip with signal processing capabilities. During the implementation process, each step of the camera internal parameter calibration method described in any embodiment of the present application can be completed by instructions in the form of hardware integrated logic circuits or software in the processor 802 .
  • the above-mentioned processor 802 can also be a general-purpose processor, a digital signal processor (Digital Signal Processing, DSP), an application specific integrated circuit (ASIC), an off-the-shelf programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices , discrete gate or transistor logic devices, discrete hardware components.
  • DSP Digital Signal Processing
  • ASIC application specific integrated circuit
  • FPGA Field Programmable Gate Array
  • a general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
  • the steps of the camera internal parameter calibration method described in any embodiment of the present application can be directly implemented by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other mature storage media in this field.
  • the storage medium is located in the memory 801.
  • the processor 802 reads the information in the memory 801, and combines its hardware to complete the functions required to be performed by the units included in the camera internal parameter calibration device described in any of the above embodiments, or to perform any of the applications.
  • the communication interface 804 uses a transceiver device such as but not limited to a transceiver to implement communication between the camera internal parameter calibration device 800 and other devices or communication networks.
  • a transceiver device such as but not limited to a transceiver to implement communication between the camera internal parameter calibration device 800 and other devices or communication networks.
  • the camera internal parameter calibration device 800 can obtain the first image through the communication interface 804.
  • the bus 803 may include a path for transmitting information between various components of the camera internal parameter calibration device 800 (eg, memory 801, processor 802, communication interface 804).
  • the camera internal parameter calibration device 800 shown in FIG. 8 only shows a memory, a processor, and a communication interface, during the specific implementation process, those skilled in the art will understand that the camera internal parameter calibration device 800 also includes the implementation of normal Other components necessary for operation. At the same time, based on specific needs, those skilled in the art should understand that the camera internal parameter calibration device 800 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the camera internal parameter calibration device 800 may only include components necessary to implement the embodiments of the present application, and does not necessarily include all components shown in FIG. 8 .
  • the disclosed systems, devices and methods can be implemented in other ways.
  • the device embodiments described above are only illustrative, such as the division of the units, It is only a logical functional division. In actual implementation, there may be other divisions. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.
  • the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application can be integrated into one processing unit, each unit can exist physically alone, or two or more units can be integrated into one unit.
  • the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.
  • the technical solution of the present application is essentially or contributes to the existing technology or part of the technical solution can be embodied in the form of a computer program product, and the computer program product is stored in a storage medium, It includes several instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program code. .
  • An embodiment of the present application also provides a chip.
  • the chip includes a processor and a data interface.
  • the processor reads instructions stored in the memory through the data interface and executes the camera internal parameter calibration method described in any embodiment.
  • the chip may further include a memory, in which instructions are stored, and the processor is configured to execute the instructions stored in the memory.
  • the processor is configured to execute the camera intrinsic parameter calibration method described in any embodiment.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)
  • Studio Devices (AREA)

Abstract

一种相机内参标定方法及相关设备，涉及图像处理领域。方法（300）先获取第一图像和目标终端的标识（301）；基于目标终端获取的第一图像和高精度三维地图确定相机的第一内参标定数据，以及根据目标终端的标识和历史内参标定数据库确定目标终端的历史内参标定数据（302）。继而可以根据第一内参标定数据和历史内参标定数据确定相机的标定内参数（303）。方法可以实时对目标终端的相机内参进行标定，提升标定效率，进而有效降低相机内参标定所需的工作量。

Description

相机内参标定方法及相关设备
本申请要求于2022年06月13日提交中国专利局、申请号为202210660950.1、申请名称为“相机内参标定方法及相关设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及图像处理领域，尤其涉及一种相机内参标定方法及相关设备。
背景技术
相机内参数是与相机自身特性相关的参数，比如相机的焦距、像素大小等。相机内参数标定技术是摄影测量和计算机视觉领域的一个重点研究内容之一，它是视觉定位、机器人视觉导航建图、工业控制等领域技术的基础。
随着终端的处理能力的不断增强，基于增强现实(Augmented Reality，AR)技术的应用程序逐渐增多，AR应用中的算法通常需要有先验的相机内参数作为输入。然而由于不同的手机机型使用的相机传感器不同，对应的相机内参数也不同，为了适配不同的手机机型，需要逐个机型进行标定获得对应的相机内参数。目前，可以利用标定板完成终端相机的内参标定，这种方法需要事先通过离线标定的方式获取每个机型内参数并储存，然后才能在AR应用的算法中读取和调用相机的内参，但是，这种标定方法需要耗费大量人力物力，而且不能获取实时的相机内参数用于AR应用后续的计算。
如何解决上述问题，是本领域技术人员正在研究的热点。
发明内容
本申请提供一种相机内参标定方法及相关设备，可以实时对相机内参进行标定，有效降低相机内参标定所需的工作量。
第一方面，本申请提供一种相机内参标定方法，该方法可以应用于电子设备上。上述电子设备例如为服务器或智能终端等。电子设备获取第一图像和目标终端的标识。该第一图像为目标终端在第一位置处利用自身的相机拍摄得到的图像，第一位置位于预设区域内。电子设备再根据第一图像和预设区域对应的高精度三维地图，确定相机的第一内参标定数据；以及，根据目标终端的标识和历史内参标定数据库确定目标终端的历史内参标定数据。历史内参标定数据库包括一个或多个标识的终端的历史内参标定数据。电子设备最后根据第一内参标定数据和目标终端的历史内参标定数据确定相机的标定内参数。
本方案先基于目标终端获取的第一图像和高精度三维地图确定相机的第一内参标定数据，以及根据目标终端的标识和历史内参标定数据库确定该目标终端的历史内参标定数据。继而可以根据第一内参标定数据和历史内参标定数据确定相机的标定内参数。该方法可以实时对目标终端的相机内参进行标定，提升标定效率，进而有效降低相机内参标定所需的工作量。
在一种可能的实施方式中，高精度三维地图包括多张场景图像、场景图像的二维特征点坐标、二维特征点的特征描述子以及二维特征点对应的三维空间点坐标。其中，多张是指大于或等于两张。
上述根据第一图像和预设区域对应的高精度三维地图，确定相机的第一内参标定数据，包括：
从高精度三维地图中确定与第一图像的图像相似度最高的N张场景图像，N为大于零的整数。确定第一图像的M个图像特征点，M为大于零的整数。根据M个图像特征点的特征描述子和N张场景图像的二维特征点对应的特征描述子进行匹配，确定M个图像特征点中的X个图像特征点的坐标对。坐标对包括二维特征点坐标和三维空间点坐标，X小于或等于M。根据X个图像特征点的坐标对确定第一内参标定数据。
本方案中，由于第一图像是在预设区域中的第一位置拍摄得到的图像，因此，利用图像特征点的特征描述子在高精度三维地图中进行匹配，以确定第一图像的X个图像特征点对应的坐标对，进而可以根据X个图像特征点的坐标对确定第一内参标定数据。
在一种可能的实施方式中，根据X个图像特征点的坐标对确定第一内参标定数据，包括：
将X个图像特征点的坐标对确定为第一内参标定数据。
进一步地，根据第一内参标定数据和目标终端的历史内参标定数据确定相机的标定内参数，包括：
根据X个图像特征点的坐标对确定相机的第一内参数。根据X个图像特征点的坐标对、目标终端的历史内参标定数据确定相机的第二内参数。将第一内参数、第二内参数中重投影误差最小的一个确定为标定内参数。
本方案中，分别确定第一内参数和第二内参数，再将前后两者中重投影误差最小的一个确定为相机的标定内参数，利用重投影误差比较可以有效提升标定内参数的精度。
在一种可能的实施方式中，根据X个图像特征点的坐标对确定第一内参标定数据，包括：
根据X个图像特征点的坐标对确定相机的第一内参数。将第一内参数确定为第一内参标定数据。
进一步地，根据第一内参标定数据和目标终端的历史内参标定数据确定相机的标定内参数，包括：
根据第一内参数及其第一权重、目标终端对应的每个历史内参标定数据及其第二权重进行加权计算，以确定相机的第二内参数。将第一内参数、第二内参数中重投影误差最小的一个确定为标定内参数。
本方案中，联合第一内参数和历史内参标定数据，利用加权计算的方式确定相机的第二内参数，再根据重投影误差将第一内参数和第二内参数中的一个确定为标定内参数。
在一种可能的实施方式中，第一权重为第一内参数对应的重投影误差的倒数；第二权重为历史内参标定数据对应的重投影误差的倒数。
在一种可能的实施方式中，上述相机内参标定方法还包括：
标定内参数满足内参数有效性条件时，电子设备根据标定内参数更新目标终端的本地内参数，并将第一内参标定数据保存至历史内参标定数据库中。而标定内参数不满足内参数有效性条件时，电子设备根据预设内参数更新目标终端的本地内参数。
其中，内参数有效性条件是预设的内参数需要满足的标准，具体的标准可以根据实际情况进行设置。本方案中，此时的电子设备为上述目标终端，即在目标终端执行获取标定内参数的方法步骤，再判断内参数的有效性，当标定内参数满足内参数有效性条件时，目标终端根据标定内参数更新其本地内参数，并将第一内参标定数据保存至历史内参标定数据库中。而标定内参数不满足内参数有效性条件时，目标终端根据预设内参数更新其本地内参数。
在一种可能的实施方式中，上述相机内参标定方法还包括：
标定内参数满足内参数有效性条件时，电子设备将第一内参标定数据保存至历史内参标定数据库中，以及将标定内参数发送给目标终端，以使目标终端根据标定内参数更新目标终端的本地内参数。标定内参数不满足内参数有效性条件时，电子设备将预设内参数发送给目标终端，以使目标终端更新目标终端的本地内参数。
在一种可能的实施方式中，上述获取第一图像和目标终端的标识，包括：
接收来自目标终端的第一图像以及目标终端的标识。
第二方面，本申请还提供一种相机内参标定装置，该装置包括获取模块、确定模块和标定模块。
获取模块，用于获取第一图像和目标终端的标识。第一图像为目标终端在第一位置处利用自身的相机拍摄得到的图像，第一位置位于预设区域内。
确定模块，用于根据第一图像和预设区域对应的高精度三维地图，确定相机的第一内参标定数据；以及，根据目标终端的标识和历史内参标定数据库确定目标终端的历史内参标定数据。历史内参标定数据库包括一个或多个标识的终端的历史内参标定数据。
标定模块，用于根据第一内参标定数据和目标终端的历史内参标定数据确定相机的标定内参数。
在一种可能的实施方式中，高精度三维地图包括多张场景图像、场景图像的二维特征点坐标、二维特征点的特征描述子以及二维特征点对应的三维空间点坐标。
上述确定模块，具体用于：
从高精度三维地图中确定与第一图像的图像相似度最高的N张场景图像，N为大于零的整数。确定第一图像的M个图像特征点，M为大于零的整数。根据M个图像特征点的特征描述子和N张场景图像的二维特征点对应的特征描述子进行匹配，确定M个图像特征点中的X个图像特征点的坐标对。坐标对包括二维特征点坐标和三维空间点坐标，X小于或等于M。根据X个图像特征点的坐标对确定第一内参标定数据。
在一种可能的实施方式中，确定模块在根据X个图像特征点的坐标对确定第一内参标定数据方面，具体用于：
将X个图像特征点的坐标对确定为第一内参标定数据。
标定模块，具体用于：
根据X个图像特征点的坐标对确定相机的第一内参数。根据X个图像特征点的坐标对、目标终端的历史内参标定数据确定相机的第二内参数。将第一内参数、第二内参数中重投影误差最小的一个确定为标定内参数。
在一种可能的实施方式中，确定模块在根据X个图像特征点的坐标对确定第一内参标定数据方面，具体用于：
根据X个图像特征点的坐标对确定相机的第一内参数。将第一内参数确定为第一内参标定数据。
上述标定模块，具体用于：
根据第一内参数及其第一权重、目标终端对应的每个历史内参标定数据及其第二权重进行加权计算，以确定相机的第二内参数。将第一内参数、第二内参数中重投影误差最小的一个确定为标定内参数。
在一种可能的实施方式中，第一权重为第一内参数对应的重投影误差的倒数；第二权重为历史内参标定数据对应的重投影误差的倒数。
在一种可能的实施方式中，相机内参标定装置还包括：
处理模块，用于标定内参数满足内参数有效性条件时，根据标定内参数更新目标终端的本地内参数，并将第一内参标定数据保存至历史内参标定数据库中。以及，标定内参数不满足内参数有效性条件时，根据预设内参数更新目标终端的本地内参数。
在一种可能的实施方式中，相机内参标定装置还包括：
处理模块，用于标定内参数满足内参数有效性条件时，将第一内参标定数据保存至历史内参标定数据库中，以及将标定内参数发送给目标终端，以使目标终端根据标定内参数更新目标终端的本地内参数。以及，标定内参数不满足内参数有效性条件时，将预设内参数发送给目标终端，以使目标终端更新目标终端的本地内参数。
在一种可能的实施方式中，获取模块，具体用于：
接收来自目标终端的第一图像以及目标终端的标识。
第三方面，本申请还提供一种相机内参标定设备，包括处理器和存储器，其中，处理器和存储器相连，其中，存储器用于存储程序代码，处理器用于调用程序代码，以执行如第一方面所述的相机内参标定方法。
第四方面，本申请还提供一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序被处理器执行以实现如第一方面所述的相机内参标定方法。
第五方面，本申请还提供一种包含指令的计算机程序产品，当所述计算机程序产品在计算机上运行时，使得计算机执行第一方面所述的相机内参标定方法。
第六方面，本申请还提供一种芯片，所述芯片包括处理器与数据接口，所述处理器通过所述数据接口读取存储器上存储的指令，执行第一方面所述的相机内参标定方法。
可选地，作为一种实现方式，所述芯片还可以包括存储器，所述存储器中存储有指令，所述处理器用于执行所述存储器上存储的指令，当所述指令被执行时，所述处理器用于执行第一方面所述的相机内参标定方法。
附图说明
下面对本申请实施例用到的附图进行介绍。
图1是本申请实施例提供的一种系统架构示意图；
图2是本申请实施例提供的一种卷积神经网络的结构示意图；
图3是本申请实施例提供的一种相机内参标定方法的流程示意图；
图4是本申请实施例提供的一种相机内参标定方法的示意图；
图5a是本申请实施例提供的一种确定第一内参标定数据的流程示意图；
图5b是本申请实施例提供的一种确定标定内参数的流程示意图；
图5c是本申请实施例提供的一种相机内参标定方法的流程示意图；
图6a是本申请实施例提供的另一种确定标定内参数的流程示意图；
图6b是本申请实施例提供的另一种相机内参标定方法的流程示意图；
图7是本申请实施例提供的一种相机内参标定装置的结构示意图；
图8是本申请实施例提供的一种相机内参标定设备的结构示意图。
具体实施方式
下面将结合附图，对本申请中的技术方案进行描述。其中，在本申请实施例中，“示例性的”或者“例如”等词用于表示作例子、例证或说明。本申请中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其他实施例或设计方案更优选或更具优 势。确切而言，使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念。
本申请中实施例提到的“至少一个”是指一个或者多个，“多个”是指两个或两个以上。“以下至少一项(个)”或其类似表达，是指的这些项中的任意组合，包括单项(个)或复数项(个)的任意组合。例如，a、b、或c中的至少一项(个)，可以表示：a、b、c、(a和b)、(a和c)、(b和c)、或(a和b和c)，其中a、b、c可以是单个，也可以是多个。“和/或”，描述关联对象的关联关系，表示可以存在三种关系，例如，a和/或b，可以表示：单独存在a、同时存在a和b、单独存在b这三种情况，其中a、b可以是单数或者复数。字符“/”一般表示前后关联对象是一种“或”的关系。而本申请中实施例的步骤的序号(如步骤S1、步骤S21等)只为了区分不同的步骤，不对步骤之间的先后执行顺序造成限定。
以及，除非有相反的说明，本申请实施例使用“第一”、“第二”等序数词是用于对多个对象进行区分，不用于限定多个对象的顺序、时序、优先级或者重要程度。例如，第一图像和第二图像，只是为了便于描述，而并不是表示这第一图像和第二图像的顺序或重要程度等的不同。
上述实施例中所用，根据上下文，术语“当……时”可以被解释为意思是“如果……”或“在……后”或“响应于确定……”或“响应于检测到……”。以上所述仅为本申请的可选实施例，并不用以限制本申请，凡在本申请的构思和原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。
为了便于理解，下面先对本申请实施例涉及的相关术语等相关概念进行介绍。
(1)、深度学习模型
机器学习是一种人工智能技术。深度学习是一种机器学习算法，通过使用多层处理单元来处理输入和提供相应输出。深度学习算法可以由多层神经网络(NN)来实现。这样的多层神经网络也被称为“深度学习模型”。
(2)、神经网络
神经网络可以是由神经单元组成的，神经单元可以是指以xs和截距1为输入的运算单元，该运算单元的输出可以为：
其中，s＝1、2、……n，n为大于1的自然数，Ws为xs的权重，b为神经单元的偏置。f为神经单元的激活函数(Activation Functions)，用于将非线性特性引入神经网络中，来将神经单元中的输入信号转换为输出信号。该激活函数的输出信号可以作为下一层卷积层的输入。激活函数可以是Sigmoid函数。神经网络是将许多个上述单一的神经单元联结在一起形成的网络，即一个神经单元的输出可以是另一个神经单元的输入。每个神经单元的输入可以与前一层的局部接受域相连，来提取局部接受域的特征，局部接受域可以是由若干个神经单元组成的区域。
(3)、深度神经网络
深度神经网络(Deep Neural Network，DNN)，也称多层神经网络，可以理解为具有很多层隐含层的神经网络，这里的“很多”并没有特别的度量标准。从DNN按不同层的位置划分，DNN内部的神经网络可以分为三类：输入层，隐含层，输出层。一般来说第一层是输入层，最后一层是输出层，中间的层数都是隐含层。层与层之间是全连接的，也就是说，第i层的任意一个神经元一定与第i+1层的任意一个神经元相连。虽然DNN看起来很复杂，但 是就每一层的工作来说，其实并不复杂，简单来说就是如下线性关系表达式：其中，是输入向量，是输出向量，是偏移向量，W是权重矩阵(也称系数)，α()是激活函数。每一层仅仅是对输入向量经过如此简单的操作得到输出向量由于DNN层数多，则系数W和偏移向量的数量也就很多了。这些参数在DNN中的定义如下所述：以系数W为例：假设在一个三层的DNN中，第二层的第4个神经元到第三层的第2个神经元的线性系数定义为上标3代表系数W所在的层数，而下标对应的是输出的第三层索引2和输入的第二层索引4。
总结就是：第L-1层的第k个神经元到第L层的第j个神经元的系数定义为需要注意的是，输入层是没有W参数的。在深度神经网络中，更多的隐含层让网络更能够刻画现实世界中的复杂情形。理论上而言，参数越多的模型复杂度越高，“容量”也就越大，也就意味着它能完成更复杂的学习任务。训练深度神经网络的也就是学习权重矩阵的过程，其最终目的是得到训练好的深度神经网络的所有层的权重矩阵(由很多层的向量W形成的权重矩阵)。
(4)、卷积神经网络
卷积神经网络(CNN，Convolutional Neuron Network)是一种带有卷积结构的深度神经网络。卷积神经网络包含了一个由卷积层和子采样层构成的特征抽取器。该特征抽取器可以看作是滤波器，卷积过程可以看作是使用一个可训练的滤波器与一个输入的图像或者卷积特征平面(Feature Map)做卷积。卷积层是指卷积神经网络中对输入信号进行卷积处理的神经元层。在卷积神经网络的卷积层中，一个神经元可以只与部分邻层神经元连接。一个卷积层中，通常包含若干个特征平面，每个特征平面可以由一些矩形排列的神经单元组成。同一特征平面的神经单元共享权重，这里共享的权重就是卷积核。共享权重可以理解为提取图像信息的方式与位置无关。这其中隐含的原理是：图像的某一部分的统计信息与其他部分是一样的。即意味着在某一部分学习的图像信息也能用在另一部分上。所以对于图像上的所有位置，都能使用同样的学习得到的图像信息。在同一卷积层中，可以使用多个卷积核来提取不同的图像信息，一般地，卷积核数量越多，卷积操作反映的图像信息越丰富。
卷积核可以以随机大小的矩阵的形式初始化，在卷积神经网络的训练过程中卷积核可以通过学习得到合理的权重。另外，共享权重带来的直接好处是减少卷积神经网络各层之间的连接，同时又降低了过拟合的风险。
(5)、相机成像
相机成像系统中，共包含四个坐标系：世界坐标系、相机坐标系、图像坐标系、像素坐标系。
世界坐标系是描述目标物体在真实世界的位置而引入的三维世界坐标系，用来描述三维空间中的物体和相机之间的坐标位置，满足右手法则。
相机坐标系是以相机为中心，从相机角度描述物体位置，作为像素坐标系和世界坐标系的桥梁。
图像坐标系是描述真实物体在相机焦距处成像的坐标系，用来连接相机坐标系与像素坐标系。
像素坐标系是描述物体在照片上数字图像的位置而引入的一种数字坐标系。
而相机成像的过程实质上是上述几个坐标系的转换，具体地，空间中的一点由世界坐标系转换到相机坐标系，然后再将其投影到成像平面(即从相机坐标系转换到图像坐标系)，最后再将成像平面上的数据转换到图像平面(即从图像坐标系转换到像素坐标系)。
把相机认为是一个函数，输入量是一个场景，输出量是一幅灰度图。这个从三维到二维 的过程的函数是不可逆的。相机标定的目标是找一个合适的数学模型，求出这个模型的参数，这样能够近似这个三维到二维的过程，使这个三维到二维的过程的函数找到反函数。这个逼近的过程就是相机标定。换句话说，相机标定是从世界坐标系转换到像素坐标系的过程，相机标定的目的包括获得相机的内参数和外参数。
相机的外参数用于把坐标从世界坐标系转换到相机坐标系中，外参数是在世界坐标系中的参数，比如相机的位置、旋转方向等。
而相机的内参数用于把坐标从相机坐标系转换到像素坐标系中，内参数是与相机自身特性相关的参数，比如相机的焦距(如X轴焦距、Y轴焦距)、像主点坐标(如X轴上的像主点坐标、Y轴上的像主点坐标)等。相机的内参数一般都是相机出厂就定下来的，可以通过相机标定的方式计算出具体的内参数。
本申请实施例提供的相机内参标定方法能够应用在增强现实(Augmented Reality，AR)场景、视觉定位导航场景等机器视觉场景，具体地，机器视觉场景中，机器视觉应用利用相机获取图像，进而根据获取到的图像完成相关的机器视觉任务。为了使得机器视觉应用的机器视觉任务正常完成，需要先对相机进行参数标定，包括标定相机的内参数。
下面将对相机内参标定进行简单的介绍。目前，可以通过标定板对终端的相机进行内参数标定，这种方法需要事先通过离线标定的方式获取内参数并储存，然后AR应用的算法才能读取和调用相机的内参，但是这样的标定方法需要耗费大量人力物力，而且不能获取实时的相机内参数用于AR应用后续的计算。
因此，本申请实施例提供一种相机内参标定方法，可以对相机内参进行实时标定，有效降低相机内参标定的工作量，节省相机内参标定所需的人力物力。
参见附图1，本申请实施例提供了一种系统架构100。如所述系统架构100所示，数据采集设备160用于采集训练数据，本申请实施例中训练数据包括训练图像；并将训练数据存入数据库130，训练设备120基于数据库130中维护的训练数据训练得到目标模型101，这里的目标模型101可以为经训练阶段训练得到的特征检测模型、图像检索模型等。在实际的应用中，所述数据库130中维护的训练数据不一定都来自于数据采集设备160的采集，也有可能是从其他设备接收得到的。另外需要说明的是，训练设备120也不一定完全基于数据库130维护的训练数据进行目标模型101的训练，也有可能从云端或其他地方获取训练数据进行模型训练，上述描述不应该作为对本申请实施例的限定。
根据训练设备120训练得到的目标模型101可以应用于不同的系统或设备中，如应用于图1所示的执行设备110，所述执行设备110可以是终端，如手机终端，平板电脑，笔记本电脑，AR/VR，车载终端等，还可以是服务器或者云端等。在附图1中，执行设备110配置有I/O接口112，用于与外部设备进行数据交互，用户可以通过客户设备140向I/O接口112输入数据，所述输入数据在本申请实施例中可以包括第一图像，也就是说，在执行设备110上执行本申请的相机内参标定方法，对第一图像进行处理，进而得到相机的标定内参数。数据库130也可以根据需要将训练数据输入至I/O接口112中，例如，将训练图像作为第一图像时。
预处理模块113用于根据I/O接口112接收到的输入数据(如所述第一图像)进行预处理，在本申请实施例中，预处理模块113可以用于输入数据进行图像复原处理或图像增强处理等预处理，以使输入数据达到预设的标准。
其中，在实际应用中常常发生图像退化现象。例如大气流的扰动，光学系统的像差，相机和物体的相对运动都会使遥感图像发生退化。因此，对图像进行复原处理的目的是校正各种原因所造成的图像退化，使重建或估计得到的图像尽可能逼近于理想无退化的像场。图像复原处理的原理为：首先要对图像退化的整个过程加以适当的估计，在此基础上建立近似的退化数学模型，之后还需要对模型进行适当的修正，以对退化过程出现的失真进行补偿，以保证复原之后所得到的图像趋近于原始图像，实现图像的最优化。
而图像增强处理是对图像中的信息有选择地加强和抑制，以改善图像的视觉效果，或将图像转变为更适合于机器处理的形式，以便于数据抽取或识别。例如一个图像增强系统可以通过高通滤波器来突出图像的轮廓线，从而使机器能够测量轮廓线的形状和周长。图像增强技术有多种方法，反差展宽、对数变换、密度分层和直方图均衡等都可用于改变图像灰调和突出细节。
在执行设备110对输入数据进行预处理，或者在执行设备110的计算模块111执行计算等相关的处理过程中，执行设备110可以调用数据存储系统150中的数据、代码等以用于相应的处理，也可以将相应处理得到的数据、指令等存入数据存储系统150中。数据存储系统150具体可以为本申请实施例中的历史内参标定数据库。
最后，I/O接口112将处理结果，如上述得到的相机的标定内参数返回给客户设备140，从而提供给用户。
在附图1中所示情况下，用户可以手动给定输入数据，该手动给定可以通过I/O接口112提供的界面进行操作。另一种情况下，客户设备140可以自动地向I/O接口112发送输入数据，如果要求客户设备140自动发送输入数据需要获得用户的授权，则用户可以在客户设备140中设置相应权限。用户可以在客户设备140查看执行设备110输出的结果，具体的呈现形式可以是显示、声音、动作等具体方式。客户设备140也可以作为数据采集端，采集如图1所示输入I/O接口112的输入数据及I/O接口112的输出结果作为新的样本数据，并存入数据库130。当然，也可以不经过客户设备140进行采集，而是由I/O接口112直接将如图1所示输入I/O接口112的输入数据及I/O接口112的输出结果，作为新的样本数据存入数据库130。
其中，附图1仅是本申请实施例提供的一种系统架构的示意图，图中所示设备、器件、模块等之间的位置关系不构成任何限制，例如，在附图1中，数据存储系统150相对执行设备110是外部存储器，在其它情况下，也可以将数据存储系统150置于执行设备110中。
如图1所示，根据训练设备120训练得到目标模型101，该目标模型101在本申请实施例中可以是特征检测模型、图像检索模型等，特征检测模型用于提取图像的特征点并对特征点进行特征描述得到特征描述子。特征描述子包括但不限于尺度不变特征变换(Scale-Invariant Feature Transform，SIFT)特征描述子、加速稳健特征(Speeded Up Robust Features，SURF)描述子、方向梯度直方图(Histogram of Oriented Gradient,HOG)特征描述子、局部二值模式(Local Binary Pattern，LBP)特征描述子以及哈尔特征描述子。图像检索模型用于匹配与目标图像相似的图像。特征检测模型、图像检索模型可以为深度学习模型。本申请方案中具体的特征检测模型或图像检索模型可以是神经网络模型。
如前文的基础概念介绍所述，卷积神经网络是一种带有卷积结构的深度神经网络，是一种深度学习(deep learning)架构，深度学习架构是指通过机器学习的算法，在不同的抽象层级上进行多个层次的学习。作为一种深度学习架构，CNN是一种前馈(feed-forward)人工神经网络，该前馈人工神经网络中的各个神经元可以对输入其中的图像作出响应。
如图2所示，图2是本申请实施例提供的一种卷积神经网络的结构示意图。卷积神经网络(CNN)200可以包括输入层210，卷积层/池化层220(其中池化层为可选的)，以及全连接层230。待处理图像通过输入层210输入卷积神经网络200中进行处理。其中：
卷积层/池化层220：
卷积层：
如图2所示卷积层/池化层220可以包括如图2中示例的221-226层，举例来说：在一种实现中，221层为卷积层，222层为池化层，223层为卷积层，224层为池化层，225为卷积层，226为池化层。在另一种实现方式中，221、222为卷积层，223为池化层，224、225为卷积层，226为池化层。即卷积层的输出可以作为随后的池化层的输入，也可以作为另一个卷积层的输入以继续进行卷积操作。
下面将以卷积层221为例，介绍一层卷积层的内部工作原理。
卷积层221可以包括很多个卷积算子，卷积算子也称为核，其在图像处理中的作用相当于一个从输入图像矩阵中提取特定信息的过滤器，卷积算子本质上可以是一个权重矩阵，这个权重矩阵通常被预先定义，在对图像进行卷积操作的过程中，权重矩阵通常在输入图像上沿着水平方向一个像素接着一个像素(或两个像素接着两个像素……这取决于步长Stride的取值)的进行处理，从而完成从图像中提取特定特征的工作。该权重矩阵的大小应该与图像的大小相关，需要注意的是，权重矩阵的纵深维度(Depth Dimension)和输入图像的纵深维度是相同的，在进行卷积运算的过程中，权重矩阵会延伸到输入图像的整个深度。因此，和一个单一的权重矩阵进行卷积会产生一个单一纵深维度的卷积化输出，但是大多数情况下不使用单一权重矩阵，而是应用多个尺寸(行×列)相同的权重矩阵，即多个同型矩阵。每个权重矩阵的输出被堆叠起来形成卷积图像的纵深维度，这里的维度可以理解为由上面所述的“多个”来决定。不同的权重矩阵可以用来提取图像中不同的特征，例如一个权重矩阵用来提取图像边缘信息，另一个权重矩阵用来提取图像的特定颜色，又一个权重矩阵用来对图像中不需要的噪点进行模糊化等。该多个权重矩阵尺寸(行×列)相同，经过该多个尺寸相同的权重矩阵提取后的特征图的尺寸也相同，再将提取到的多个尺寸相同的特征图合并形成卷积运算的输出。
这些权重矩阵中的权重值在实际应用中需要经过大量的训练得到，通过训练得到的权重值形成的各个权重矩阵可以用来从输入图像中提取信息，从而使得卷积神经网络200进行正确的预测。
当卷积神经网络200有多个卷积层的时候，初始的卷积层(例如221)往往提取较多的一般特征，该一般特征也可以称之为低级别的特征；随着卷积神经网络200深度的加深，越往后的卷积层(例如226)提取到的特征越来越复杂，比如高级别的语义之类的特征，语义越高的特征越适用于待解决的问题。
池化层：
由于常常需要减少训练参数的数量，因此卷积层之后常常需要周期性的引入池化层，在如图2中220所示例的221-226各层，可以是一层卷积层后面跟一层池化层，也可以是多层卷积层后面接一层或多层池化层。在图像处理过程中，池化层的唯一目的就是减少图像的空间大小。池化层可以包括平均池化算子和/或最大池化算子，以用于对输入图像进行采样得到较小尺寸的图像。平均池化算子可以在特定范围内对图像中的像素值进行计算产生平均值作为平均池化的结果。最大池化算子可以在特定范围内取该范围内值最大的像素作为最大池化的结果。另外，就像卷积层中用权重矩阵的大小应该与图像尺寸相关一样，池化层中的运算 符也应该与图像的大小相关。通过池化层处理后输出的图像尺寸可以小于输入池化层的图像的尺寸，池化层输出的图像中每个像素点表示输入池化层的图像的对应子区域的平均值或最大值。
全连接层230：
在经过卷积层/池化层220的处理后，卷积神经网络200还不足以输出所需要的输出信息。因为如前所述，卷积层/池化层220只会提取特征，并减少输入图像带来的参数。然而为了生成最终的输出信息(所需要的类信息或其他相关信息)，卷积神经网络200需要利用全连接层230来生成一个或者一组所需要的类的数量的输出。因此，在全连接层230中可以包括多层隐含层(如图2所示的231、232至23n)以及输出层240，该多层隐含层中所包含的参数可以根据具体的任务类型的相关训练数据进行预先训练得到，例如该任务类型可以包括图像特征检测，图像检索等等。
在全连接层230中的多层隐含层之后，也就是整个卷积神经网络200的最后层为输出层240，该输出层240具有类似分类交叉熵的损失函数，具体用于计算预测误差，一旦整个卷积神经网络200的前向传播(如图2由210至240方向的传播为前向传播)完成，反向传播(如图2由240至210方向的传播为反向传播)就会开始更新前面提到的各层的权重值以及偏差，以减少卷积神经网络200的损失，及卷积神经网络200通过输出层输出的结果和理想结果之间的误差。
需要说明的是，如图2所示的卷积神经网络200仅作为一种卷积神经网络的示例，在具体的应用中，卷积神经网络还可以以其他网络模型的形式存在。
下面对相机内参标定方法进行具体说明。
该相机内参标定方法可以应用于电子设备。本申请实施例中，电子设备可以为智能手机、服务器、智能电视、平板电脑、桌面型计算机、膝上型计算机、手持计算机、笔记本电脑、超级移动个人计算机(Ultra-Mobile Personal Computer，UMPC)、上网本，以及蜂窝电话、个人数字助理(Personal Digital Assistant，PDA)、增强现实(Augmented Reality，AR)设备、虚拟现实(Virtual Reality，VR)设备、人工智能(Artificial Intelligence,AI)设备、可穿戴式设备、车载设备、智能家居设备和/或智慧城市设备等。本申请实施例对该电子设备的具体类型不作特殊限制。电子设备根据第一图像、目标终端的标识、高精度三维地图以及历史内参标定数据库进行处理，最终可以获得目标终端的相机的标定内参。可以实时对相机内参进行标定，有效降低相机内参标定的工作量。
参考图3和图4，图3是本申请实施例提供的一种相机内参标定方法的流程示意图，图4是本申请实施例提供的一种相机内参标定方法的示意图。相机内参标定方法300包括步骤301、步骤302和步骤303。
301、获取第一图像和目标终端的标识。
具体地，该第一图像为目标终端在第一位置处利用自身的相机拍摄得到的图像，第一位置位于预设区域内，第一图像所拍摄的场景位于预设区域内。第一图像可以为一张或多张。上述预设区域的具体范围可以根据实际需要进行设置，不做具体限定。例如，预设区域为公园A、广场B或者商场C等。而目标终端的标识可以为目标终端的唯一标记信息或者目标终端的型号信息等可以标识终端的信息，目标终端的唯一标记可以由数字、字母、特殊字符等中的至少一个来组成，例如，目标终端的唯一标记为BJ22-2598。目标终端的型号可以由数字、字母、特殊字符等中的至少一个来组成，例如，目标终端的型号为XH1-123。
进一步地，目标终端可以为上述电子设备本身，此时，用户可以利用目标终端在预设区域内获取第一图像。或者，目标终端和电子设备为两个不同的设备，此时，目标终端获取第一图像和自身的标识，将第一图像和标识发送给电子设备，电子设备接收来自目标终端的第一图像以及目标终端的标识，并根据第一图像和目标终端的标识进行图3所示的方法步骤的处理。
302、根据第一图像和预设区域对应的高精度三维地图，确定相机的第一内参标定数据；以及，根据目标终端的标识和历史内参标定数据库确定目标终端的历史内参标定数据。
具体地，高精度三维地图为上述预设区域对应的三维地图。历史内参标定数据库包括一个或多个标识的终端的历史内参标定数据。历史内参标定数据库可以设置在电子设备中，例如，当电子设备为云端的服务器时，服务器执行图3所示的方法，并设置有历史内参标定数据库。或者，历史内参标定数据库可以设置在其他设备上，电子设备通过访问其他设备以建立与历史内参标定数据库的连接。而第一内参标定数据是与目标终端的标定参数相关的数据。
进一步地，高精度三维地图为精度更高的电子地图。预设区域对应的高精度三维地图包括预设区域中的多张场景图像、上述场景图像的二维特征点坐标、二维特征点的特征描述子以及二维特征点对应的三维空间点坐标。其中，多张是指大于或等于两张，二维特征点坐标是指场景图像的二维特征点在场景图像中的坐标，即二维特征点坐标在像素坐标系下的坐标，而三维空间点坐标是指场景图像的二维特征点在世界坐标系中的坐标。示例性地，高精度三维地图的二维特征点坐标的精度在5个像素以内。高精度三维地图的坐标精度在10-50厘米左右。例如，预设区域为室内场景时，高精度三维地图的三维空间点坐标的精度为10厘米；预设区域为室外场景时，高精度三维地图的三维空间点坐标的精度为20厘米。
具体地，上述场景图像是在预设区域中的若干个图像获取位置上获取的预设方向(如上下左右等方向)的图像。利用场景图像可以进行特征点提取以确定二维特征点坐标以及二维特征点对应的特征描述子，再获取二维特征点对应的三维空间点坐标，即可以建立高精度三维地图。上述二维特征点坐标以及相应的三维空间点坐标可以理解为像素坐标系中的二维特征点与世界坐标系中的三维空间点的坐标对应关系。
例如，预设区域为公园A，则可以从公园A的入口出发，每间隔一定距离(例如0.5米等)获取一次场景图像，每次获取至少一张场景图像，直到公园A的出口，可以确定若干个图像获取位置。在每个图像获取位置上获取场景图像时，可以是获取图像获取位置的左侧视野图像作为场景图像，同样地，还可以获取图像获取位置的右侧、上方、前方等方向的视野图像作为场景图像。基于上述获得的场景图像可以建立预设区域对应的高精度三维地图。
303、根据第一内参标定数据和目标终端的历史内参标定数据确定相机的标定内参数。
具体地，本申请实施例利用高精度三维地图和历史内参标定数据库先分别得到第一内参标定数据和目标终端的历史内参标定数据，进而根据第一内参标定数据和目标终端的历史内参标定数据进行内参数标定，确定相机的标定内参数。
本申请实施例的标定方法，不需要利用额外的标定设备(如标定板)进行离线标定，也不需要用户采用特定的动作来完成内参标定。用户只需要在预设区域中，使用目标终端的相机获取第一图像即可自动完成标定，因此，可以实现实时对目标终端的相机内参进行标定，有效提升了标定效率，相比现有技术中需要人工离线标定，可以有效降低相机内参标定所需的工作量。进一步地，目标终端中的机器视觉应用可以根据标定内参数进行其他相关处理，以正常完成相关的机器视觉任务。因此，本申请实施例的标定方法，不会阻塞机器视觉应用的进程，即使是未标定过的目标终端，也能够和已标定的目标终端一样能够正常使用机器视 觉应用，减少用户的操作，给用户带来更大的便利，还能提升机器视觉应用在不同目标终端的普及率。
参考图3，示例性地，上述相机内参标定方法还包括：
标定内参数满足内参数有效性条件时，电子设备根据标定内参数更新目标终端的本地内参数，并将第一内参标定数据保存至历史内参标定数据库中。而标定内参数不满足内参数有效性条件时，电子设备根据预设内参数更新目标终端的本地内参数。
具体地，内参数有效性条件是内参数需要满足的标准，具体的标准可以根据实际情况进行设置。而预设内参数的具体数值可以根据实际需要进行设置，不做特别限定。本申请实施例中，此时的电子设备为上述目标终端，即在目标终端执行获取标定内参数的方法步骤，再判断进行标定内参数有效性判断，当标定内参数满足内参数有效性条件时，目标终端根据标定内参数更新其本地内参数，并将第一内参标定数据保存至历史内参标定数据库中，更新历史内参标定数据库。而标定内参数不满足内参数有效性条件时，目标终端根据预设内参数更新其本地内参数。
可见，利用上述数据更新的步骤，无需人工专门更新和维护历史内参标定数据库；而且可以在用户后续使用机器视觉应用的过程中，基于历史内参标定数据库中保存的数据越来越多，可以不断提升相机内参数的标定精度，给用户带来越用越好的体验。另外，相同型号的目标终端之间可以通过历史内参标定数据库实现标定内参数的共享，有利于提升首次标定的目标终端的内参标定精度。
参考图3，又一示例，上述相机内参标定方法还包括：
标定内参数满足内参数有效性条件时，电子设备将第一内参标定数据保存至历史内参标定数据库中，以及将标定内参数发送给目标终端，以使目标终端根据标定内参数更新本地内参数。而标定内参数不满足内参数有效性条件时，电子设备将预设内参数发送给目标终端，以使目标终端更新目标终端的本地内参数。
本申请实施例中，电子设备和目标终端为不同的设备，在标定内参数满足内参数有效性条件时，电子设备将标定内参数发送给目标终端，以使目标终端更新其本地内参数，即将本地内参数更新为上述标定内参数，并将第一内参标定数据保存至历史内参标定数据库中。而标定内参数不满足内参数有效性条件时，电子设备将预设内参数发送给目标终端，以使目标终端将本地内参数更新为接收到的预设内参数。
参考图5a，图5a是本申请实施例提供的一种确定第一内参标定数据的流程示意图；上述根据第一图像和预设区域对应的高精度三维地图，确定相机的第一内参标定数据，包括步骤511、步骤512、步骤513和步骤514。其中：
511、从高精度三维地图中确定与第一图像的图像相似度最高的N张场景图像，N为大于零的整数。
具体地，N的具体数值可以根据实际情况进行设置，不做特别限定。根据第一图像，利用图像检索模型可以从高精度三维地图中确定与第一图像的图像相似度最高的N张场景图像，例如十张场景图像。
512、确定第一图像的M个图像特征点，M为大于零的整数。
513、根据M个图像特征点的特征描述子和N张场景图像的二维特征点对应的特征描述子进行匹配，确定M个图像特征点中的X个图像特征点的坐标对。坐标对包括二维特征点坐标和三维空间点坐标，X小于或等于M。
具体地，利用特征检测模型，可以对第一图像进行图像特征点提取，以得到第一图像对 应的M个图像特征点，另外，特征检测模型还可以对图像特征点进行特征描述得到M个图像特征点对应的特征描述子。本申请实施例中，对特征描述子的具体种类不做特别限定。
进一步地，由于第一图像是在预设区域中的第一位置拍摄得到的图像，因此，利用图像特征点的特征描述子在高精度三维地图中进行匹配，可以确定第一图像的X个图像特征点对应的坐标对。具体地，根据M个图像特征点的特征描述子和上述N张场景图像的二维特征点对应的特征描述子进行匹配。当前后两者的特征描述子相同时，则将匹配到的特征描述子对应的二维特征点坐标以及三维空间点坐标作为图像特征点的坐标对。
514、根据X个图像特征点的坐标对确定第一内参标定数据。
本申请实施例中，由于相机的外参数(已知)用于把坐标从世界坐标系转换到相机坐标系中，相机的内参数用于把坐标从相机坐标系转换到像素坐标系中；而二维特征点坐标为像素坐标系中的坐标，三维空间点坐标为世界坐标系中的坐标，因此，利用二维特征点坐标以及三维空间点坐标之间的对应关系可以确定第一内参标定数据。
示例性地，在根据X个图像特征点的坐标对确定第一内参标定数据时，可以直接将X个图像特征点的坐标对确定为第一内参标定数据。
参考图5b，图5b是本申请实施例提供的一种确定标定内参数的流程示意图；根据第一内参标定数据和目标终端的历史内参标定数据确定相机的标定内参数，包括步骤521、步骤522和步骤523。其中：
521、根据X个图像特征点的坐标对确定相机的第一内参数。
具体地，此时的第一内参标定数据为X个图像特征点的坐标对，基于X个图像特征点的坐标对进行计算，可以确定相机的第一内参数，示例性地，具体的计算过程如下：
假设在目标终端获取第一图像的时刻，目标终端的相机在世界坐标系中的位姿为[R t]，R为旋转矩阵，t为平移向量；世界坐标系中的一个三维空间点的位置为(Xi Yi Zi)T，将该三维空间点的位置坐标投影至像素坐标系上，可以得到像素坐标(ui vi)T。相应的变换关系如下：
其中，Si为缩放尺度，K为相机的内参数矩阵，令M＝K[R t]，则M矩阵为3×4维数，公式(1)可以转化为公式(2)：
而公式(2)可以简化为以下两个等式约束，如下所示：
将公式(3)矩阵化可以得到公式(4)：
从公式(4)可以看出，一个坐标对可以得到两个约束，M矩阵一共12个未知数，至少6个不共面的特征点就可以解算出M矩阵。进一步再根据M矩阵和相机的内参数矩阵K之间的关系即可计算出相机的内参数矩阵K，即相机的第一内参数。
522、根据X个图像特征点的坐标对、目标终端的历史内参标定数据确定相机的第二内参数。
具体地，当将X个图像特征点的坐标对确定为第一内参标定数据时，目标终端的历史内参标定数据也为坐标对的形式，即历史内参标定数据为历史保存的与目标终端的标识相同的第一内参标定数据(当目标终端的标识为目标终端的唯一标记时，目标终端对应的历史内参标定数据是目标终端在过往标定过程中保存到数据库中的第一内参标定数据。而当目标终端的标识为目标终端的型号时，目标终端对应的历史内参标定数据为包括目标终端在内的型号相同的终端在过往标定过程中保存到数据库中的第一内参标定数据)。因此，联合上述X个图像特征点的坐标对和目标终端的历史内参标定数据，可以确定相机的第二内参数。第二内参数的计算方法与第一内参数的计算方法类似，不同的是，在第二内参数的计算过程中，需要为X个图像特征点的坐标对和目标终端的历史内参标定数据分别赋予不同的权重，以加权运算的结果进行运算。其中，上述权重可以是三维空间点的深度距离的倒数或者是二维特征点的响应值。
示例性地，具体加权计算的过程如下：
假设X个图像特征点的坐标对以及目标终端的历史内参标定数据总共有n个坐标对，在计算M矩阵时，可以根据坐标对各自的权重进行加权计算，从而获得加权计算后的M矩阵用于计算相机内参数，即第二内参数。
假设公式(4)中的系数为Ai矩阵，M矩阵的12个未知数为X矩阵，则公式(4)可以写成如下形式：
AiX＝0           (5)
设定第i个坐标对的权重为wi，wi矩阵为2×2维数，坐标对加权后的公式(5)可以表示为：
wiAiX＝0          (6)
则n个坐标对可以构成方程组，如下：
根据公式(7)即可计算出未知数X矩阵，也就是M矩阵的12个未知数，从而得到M矩阵。
523、将第一内参数、第二内参数中重投影误差最小的一个确定为标定内参数。
具体地，本申请实施例中，分别确定第一内参数和第二内参数，再将前后两者中重投影误差最小的一个确定为相机的标定内参数，利用重投影误差比较可以有效提升标定内参数的精度。
其中，示例性地，第一内参数的重投影误差是基于第一内参数对坐标对集合(此处的坐标对集合是指X个图像特征点的坐标对以及目标终端的历史内参标定数据所组成的集合)中的三维空间点进行重投影运算，可以确定重投影后的二维特征点坐标，根据重投影后得到的二维特征点坐标和坐标对集合中的二维特征点坐标可以确定第一内参数的重投影误差。其中，对于每个坐标对可以得到一个重投影后的二维特征点坐标，相应地可以得到一个子重投影误差，根据坐标对集合对应的多个子重投影误差计算平均值，将该平均值作为第一内参数的重投影误差。或者，根据坐标对集合对应的多个重投影前的二维特征点坐标和重投影后的二维特征点坐标计算均方根误差，将该均方根误差作为第一内参数的重投影误差。同理，利用第一内参数的重投影误差的计算方法，可以确定第二内参数的重投影误差。
进一步地，标定内参数满足内参有效性条件时，除了将X个图像特征点的坐标对存入历史内参标定数据库中之外，还可以将X个图像特征点的坐标对对应的权重存入历史内参标定数据库中，该权重可以坐标对中三维空间点的深度距离的倒数或者是二维特征点的响应值。这样，下一次标定在计算第二内参数时，可以直接从历史内参标定数据库中取出权重进行加权计算。
参考图5c，图5c是本申请实施例提供的一种相机内参标定方法的流程示意图。下面机器视觉应用以增强现实应用为例，而目标终端的标识以型号为例，而电子设备为云端服务器为例，对本申请实施例的标定方法进行具体说明。
步骤一：打开终端设备，用户进入增强现实应用；其中，增强现实应用中包含本申请实施例的标定方法对应的算法。
步骤二：目标终端获取终端型号和一张在当前位置上利用终端的相机拍摄的图像等数据，目标终端向云端服务器上传终端型号和拍摄的图像等数据。
步骤三：根据当前位置图像从高精度三维地图中完成图像检索，得到场景图像，并通过当前位置的图像和场景图像之间的特征点匹配得到坐标对。
步骤四：根据终端型号从历史内参标定数据库中检索出该型号对应的历史有效坐标对。
步骤五：基于步骤三中匹配得到的坐标对实时计算得到相机的内参数K1。
步骤六：结合步骤三中匹配得到的坐标对和步骤四从历史内参标定数据库中检索出的相同型号的有效坐标对数据，联合计算得到相机内参数在计算时可以根据坐标对各自的权重进行加权计算，从而获得相机内参数
步骤七：比较使用步骤五和步骤六两种方式得到的相机内参数K1和分别计算三维空间点的重投影误差大小，使用重投影误差较小的相机内参数作为内参数标定结果输出。
步骤八：对步骤七得到的内参数标定结果进行有效性的判断。如果内参数标定结果满足有效性判断条件，则认为本次标定结果有效，并返回内参数标定成功的标志位和有效的相机内参数，同时把该次匹配得到的坐标对数据和对应的权重(权重可以为坐标对中三维空间点的深度距离的倒数或者是二维特征点的响应值)作为有效标定数据保存至历史内参标定数据库中。如果有任意一个有效性判断条件不成立，则认为当前内参数标定结果无效，并返回内参数标定失败的标志位和默认的相机内参数。
步骤九：终端接收云端服务器返回的标志位和标定结果，终端使用返回的相机内参数用 于增强现实应用。如果是内参数标定成功的标志位，则使用计算的标定结果作为本次增强现实应用使用过程中的相机内参数，并在本次应用使用过程中不再进行标定。如果是内参数标定失败的标志位，则先使用默认的相机内参数用于增强现实应用，不会阻塞应用的进程，并返回步骤二再次请求标定，直至标定成功。
又一示例性地，在根据X个图像特征点的坐标对确定第一内参标定数据时，可以先根据X个图像特征点的坐标对确定相机的第一内参数，再将第一内参数确定为第一内参标定数据。其中，根据X个图像特征点的坐标对确定相机的第一内参数可以参考上述记载，不再赘述。
参考图6a，图6a是本申请实施例提供的另一种确定标定内参数的流程示意图；根据第一内参标定数据和目标终端的历史内参标定数据确定相机的标定内参数，包括步骤601和步骤602。其中：
601、根据第一内参数及其第一权重、目标终端对应的每个历史内参标定数据及其第二权重进行加权计算，以确定相机的第二内参数。
具体地，此时历史内参标定数据的形式为相机的内参数。当目标终端的标识为目标终端的唯一标记时，目标终端对应的历史内参标定数据是目标终端在过往标定过程中保存到数据库中的第一内参标定数据，即第一内参数。而当目标终端的标识为目标终端的型号时，目标终端对应的历史内参标定数据为包括目标终端在内的型号相同的终端在过往标定过程中保存到数据库中的第一内参标定数据，即第一内参数。目标终端对应的历史内参标定数据，即历史保存的第一内参数的个数是一个以上。
示例性地，加权计算的具体计算过程如下：
假设第一内参数和历史内参标定数据为(K1,K2,…,Kn)，每个相机内参数对应的权重为(w1,w2,…,wn)，则加权内参数，即第二内参数的计算公式如下：
602、将第一内参数、第二内参数中重投影误差最小的一个确定为标定内参数。
本方案中，联合第一内参数和历史内参标定数据，利用加权计算的方式确定相机的第二内参数，再根据重投影误差将第一内参数和第二内参数中的一个确定为标定内参数，具体描述可以参考步骤523的相关描述，不再赘述。
进一步地，标定内参数满足内参有效性条件时，除了将第一内参数存入历史内参标定数据库中之外，还可以将第一内参数对应的第一权重，和/或，用于计算第一内参数的X个图像特征点的坐标对存入历史内参标定数据库中；这样，下一次标定在计算第二内参数时，可以直接从历史内参标定数据库中取出第一权重进行加权计算，而X个图像特征点的坐标对用于下一次标定时的重投影误差计算。
示例性地，第一权重为第一内参数对应的重投影误差的倒数；第二权重为历史内参标定数据对应的重投影误差的倒数。
参考图6b，图6b是本申请实施例提供的另一种相机内参标定方法的流程示意图。下面机器视觉应用以增强现实应用为例，而目标终端的标识以型号为例，而电子设备为云端服务器为例，对本申请实施例的标定方法进行具体说明。
步骤一：打开终端设备，进入增强现实应用；其中，增强现实应用包含本申请实施例的标定方法对应的算法。
步骤二：目标终端获取终端型号和一张在当前位置上利用终端的相机拍摄的图像等数据，目标终端向云端服务器上传终端型号和拍摄的图像等数据。
步骤三：根据当前位置的图像从高精度三维地图中完成图像检索，得到场景图像，并通过当前位置图像和场景图像之间的特征点匹配得到坐标对。
步骤四：根据终端型号从历史内参标定数据库中检索出该型号对应的多组历史有效相机内参数。
步骤五：基于步骤三中匹配得到的坐标对计算得到相机的内参数K1。
步骤六：结合步骤五中基于匹配得到的坐标对计算得到的相机内参数K1和步骤四从历史标定数据库中检索出的相同型号的多组历史有效相机内参数，联合计算得到相机内参数在计算时可以根据坐标对各自的权重进行加权计算，从而获得相机内参数
步骤七：比较使用步骤五和步骤六两种方式得到的相机内参数K1和计算两者的三维空间点的重投影误差大小，使用重投影误差较小的相机内参数作为标定结果输出。
步骤八：对步骤七得到的内参数标定结果进行有效性的判断。如果内参数标定结果满足有效性判断条件，则认为本次标定结果有效，并返回内参数标定成功的标志位和有效的相机内参数，同时把步骤五中得到的相机内参数K1和对应的权重保存至历史内参标定数据库。如果有任意一个有效性判断条件不成立，则认为当前内参数标定结果无效，并返回内参数标定失败的标志位和一个默认的相机内参数。
步骤九：终端接收云端标定算法返回的标志位和标定结果，终端使用返回的相机内参数用于增强现实应用。如果是内参数标定成功的标志位，则使用计算的标定结果作为本次增强现实应用使用过程中的相机内参数，并在本次应用使用过程中不再进行标定。如果是内参数标定失败的标志位，则先使用默认的相机内参数用于增强现实应用，不会阻塞应用的进程，并返回步骤二再次请求标定，直至标定成功。
示例性地，相机的内参数以包括X轴焦距fx、Y轴焦距fy、X轴上的像主点坐标cx以及Y轴上的像主点坐标cy为例，此时，内参数有效性条件可以为：
相机的标定内参数的结果需大于0，即fx、fy、cx和cy均大于0；
fx和fy的差值不超过a个像素，a的具体数值可以根据实际情况进行设置，不做特别限定，例如a为50；
cx与图像宽度的一半之间的差值不超过b个像素，b的具体数值可以根据实际情况进行设置，不做特别限定，例如b为15；
cy与图像高度的一半之间的差值不超过c个像素，c的具体数值可以根据实际情况进行设置，不做特别限定，例如c为15；
根据相机的内参数计算坐标对集合的重投影误差不超过d个像素，d的具体数值可以根据实际情况进行设置，不做特别限定，例如d为15。其中，根据相机的内参数和坐标对集合中的三维空间点坐标进行重投影处理，可以得到重投影的二维特征点坐标，根据重投影的二维特征点坐标和坐标对集合中的二维特征点坐标可以确定重投影误差。上述坐标对集合可以为上述X个图像特征点的坐标对以及目标终端的历史内参标定数据所组成的集合。
当相机的内参数满足上述所有的内参数有效性条件时，可以确认该相机的内参数为有效的内参数。反之，确认该相机的内参数为无效的内参数。
参考图7，图7是本申请实施例提供的一种相机内参标定装置的结构示意图。相机内参标定装置700包括获取模块701、确定模块702和标定模块703。
获取模块701，用于获取第一图像和目标终端的标识。第一图像为目标终端在第一位置处利用自身的相机拍摄得到的图像，第一位置位于预设区域内。
确定模块702，用于根据第一图像和预设区域对应的高精度三维地图，确定相机的第一内参标定数据；以及，根据目标终端的标识和历史内参标定数据库确定目标终端的历史内参标定数据。历史内参标定数据库包括一个或多个标识的终端的历史内参标定数据。
标定模块703，用于根据第一内参标定数据和目标终端的历史内参标定数据确定相机的标定内参数。
在一些可能的实施方式中，获取模块701，具体用于：
接收来自目标终端的第一图像以及目标终端的标识。
在一些可能的实施方式中，高精度三维地图包括多张场景图像、场景图像的二维特征点坐标、二维特征点的特征描述子以及二维特征点对应的三维空间点坐标。
上述确定模块702，具体用于：
从高精度三维地图中确定与第一图像的图像相似度最高的N张场景图像，N为大于零的整数。确定第一图像的M个图像特征点，M为大于零的整数。根据M个图像特征点的特征描述子和N张场景图像的二维特征点对应的特征描述子进行匹配，确定M个图像特征点中的X个图像特征点的坐标对。坐标对包括二维特征点坐标和三维空间点坐标，X小于或等于M。根据X个图像特征点的坐标对确定第一内参标定数据。
在一些可能的实施方式中，确定模块702在根据X个图像特征点的坐标对确定第一内参标定数据方面，具体用于：将X个图像特征点的坐标对确定为第一内参标定数据。
相应地，上述标定模块703，具体用于：
根据X个图像特征点的坐标对确定相机的第一内参数。根据X个图像特征点的坐标对、目标终端的历史内参标定数据确定相机的第二内参数。将第一内参数、第二内参数中重投影误差最小的一个确定为标定内参数。
在一些可能的实施方式中，确定模块702在根据X个图像特征点的坐标对确定第一内参标定数据方面，具体用于：根据X个图像特征点的坐标对确定相机的第一内参数。将第一内参数确定为第一内参标定数据。
相应地，上述标定模块703，具体用于：
根据第一内参数及其第一权重、目标终端对应的每个历史内参标定数据及其第二权重进行加权计算，以确定相机的第二内参数。将第一内参数、第二内参数中重投影误差最小的一个确定为标定内参数。
在一些可能的实施方式中，第一权重为第一内参数对应的重投影误差的倒数；第二权重为历史内参标定数据对应的重投影误差的倒数。
在一种可能的实施方式中，相机内参标定装置700还包括：
处理模块，用于标定内参数满足内参数有效性条件时，根据标定内参数更新目标终端的本地内参数，并将第一内参标定数据保存至历史内参标定数据库中。以及，标定内参数不满足内参数有效性条件时，根据预设内参数更新目标终端的本地内参数。
在一些可能的实施方式中，相机内参标定装置700还包括：
处理模块，用于标定内参数满足内参数有效性条件时，将第一内参标定数据保存至历史内参标定数据库中，以及将标定内参数发送给目标终端，以使目标终端根据标定内参数更新目标终端的本地内参数。以及，标定内参数不满足内参数有效性条件时，将预设内参数发送给目标终端，以使目标终端更新目标终端的本地内参数。
需要说明的是，相机内参标定装置700的具体执行过程以及相应的有益效果可以参考相机内参标定方法300的相关描述，不再赘述。
参考图8，图8是本申请实施例提供的一种相机内参标定设备的结构示意图。相机内参标定设备800包括存储器801、处理器802、通信接口804以及总线803。其中，存储器801、处理器802、通信接口804通过总线803实现彼此之间的通信连接。
存储器801可以是只读存储器(Read Only Memory，ROM)，静态存储设备，动态存储设备或者随机存取存储器(Random Access Memory，RAM)。存储器801可以存储程序，当存储器801中存储的程序被处理器802执行时，处理器802用于执行上述任意实施例所述的相机内参标定方法的各个步骤。
处理器802可以采用通用的中央处理器(Central Processing Unit，CPU)，微处理器，应用专用集成电路(Application Specific Integrated Circuit，ASIC)，图形处理器(graphics processing unit，GPU)或者一个或多个集成电路，用于执行相关程序，以实现上述任一实施例所述的相机内参标定装置中的单元所需执行的功能，或者执行上述任一实施例所述的相机内参标定方法。
处理器802还可以是一种集成电路芯片，具有信号的处理能力。在实现过程中，本申请任一实施例所述的相机内参标定方法的各个步骤可以通过处理器802中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器802还可以是通用处理器、数字信号处理器(Digital Signal Processing，DSP)、专用集成电路(ASIC)、现成可编程门阵列(Field Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请任一实施例所述的相机内参标定方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器801，处理器802读取存储器801中的信息，结合其硬件完成上述任一实施例所述的相机内参标定装置中包括的单元所需执行的功能，或者执行本申请任一实施例所述的相机内参标定方法。
通信接口804使用例如但不限于收发器一类的收发装置，来实现相机内参标定设备800与其他设备或通信网络之间的通信。例如，相机内参标定设备800可以通过通信接口804获取第一图像等。
总线803可包括在相机内参标定设备800各个部件(例如，存储器801、处理器802、通信接口804)之间传送信息的通路。
应注意，尽管图8所示的相机内参标定设备800仅仅示出了存储器、处理器、通信接口，但是在具体实现过程中，本领域的技术人员应当理解，相机内参标定设备800还包括实现正常运行所必须的其他器件。同时，根据具体需要，本领域的技术人员应当理解，相机内参标定设备800还可包括实现其他附加功能的硬件器件。此外，本领域的技术人员应当理解，相机内参标定设备800也可仅仅包括实现本申请实施例所必须的器件，而不必包括图8中所示的全部器件。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分， 仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以计算机程序产品的形式体现出来，该计算机程序产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
本申请实施例还提供一种芯片，所述芯片包括处理器与数据接口，所述处理器通过所述数据接口读取存储器上存储的指令，执行任一实施例所述的相机内参标定方法。
可选地，作为一种实现方式，所述芯片还可以包括存储器，所述存储器中存储有指令，所述处理器用于执行所述存储器上存储的指令，当所述指令被执行时，所述处理器用于执行任一实施例所述的相机内参标定方法。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (18)

  1. 一种相机内参标定方法，其特征在于，所述方法包括：
    获取第一图像和目标终端的标识；所述第一图像为所述目标终端在第一位置处利用自身的相机拍摄得到的图像，所述第一位置位于预设区域内；
    根据所述第一图像和所述预设区域对应的高精度三维地图，确定所述相机的第一内参标定数据；以及，根据所述目标终端的标识和历史内参标定数据库确定所述目标终端的历史内参标定数据，所述历史内参标定数据库包括一个或多个标识的终端的历史内参标定数据；
    根据所述第一内参标定数据和所述目标终端的历史内参标定数据确定所述相机的标定内参数。
  2. 根据权利要求1所述的方法，其特征在于，所述高精度三维地图包括多张场景图像、所述场景图像的二维特征点坐标、所述二维特征点的特征描述子以及所述二维特征点对应的三维空间点坐标；
    所述根据所述第一图像和所述预设区域对应的高精度三维地图，确定所述相机的第一内参标定数据，包括：
    从所述高精度三维地图中确定与所述第一图像的图像相似度最高的N张场景图像，所述N为大于零的整数；
    确定所述第一图像的M个图像特征点，所述M为大于零的整数；
    根据所述M个图像特征点的特征描述子和所述N张场景图像的二维特征点对应的特征描述子进行匹配，确定所述M个图像特征点中的X个图像特征点的坐标对；所述坐标对包括二维特征点坐标和三维空间点坐标，所述X小于或等于所述M；
    根据所述X个图像特征点的坐标对确定所述第一内参标定数据。
  3. 根据权利要求2所述的方法，其特征在于，所述根据所述X个图像特征点的坐标对确定所述第一内参标定数据，包括：
    将所述X个图像特征点的坐标对确定为所述第一内参标定数据；
    所述根据所述第一内参标定数据和所述目标终端的历史内参标定数据确定所述相机的标定内参数，包括：
    根据所述X个图像特征点的坐标对确定所述相机的第一内参数；
    根据所述X个图像特征点的坐标对、所述目标终端的历史内参标定数据确定所述相机的第二内参数；
    将所述第一内参数、所述第二内参数中重投影误差最小的一个确定为所述标定内参数。
  4. 根据权利要求2所述的方法，其特征在于，所述根据所述X个图像特征点的坐标对确定所述第一内参标定数据，包括：
    根据所述X个图像特征点的坐标对确定所述相机的第一内参数；
    将所述第一内参数确定为所述第一内参标定数据；
    所述根据所述第一内参标定数据和所述目标终端的历史内参标定数据确定所述相机的标定内参数，包括：
    根据所述第一内参数及其第一权重、所述目标终端对应的每个所述历史内参标定数据及 其第二权重进行加权计算，以确定所述相机的第二内参数；
    将所述第一内参数、所述第二内参数中重投影误差最小的一个确定为所述标定内参数。
  5. 根据权利要求4所述的方法，其特征在于，所述第一权重为所述第一内参数对应的重投影误差的倒数；所述第二权重为所述历史内参标定数据对应的重投影误差的倒数。
  6. 根据权利要求1至5任一项所述的方法，其特征在于，所述方法还包括：
    所述标定内参数满足内参数有效性条件时，根据所述标定内参数更新所述目标终端的本地内参数，并将所述第一内参标定数据保存至所述历史内参标定数据库中；
    所述标定内参数不满足所述内参数有效性条件时，根据预设内参数更新所述目标终端的本地内参数。
  7. 根据权利要求1至5任一项所述的方法，其特征在于，所述方法还包括：
    所述标定内参数满足内参数有效性条件时，将所述第一内参标定数据保存至所述历史内参标定数据库中，以及将所述标定内参数发送给所述目标终端，以使所述目标终端根据所述标定内参数更新所述目标终端的本地内参数；
    所述标定内参数不满足所述内参数有效性条件时，将预设内参数发送给所述目标终端，以使所述目标终端更新所述目标终端的本地内参数。
  8. 根据权利要求1至5、7中任一项所述的方法，其特征在于，所述获取第一图像和目标终端的标识，包括：
    接收来自所述目标终端的所述第一图像以及所述目标终端的标识。
  9. 一种相机内参标定装置，其特征在于，所述装置包括：
    获取模块，用于获取第一图像和目标终端的标识；所述第一图像为所述目标终端在第一位置处利用自身的相机拍摄得到的图像，所述第一位置位于预设区域内；
    确定模块，用于根据所述第一图像和所述预设区域对应的高精度三维地图，确定所述相机的第一内参标定数据；以及，根据所述目标终端的标识和历史内参标定数据库确定所述目标终端的历史内参标定数据，所述历史内参标定数据库包括一个或多个标识的终端的历史内参标定数据；
    标定模块，用于根据所述第一内参标定数据和所述目标终端的历史内参标定数据确定所述相机的标定内参数。
  10. 根据权利要求9所述的装置，其特征在于，所述高精度三维地图包括多张场景图像、所述场景图像的二维特征点坐标、所述二维特征点的特征描述子以及所述二维特征点对应的三维空间点坐标；
    所述确定模块，具体用于：
    从所述高精度三维地图中确定与所述第一图像的图像相似度最高的N张场景图像，所述N为大于零的整数；
    确定所述第一图像的M个图像特征点，所述M为大于零的整数；
    根据所述M个图像特征点的特征描述子和所述N张场景图像的二维特征点对应的特征描 述子进行匹配，确定所述M个图像特征点中的X个图像特征点的坐标对；所述坐标对包括二维特征点坐标和三维空间点坐标，所述X小于或等于所述M；
    根据所述X个图像特征点的坐标对确定所述第一内参标定数据。
  11. 根据权利要求10所述的装置，其特征在于，所述确定模块在根据所述X个图像特征点的坐标对确定所述第一内参标定数据方面，具体用于：
    将所述X个图像特征点的坐标对确定为所述第一内参标定数据；
    所述标定模块，具体用于：
    根据所述X个图像特征点的坐标对确定所述相机的第一内参数；
    根据所述X个图像特征点的坐标对、所述目标终端的历史内参标定数据确定所述相机的第二内参数；
    将所述第一内参数、所述第二内参数中重投影误差最小的一个确定为所述标定内参数。
  12. 根据权利要求10所述的装置，其特征在于，所述确定模块在根据所述X个图像特征点的坐标对确定所述第一内参标定数据方面，具体用于：
    根据所述X个图像特征点的坐标对确定所述相机的第一内参数；
    将所述第一内参数确定为所述第一内参标定数据；
    所述标定模块，具体用于：
    根据所述第一内参数及其第一权重、所述目标终端对应的每个所述历史内参标定数据及其第二权重进行加权计算，以确定所述相机的第二内参数；
    将所述第一内参数、所述第二内参数中重投影误差最小的一个确定为所述标定内参数。
  13. 根据权利要求12所述的装置，其特征在于，所述第一权重为所述第一内参数对应的重投影误差的倒数；所述第二权重为所述历史内参标定数据对应的重投影误差的倒数。
  14. 根据权利要求9至13任一项所述的装置，其特征在于，所述装置还包括：
    处理模块，用于所述标定内参数满足内参数有效性条件时，根据所述标定内参数更新所述目标终端的本地内参数，并将所述第一内参标定数据保存至所述历史内参标定数据库中；
    以及，所述标定内参数不满足所述内参数有效性条件时，根据预设内参数更新所述目标终端的本地内参数。
  15. 根据权利要求9至13任一项所述的装置，其特征在于，所述装置还包括：
    处理模块，用于所述标定内参数满足内参数有效性条件时，将所述第一内参标定数据保存至所述历史内参标定数据库中，以及将所述标定内参数发送给所述目标终端，以使所述目标终端根据所述标定内参数更新所述目标终端的本地内参数；
    以及，所述标定内参数不满足所述内参数有效性条件时，将预设内参数发送给所述目标终端，以使所述目标终端更新所述目标终端的本地内参数。
  16. 根据权利要求9至13、15中任一项所述的装置，其特征在于，所述获取模块，具体用于：
    接收来自所述目标终端的所述第一图像以及所述目标终端的标识。
  17. 一种相机内参标定设备，其特征在于，包括处理器和存储器，其中，处理器和存储器相连，其中，存储器用于存储程序代码，处理器用于调用程序代码，以执行如权利要求1至8任一项所述的相机内参标定方法。
  18. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质存储有计算机程序，所述计算机程序被处理器执行以实现如权利要求1至8任一项所述的相机内参标定方法。
PCT/CN2023/097798 2022-06-13 2023-06-01 相机内参标定方法及相关设备 WO2023241372A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210660950.1 2022-06-13
CN202210660950.1A CN117274392A (zh) 2022-06-13 2022-06-13 相机内参标定方法及相关设备

Publications (1)

Publication Number Publication Date
WO2023241372A1 true WO2023241372A1 (zh) 2023-12-21

Family

ID=89192102

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/097798 WO2023241372A1 (zh) 2022-06-13 2023-06-01 相机内参标定方法及相关设备

Country Status (2)

Country Link
CN (1) CN117274392A (zh)
WO (1) WO2023241372A1 (zh)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110766760A (zh) * 2019-10-21 2020-02-07 北京百度网讯科技有限公司 用于相机标定的方法、装置、设备和存储介质
CN111612853A (zh) * 2020-05-25 2020-09-01 追创科技(苏州)有限公司 相机参数标定方法及装置
CN112771577A (zh) * 2020-05-28 2021-05-07 深圳市大疆创新科技有限公司 相机参数确定方法、装置和可读存储介质
CN112991459A (zh) * 2021-03-09 2021-06-18 北京百度网讯科技有限公司 一种相机标定方法、装置、设备以及存储介质
CN113112553A (zh) * 2021-05-26 2021-07-13 北京三快在线科技有限公司 双目相机的参数标定方法、装置、电子设备及存储介质
CN113240806A (zh) * 2021-05-13 2021-08-10 深圳市慧鲤科技有限公司 信息处理方法、装置、电子设备及存储介质
WO2022095537A1 (zh) * 2020-11-09 2022-05-12 Oppo广东移动通信有限公司 虚拟对象显示方法及装置、存储介质和电子设备
CN114549652A (zh) * 2022-01-13 2022-05-27 湖南视比特机器人有限公司 一种相机标定方法、装置、设备及计算机可读介质

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110766760A (zh) * 2019-10-21 2020-02-07 北京百度网讯科技有限公司 用于相机标定的方法、装置、设备和存储介质
CN111612853A (zh) * 2020-05-25 2020-09-01 追创科技(苏州)有限公司 相机参数标定方法及装置
CN112771577A (zh) * 2020-05-28 2021-05-07 深圳市大疆创新科技有限公司 相机参数确定方法、装置和可读存储介质
WO2022095537A1 (zh) * 2020-11-09 2022-05-12 Oppo广东移动通信有限公司 虚拟对象显示方法及装置、存储介质和电子设备
CN112991459A (zh) * 2021-03-09 2021-06-18 北京百度网讯科技有限公司 一种相机标定方法、装置、设备以及存储介质
CN113240806A (zh) * 2021-05-13 2021-08-10 深圳市慧鲤科技有限公司 信息处理方法、装置、电子设备及存储介质
CN113112553A (zh) * 2021-05-26 2021-07-13 北京三快在线科技有限公司 双目相机的参数标定方法、装置、电子设备及存储介质
CN114549652A (zh) * 2022-01-13 2022-05-27 湖南视比特机器人有限公司 一种相机标定方法、装置、设备及计算机可读介质

Also Published As

Publication number Publication date
CN117274392A (zh) 2023-12-22

Similar Documents

Publication Publication Date Title
CN109584248B (zh) 基于特征融合和稠密连接网络的红外面目标实例分割方法
CN109902548B (zh) 一种对象属性识别方法、装置、计算设备及系统
US11232286B2 (en) Method and apparatus for generating face rotation image
WO2021175050A1 (zh) 三维重建方法和三维重建装置
CN108108764B (zh) 一种基于随机森林的视觉slam回环检测方法
CN110222718B (zh) 图像处理的方法及装置
CN112990010B (zh) 点云数据处理方法、装置、计算机设备和存储介质
US20220148291A1 (en) Image classification method and apparatus, and image classification model training method and apparatus
CN111931764B (zh) 一种目标检测方法、目标检测框架及相关设备
CN112990228B (zh) 图像特征匹配方法和相关装置、设备及存储介质
US20220292728A1 (en) Point cloud data processing method and device, computer device, and storage medium
JP7439153B2 (ja) 全方位場所認識のためのリフトされたセマンティックグラフ埋め込み
CN110222572A (zh) 跟踪方法、装置、电子设备及存储介质
WO2022165722A1 (zh) 单目深度估计方法、装置及设备
CN112489099A (zh) 点云配准方法、装置、存储介质及电子设备
CN114219855A (zh) 点云法向量的估计方法、装置、计算机设备和存储介质
CN112258565B (zh) 图像处理方法以及装置
CN115222896A (zh) 三维重建方法、装置、电子设备及计算机可读存储介质
WO2022247126A1 (zh) 视觉定位方法、装置、设备、介质及程序
CN112329662B (zh) 基于无监督学习的多视角显著性估计方法
CN111353325A (zh) 关键点检测模型训练方法及装置
CN112668662A (zh) 基于改进YOLOv3网络的野外山林环境目标检测方法
CN113065575A (zh) 一种图像处理方法及相关装置
CN116758419A (zh) 针对遥感图像的多尺度目标检测方法、装置和设备
WO2023241372A1 (zh) 相机内参标定方法及相关设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23822941

Country of ref document: EP

Kind code of ref document: A1