CN113744352A - Visual space calibration method, device and storage medium - Google Patents

Visual space calibration method, device and storage medium Download PDF

Info

Publication number
CN113744352A
CN113744352A CN202111071931.7A CN202111071931A CN113744352A CN 113744352 A CN113744352 A CN 113744352A CN 202111071931 A CN202111071931 A CN 202111071931A CN 113744352 A CN113744352 A CN 113744352A
Authority
CN
China
Prior art keywords
target object
oblique
image
calibration
vertical
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202111071931.7A
Other languages
Chinese (zh)
Other versions
CN113744352B (en
Inventor
陆冬云
石宝庆
熊磊
耿頔
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Guanhai Technology Development Co ltd
Original Assignee
Beijing Guanhai Technology Development Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Guanhai Technology Development Co ltd filed Critical Beijing Guanhai Technology Development Co ltd
Priority to CN202111071931.7A priority Critical patent/CN113744352B/en
Publication of CN113744352A publication Critical patent/CN113744352A/en
Application granted granted Critical
Publication of CN113744352B publication Critical patent/CN113744352B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Analysis (AREA)

Abstract

The application relates to a visual space calibration method, a device and a storage medium, wherein the visual space calibration method comprises the following steps: obtaining a vertical overlook image and an oblique overlook image of a calibration training space which are acquired at the same time; identifying a spatial position parameter and an orientation parameter of each target object in the vertical top view image and an oblique top view position parameter of each target object in the oblique top view image from the vertical top view image and the oblique top view image; constructing a sample mapping set of the oblique overlooking position parameter of each target object and the spatial position parameter and the orientation parameter of each target object respectively; and training the sample mapping set to obtain a calibration model. In practical application, the calibration model is obtained through training, the calibration of the object to be calibrated in the space projection coordinate is realized accurately and effectively by data driving only according to the oblique overlooking monitoring image, and the problem of the existing space calibration mode is effectively avoided.

Description

Visual space calibration method, device and storage medium
Technical Field
The present application relates to the field of spatial calibration technologies, and in particular, to a method, an apparatus, and a storage medium for calibrating a visual space.
Background
Along with the intelligent popularization of animal husbandry, the judgment of accurate spatial relationship can provide accurate control signals for equipment, for example, the spatial relationship between livestock (such as cows) and the equipment (such as fans) is judged, which is of great significance for improving the service quality of the equipment, saving energy and reducing emission. Based on a visual image analysis technology, the real space relationship between the livestock and the equipment is calculated by analyzing the positions of the livestock and the equipment in a monitoring picture of a monitoring camera to obtain the coordinates of the livestock in a real physical space, and the establishment of the coordinate mapping relationship is called space calibration.
At present, mapping is generally established on the basis of a pixel level, and a mapping relation is established according to camera attitude estimation, lens distortion correction and the like to perform spatial calibration. The existing space calibration has the following problems in practical application:
1) the camera attitude estimation needs to adopt a standard calibration plate to implement calibration, and is difficult to implement in a field working environment;
2) the lens distortion correction is based on the characteristics of an ideal lens and is corrected by adopting theoretical methods such as affine and the like, but the distortion of the actual lens is not uniform, and the distortion correction result is not ideal;
3) the calibration is based on pixels and is suitable for processing a plane object, the actual working object is a three-dimensional livestock, and due to the lack of height information, the coordinate estimation of the livestock in a physical space is difficult to accurately realize in practical application.
In view of the above problems in the prior art, no effective solution has been proposed.
Disclosure of Invention
To solve the above technical problem or at least partially solve the above technical problem, the present application provides a visual space calibration method, device and storage medium.
In a first aspect, the present application provides a visual space calibration method, including:
obtaining a vertical overlook image and an oblique overlook image of a calibration training space which are acquired at the same time;
identifying a spatial position parameter and an orientation parameter of each target object in the vertical top view image and an oblique top view position parameter of each target object in the oblique top view image from the vertical top view image and the oblique top view image;
constructing a sample mapping set of the oblique overlooking position parameter of each target object and the spatial position parameter and the orientation parameter of each target object respectively;
and training the sample mapping set to obtain a calibration model, wherein the calibration model is used for a physical space to be calibrated, and the spatial position parameter and the orientation parameter of each object to be calibrated are determined according to the oblique overlooking position parameter of each object to be calibrated.
Optionally, before obtaining the vertical overhead image and the oblique overhead image of the calibration training space acquired at the same time, the method includes:
respectively acquiring a vertical overlook image and an oblique overlook image of the calibration training space at the same time through a vertical overlook camera and an oblique overlook camera which are arranged in the calibration training space; wherein, the calibration training space is provided with a neck cangue with the height of 1.5 meters and the length of 8 meters; the deployment parameters of the vertical overlook camera are as follows: vertically overlooking the ground plane of the calibration training space, wherein the height of the ground plane is 5 meters, and the horizontal distance between the ground plane and the neck cangue is 1.5 meters; the deployment parameters of the oblique overlooking camera are as follows: the included angle between the horizontal angle and the horizontal line is 48-55 degrees, the horizontal line faces the oblique lower part of the neck cangue, the height of the horizontal line is 4 meters, and the horizontal distance from the horizontal line to the neck cangue is 6 meters.
Optionally, the visual space calibration method further includes:
acquiring a monitoring image from a monitoring camera in a physical space to be calibrated; the monitoring camera is set according to the deployment parameters of the oblique overlooking camera;
identifying the oblique overlooking position parameters of each object to be calibrated in the monitoring image from the monitoring image;
and inputting the oblique overlooking position parameters of each object to be calibrated into the calibration model to obtain the spatial position parameters and the orientation parameters of each object to be calibrated in the physical space to be calibrated.
Optionally, the identifying, from the vertical top view image and the oblique top view image, a spatial position parameter and an orientation parameter of each target object in the vertical top view image and an oblique top view position parameter of each target object in the oblique top view image includes:
respectively identifying a first identification area of each target object in the vertical overlooking image and a second identification area of each target object in the oblique overlooking image from the vertical overlooking image and the oblique overlooking image by adopting a preset convolutional neural network;
according to the first identification area of each target object, determining a spatial position parameter and an orientation parameter of each target object in the vertical overlook image;
determining an oblique overlooking position parameter of each target object in the oblique overlooking image according to the second identification area of each target object;
optionally, the constructing a sample mapping set of the oblique top view position parameter of each target object and the spatial position parameter and the orientation parameter of each target object includes:
establishing a corresponding relation between a first identification area and a second identification area of each target object;
and according to the corresponding relation, constructing a sample mapping set of the oblique overlooking position parameter of each target object and the spatial position parameter and the orientation parameter of each target object respectively.
Optionally, the following steps are adopted to identify the orientation parameter of each target object in the vertical overhead view image:
identifying the head of each target object from the first identification area of each target object, and obtaining a third identification area of the head of each target object;
acquiring a maximum circumscribed ellipse surrounding each target object;
determining orientation parameters of each target object in the vertical overlook image according to the maximum circumscribed ellipse of each target object and the third identification area of each target object; the first identification area, the second identification area and the third identification area are identification frames; the spatial position parameter and the oblique overlooking position parameter are composed of an origin coordinate, a width and a height of the identification frame.
Optionally, the determining, according to the maximum circumscribed ellipse of each target object and the third identification area of each target object, an orientation parameter of each target object in the vertical overhead view image further includes:
and according to a preset orientation approaching principle, carrying out normalization processing on orientation parameters of each target object in the vertical overlooking image.
Optionally, the establishing a correspondence between the first identification area and the second identification area of each target object includes:
determining a first center distance of each target object according to the distance between the center of the first identification area of each target object and the origin of the vertical overhead image on the vertical overhead image;
determining a second center distance of each target object according to the distance between the center of the second identification area of each target object and the origin of the oblique overhead view image on the oblique overhead view image;
respectively sequencing the first center distance and the second center distance of each target object according to the distance;
determining the difference value of the first center distance and the second center distance of the same sequencing position;
if each difference value is smaller than a preset threshold value, establishing a corresponding relation between a first identification area and a second identification area of each target object according to the sequence; otherwise, the vertical overlooking image and the oblique overlooking image of the calibration training space acquired at the same time are obtained again.
Optionally, the training the sample mapping set to obtain a calibration model includes:
establishing a BP and SVM mixed model, establishing oblique overlook position parameter and spatial position parameter and orientation parameter mapping by using the BP model, establishing oblique overlook position parameter and orientation parameter mapping by using the SVM model, and optimizing the BP model and the searched SVM model parameter by using the minimum total error of the mixed model as an optimization object through a PSO algorithm to obtain the calibration model.
In a second aspect, the present application provides a visual space calibration apparatus comprising a memory, a processor, and a computer program stored on the memory and executable on the processor;
the computer program when executed by the processor performs the steps of the visual space calibration method of any one of the above.
In a third aspect, the present application provides a computer-readable storage medium having stored thereon a visual space calibration program, which when executed by a processor, implements the steps of the visual space calibration method as described in any one of the above.
Compared with the prior art, the technical scheme provided by the embodiment of the application has the following advantages:
in each embodiment of the application, a spatial position parameter and an orientation parameter of each target object in a vertical overhead view image and an oblique overhead view position parameter of each target object in the oblique overhead view image are identified from the vertical overhead view image and the oblique overhead view image of a calibration training space acquired at the same time; therefore, a sample mapping set of the oblique overlooking position parameters of each target object and the spatial position parameters and orientation parameters of each target object can be constructed, the sample mapping set is trained to obtain a calibration model, and calibration of the object to be calibrated in the spatial projection coordinate can be accurately and effectively realized only according to the oblique overlooking monitoring image in practical application.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
FIG. 1 is a flow chart of an alternative visual space calibration method provided in various embodiments of the present application;
fig. 2 is a schematic diagram illustrating a relative position relationship among a cow, a neck cangue and two cameras according to various embodiments of the present application;
FIG. 3 is a schematic diagram of a physical scenario in accordance with various embodiments of the present application;
fig. 4 is a schematic view of a picture seen by a camera a according to various embodiments of the present application;
5a, 5b, 5c, 5d are schematic diagrams illustrating normalization of the first identification region, the third identification region and the maximum circumscribed ellipse, orientation, respectively, according to various embodiments of the present application;
fig. 6 is a schematic view of a picture seen by a camera B provided in various embodiments of the present application;
fig. 7 is a schematic diagram of a CNN detection network and BBX output according to various embodiments of the present disclosure;
fig. 8a and 8b are schematic diagrams of BBX and the central point and the ranking distance of BBX corresponding to the same cow in the synchronous A, B picture provided in the embodiments of the present application;
fig. 9 is a schematic structural diagram of a BP neural network provided in various embodiments of the present application;
FIG. 10 is a schematic diagram of SVM based data classification provided by various embodiments of the present application;
FIG. 11 is a flow chart of another alternative visual space calibration method provided in various embodiments of the present application;
fig. 12 is a flowchart of an application of a calibration model according to various embodiments of the present application.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In the following description, suffixes such as "module", "component", or "unit" used to denote elements are used only for facilitating the explanation of the present invention, and have no specific meaning in itself. Thus, "module", "component" or "unit" may be used mixedly.
Example one
The embodiment of the invention provides a visual space calibration method, which comprises the following steps of:
s101, acquiring a vertical overlook image and an oblique overlook image of a calibration training space which are acquired at the same time;
s102, identifying a spatial position parameter and an orientation parameter of each target object in the vertical overhead view image and an oblique overhead view position parameter of each target object in the oblique overhead view image from the vertical overhead view image and the oblique overhead view image;
s103, constructing a sample mapping set of the oblique overlooking position parameters of each target object and the spatial position parameters and orientation parameters of each target object respectively;
and S104, training the sample mapping set to obtain a calibration model, wherein the calibration model is used for determining the spatial position parameter and the orientation parameter of each object to be calibrated in the physical space to be calibrated according to the oblique overlooking position parameter of each object to be calibrated.
The target object in the embodiment of the invention can be a livestock, such as a cow, the calibration training space refers to a space place for training a calibration model, and can be a livestock farm; the physical space to be calibrated refers to a livestock farm needing to be calibrated, and the object to be calibrated refers to livestock in the livestock farm needing to be calibrated.
According to the embodiment of the invention, the spatial position parameter and orientation parameter of each target object in the vertical overlook image and the oblique overlook image of the calibration training space acquired at the same time are identified, and the oblique overlook position parameter of each target object in the oblique overlook image is identified; therefore, a sample mapping set of the oblique overlooking position parameters of each target object and the spatial position parameters and orientation parameters of each target object can be constructed, the sample mapping set is trained to obtain a calibration model, calibration of the object to be calibrated in a spatial projection coordinate can be accurately and effectively realized only according to an oblique overlooking monitoring image in practical application through data driving, and the problems of the existing spatial calibration mode are effectively avoided.
The embodiment of the invention can be divided into applications of defining a physical scene, acquiring an image, establishing space calibration and calibrating in a specific implementation process.
Defining a physical scene in some embodiments may include: respectively acquiring a vertical overlook image and an oblique overlook image of the calibration training space at the same time through a vertical overlook camera and an oblique overlook camera which are arranged in the calibration training space; wherein, the calibration training space is provided with a neck cangue with the height of 1.5 meters and the length of 8 meters; the deployment parameters of the vertical overlook camera are as follows: vertically overlooking the ground plane of the calibration training space, wherein the height of the ground plane is 5 meters, and the horizontal distance between the ground plane and the neck cangue is 1.5 meters; the deployment parameters of the oblique overlooking camera are as follows: the included angle between the horizontal angle and the horizontal line is 48-55 degrees, the horizontal line faces the oblique lower part of the neck cangue, the height of the horizontal line is 4 meters, and the horizontal distance from the horizontal line to the neck cangue is 6 meters.
That is, before performing the visual space calibration, a physical scene may be defined, and the defined physical scene is suitable for calibrating the training space and the physical space to be calibrated.
The cow farm is taken as an example below. A typical physical scenario designed is a dairy farm. As shown in fig. 2, there are several cows, neck cangues and 2 monitoring cameras, including a vertical overhead camera and an oblique overhead camera. Typical scene parameters are: the height of the neck cangue is 1.5 meters, and the length is 8 meters; a camera/vertical overhead camera: the vertical 90-degree overlooking is carried out, the installation height is 5 meters, and the distance from the neck cangue is 1.5 meters; the B camera/oblique overlook camera faces the neck cangue, overlooks obliquely downwards and forms an included angle of 48-55 degrees with the horizontal, typically 52 degrees, the installation height is 4 meters, and the distance from the neck cangue is 6 meters. Fig. 3 shows a picture seen by the B camera.
In some embodiments, the image acquisition mainly includes A, B acquisition of a head monitoring image, and a preset convolutional neural network can be used to respectively identify a first identification area of each target object in the vertical top-view image and a second identification area in the oblique top-view image from the vertical top-view image and the oblique top-view image; according to the first identification area of each target object, determining a spatial position parameter and an orientation parameter of each target object in the vertical overlook image; and determining the oblique overlooking position parameter of each target object in the oblique overlooking image according to the second identification area of each target object.
In detail, for the acquisition of a head monitoring image:
as shown in fig. 4, the a-camera picture contains spatial position and orientation information of the cow. In this embodiment, in order to obtain the projection coordinates, a Convolutional Neural Network (CNN) is adopted, and the detection model is mobilesds or YoloV3, so as to perform object detection on the cows.
As shown in fig. 5a, the obtained recognition frame (BBX, described later) contains information of coordinates (X, Y) of the upper left corner of the recognition frame and its width and height (W, H). This information is recorded as A (X, Y, W, H). The origin of the coordinates can be arbitrarily selected, and the upper left corner of the screen is generally selected as the origin.
The cow orientation identification is optionally processed as follows: identifying the head of each target object from the first identification area of each target object, and obtaining a third identification area of the head of each target object; acquiring a maximum circumscribed ellipse surrounding each target object; determining orientation parameters of each target object in the vertical overlook image according to the maximum circumscribed ellipse of each target object and the third identification area of each target object; the first identification area, the second identification area and the third identification area are identification frames; the spatial position parameter and the oblique overlooking position parameter are composed of an origin coordinate, a width and a height of the identification frame. The method specifically comprises the following steps:
1) and cutting out a sub-image corresponding to the BBX from the picture of the camera A according to the cattle body BBX (the first identification area).
2) As shown in fig. 5 b. And performing CNN recognition on the subgraph again to obtain a cattle head BBX (third recognition area), wherein the cattle head BBX is smaller and is usually positioned in a cattle body BBX, and the relation between the cattle head BBX and the cattle body BBX reflects the overall orientation.
3) As shown in fig. 5 b. On the subgraph, an OpenCV image processing method is adopted to obtain a maximum circumscribed ellipse surrounding the cattle body.
4) As shown in fig. 5 c. According to the long axis of the ellipse and the position of the cow head, the orientation of the cow can be accurately determined.
5) As shown in fig. 5 d. The orientation of the cattle with any angle is normalized into 8 orientations according to the principle of approximation, and the orientations are respectively defined as D1-D8. That is, the orientation parameters of each target object in the vertical overhead view image need to be normalized according to a preset orientation approach principle.
B, collecting monitoring pictures of a camera:
as shown in fig. 6, the B camera observes the cow from the side and the rear, and the picture of the B camera is in the oblique direction and forms a certain included angle with the ground. Since the cow is a three-dimensional object with a certain height, the height of the cow is difficult to estimate, and therefore, it is difficult to estimate the projection of the cow on the ground from the B picture perspective only. It should be noted that the projection of the cow on the ground is BBX in the a frame.
Based on the same CNN algorithm, object detection is performed on the B picture, and BBX (second identification area) information of B, that is, B (X, Y, W, H), can be obtained. But it is difficult to obtain information on the orientation of the cow from the B picture alone.
The principle of CNN detection network and BBX output is shown in fig. 7, where a CNN skeleton (backbone) part is responsible for extracting original image features, an RPN network is responsible for predicting the position of a background cow in a picture, an FC full connection layer and a softmax classification layer are responsible for predicting the category of each BBX, and a bboxreg layer is responsible for outputting the coordinates of each BBX.
The establishment of spatial calibration in some embodiments may include: establishing a corresponding relation between a first identification area and a second identification area of each target object; according to the corresponding relation, a sample mapping set of the oblique overlooking position parameter of each target object and the spatial position parameter and the orientation parameter of each target object is established; and training the sample mapping set to obtain a calibration model.
According to the definition of the space calibration of the present invention, the present invention aims to obtain the information of the real physical space of the cow through the information of the cow on the B picture, that is, the BBX and the orientation D of a through the BBX of B.
For this purpose, when the camera a and the camera B are designed to perform image acquisition, time synchronization needs to be accurately guaranteed, that is, the time of the camera A, B is accurately consistent through NTP time service, and the time synchronization precision required in this example is within 1 second.
On the premise of time synchronization, as shown in fig. 8a, the BBX of the a picture and the BBX of the B picture have an accurate corresponding relationship for the same cow, so that the relationship between the two BBXs corresponding to the same cow is accurately established by an automated image processing method, for example:
determining a first center distance of each target object according to the distance between the center of the first identification area of each target object and the origin of the vertical overhead image on the vertical overhead image; determining a second center distance of each target object according to the distance between the center of the second identification area of each target object and the origin of the oblique overhead view image on the oblique overhead view image; respectively sequencing the first center distance and the second center distance of each target object according to the distance; determining the difference value of the first center distance and the second center distance of the same sequencing position; if each difference value is smaller than a preset threshold value, establishing a corresponding relation between a first identification area and a second identification area of each target object according to the sequence; otherwise, the vertical overlooking image and the oblique overlooking image of the calibration training space acquired at the same time are obtained again.
The method comprises the following specific steps:
1) as shown in fig. 8 b. Based on BBX of the A, B frame, coordinates of the center of BBX are obtained.
2) Calculating the distance L from the BBX center to the origin, L in A, B pictures is called L respectivelyA、LB；
Figure BDA0003260668050000101
3) At the same time, one image of the camera A and one image of the camera B can be acquired. Through a traversal algorithm, the corresponding relation between the BBX in the picture A and the BBX in the picture B is obtained, and the rule is as follows:
calculate L for all BBXs of A frameAAnd obtaining the sorted sequence according to the sequence from small to large
Figure BDA0003260668050000102
Calculate L for all BBXs of A frameBAnd obtaining the sorted sequence according to the sequence from small to large
Figure BDA0003260668050000111
And for the sequencing sequence, calculating the difference value of two L at the same sequence position, and if all the difference values are within the threshold value, successfully matching:
Figure BDA0003260668050000112
where n is the number of BBXs and δ is the threshold.
If there is a difference exceeding the threshold, then the present group A, B of images is discarded and another group A, B of images is selected to continue;
if the number of BBX n of A, B images is not equal, then the present set of A, B images is discarded and another set of A, B images is chosen to continue.
4) By the algorithm of the step 3, the corresponding relation of the A, B picture BBX is obtained, and after the BBX corresponding relation of a sufficient amount is completed, a sample mapping set can be constructed:
Bi(X，Y，W，H)→Ai(X，Y，W，H，D)
wherein A isi、BiBBX information representing A, B frames, i being 1 to N, N representing the total number of BBX pairs in the mapping set, and the meanings of X, Y, W, H, and DAs defined above.
In some embodiments the establishing of the association comprises:
generally, the BP artificial neural network has good precision on numerical fitting, and the SVM has better accuracy on the aspect of classification. In the embodiment, two algorithms are mixed, and in order to achieve the overall optimal correlation precision, the particle swarm PSO algorithm is used for optimizing the model parameters based on the mixed model error term as the target function. The specific method comprises the following steps:
mapping BBX oblique overlooking position parameters of the B to BBX space position parameters of the A is realized by using a standard BP network algorithm; and mapping of the BBX coordinate information of the B to the orientation in the BBX of the A is realized by adopting an SVM algorithm.
Fig. 9 shows the structure of the BP neural network used in this example, in which 4 input nodes correspond to BBX information [ X, Y, W, H ] (B, i) of the B screen, 5 output nodes correspond to BBX information [ X, Y, W, H, D ] (a, i) of the a screen. The number of nodes of the hidden layer is 10-20; and the difference between the predicted output and the actual value is minimized by adjusting the weight of each layer of the BP neural network.
Wherein the input layer to hidden layer weight is defined as ωijThe weight from hidden layer to output layer is defined as ωjk. The excitation function is chosen to be a Sigmoid function. The final output of the BP network is:
Ok＝f(ωij，ωjk，[X，Y，W，H]b)
the final output error is defined as:
Figure BDA0003260668050000121
wherein, YkTo desired output, OkIs the prediction output. m is the number of samples of the BBX pair.
The SVM is a two-class classifier model, and the basic model of the SVM is a linear classifier with the maximum interval defined on a feature space. As shown in fig. 10, when data is classified, it looks for a hyperplane that can implement class two classification.
In the case where the sample point is n-dimensional, one point may be defined as X ═ X (X)1，x2，...xn) Hyperplane parameter wT＝(w1，w2，...ωn) Also n-dimensional, the hyperplane is defined as:
wTx-b＝0
ωTis the hyperplane parameter that needs to be optimized, which is an n-dimensional vector. The distance of the sample point X from the hyperplane is defined as:
Figure BDA0003260668050000122
when the distances from all the points to the hyperplane are maximum, the optimal hyperplane parameter w is obtainedT。
The sample of the example is 4-dimensional, the classification error is taken as an optimization object, and the error term is defined as:
Esvm＝(f1(wT，X)-f2(D))2
wherein f is1For SVM classification output, f2(D) Is the known orientation of the a picture BBX.
In some embodiments, the training the sample mapping set to obtain a calibration model includes:
establishing a BP and SVM mixed model, establishing oblique overlook position parameter and spatial position parameter and orientation parameter mapping by using the BP model, establishing oblique overlook position parameter and orientation parameter mapping by using the SVM model, and optimizing the BP model and the searched SVM model parameter by using the minimum total error of the mixed model as an optimization object through a PSO algorithm to obtain the calibration model.
The PSO realizes an optimization algorithm, any parameter group can be adjusted based on the optimization of the PSO on the mixed model, and parameters are adjusted according to the error change condition obtained by each model calculation. In order to realize a mixed model algorithm of BP and SVM, firstly, parameter groups of two models are constructed:
P＝(wij，ωjk，ωT)
the parameter group P is formed by all adjustable parameters of the two models. Then, the total error of the output of the mixed model is constructed:
Etotal＝Ebp+Esvm
through the PSO optimization algorithm, the parameters of the two models can be adjusted simultaneously, and coordinate information and orientation information with high accuracy can be obtained. When the optimization is completed, an independent BP model M can be obtainedbpAnd SVM model MsvmDefining the mixed model as M:
M＝[Mbp，Msvm]
when the calibration model M is applied, BBX information of B is input, and BBX information of A and orientation information of A are output.
In some embodiments, the calibration is applied mainly by: in a physical space to be calibrated, acquiring a monitoring image from a monitoring camera set according to the deployment parameters of the oblique overlooking camera; identifying the oblique overlooking position parameters of each object to be calibrated in the monitoring image from the monitoring image; and inputting the oblique overlooking position parameters of each object to be calibrated into the calibration model to obtain the spatial position parameters and orientation parameters of each object to be calibrated in the target physical space.
Specifically, the a camera in this embodiment is used only in the annotation stage. In an actual application scene, only a B camera needs to be installed, a B picture is collected, object detection is achieved by using a convolutional neural network CNN, BBX of a cow is obtained, then BBX data [ X, Y, W, H ] is used as input through a model M, data [ X, Y, W, H, D ] is obtained through reasoning, the data are prediction of BBX and orientation of a corresponding A picture, the information are prediction of real physical space position and standardized orientation of the cow body, and the information can be directly used for next production application.
The visual space calibration method provided by the embodiment of the present invention is described below by an optional implementation manner, as shown in fig. 11, the implementation manner provides a method including:
1) installing A, B cameras according to the deployment parameters;
2) the A, B cameras are synchronized in time through NTP service, and the synchronization time is less than 1 second;
3) simultaneously collecting A, B pictures of the camera; the collection frequency is 1 HZ; the number of the collected samples is more than 20000 groups of pictures, and each group of pictures consists of an A picture and a B picture at the same time.
4) Performing CNN detection on the A pictures in the group to obtain BBX;
5) for each BBX of the A picture in the group, taking a subgraph, and applying bovine head CNN recognition and OpenCV image processing to the subgraph to obtain the normalized orientation D1-D8;
6) performing CNN detection on the B pictures in the group to obtain BBX;
7) and calculating and sequencing the L of A, B pictures BBX at the same time, and automatically acquiring the corresponding relation of the BBX of the same cow.
8) Repeating the steps of 4-7 until enough BBX corresponding relations are obtained, and constructing a sample set;
9) constructing a mixed model, constructing a mapping from the BBX of the B to the BBX of the A by using a BP network, and constructing a mapping from the BBX of the B to the orientation D of the A by using an SVM; and optimizing parameters of the BP and SVM models by taking the minimum total error of the mixed model as an optimization object through a PSO algorithm.
10) And obtaining an optimized calibration model M and storing the calibration model M.
As shown in fig. 12, the procedure of applying the calibration model is as follows:
1) ensuring that the deployment parameters of the camera B are basically the same as the deployment parameters of the calibration environment;
2) collecting pictures of a camera B in real time, wherein the frequency of the scheme is 1 Hz;
3) performing CNN object detection on the B picture in real time to obtain BBX;
4) the BBX data is sent to the model M to predict the BBX and orientation of the model M on the A picture. There are 5 parameters, the first 4 being the physical spatial location of the cow and the 5 th being the orientation.
According to the embodiment of the invention, the spatial position parameter and orientation parameter of each target object in the vertical overlook image and the oblique overlook image of the calibration training space acquired at the same time are identified, and the oblique overlook position parameter of each target object in the oblique overlook image is identified; therefore, a sample mapping set of the oblique overlooking position parameter of each target object and the spatial position parameter and orientation parameter of each target object can be constructed, the sample mapping set is trained to obtain a calibration model, and then the calibration of the object to be calibrated in a spatial projection coordinate can be accurately and effectively realized through data driving according to an oblique overlooking monitoring image in practical application, so that the following problems existing in the conventional spatial calibration in the practical application are effectively solved:
1) the camera attitude estimation needs to adopt a standard calibration plate to implement calibration, and is difficult to implement in a field working environment;
2) the lens distortion correction is based on the characteristics of an ideal lens and is corrected by adopting theoretical methods such as affine and the like, but the distortion of the actual lens is not uniform, and the distortion correction result is not ideal;
3) the calibration is based on pixels and is suitable for processing a plane object, the actual working object is a three-dimensional livestock, and due to the lack of height information, the coordinate estimation of the livestock in a physical space is difficult to accurately realize in practical application.
Example two
The embodiment of the invention provides a visual space calibration device, which comprises a memory, a processor and a computer program, wherein the computer program is stored on the memory and can run on the processor;
the computer program, when being executed by the processor, realizes the steps of the visual space calibration method according to any one of the embodiments.
EXAMPLE III
The embodiment of the present invention provides a computer-readable storage medium, where a visual space calibration program is stored on the computer-readable storage medium, and when executed by a processor, the visual space calibration program implements the steps of the visual space calibration method according to any one of the embodiments.
The specific implementation of the second embodiment to the third embodiment can be referred to as the first embodiment, and has corresponding technical effects.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
While the present invention has been described with reference to the embodiments shown in the drawings, the present invention is not limited to the embodiments, which are illustrative and not restrictive, and it will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the spirit and scope of the invention as defined in the appended claims.

Claims (10)

1. A visual space calibration method is characterized by comprising the following steps:
obtaining a vertical overlook image and an oblique overlook image of a calibration training space which are acquired at the same time;
identifying a spatial position parameter and an orientation parameter of each target object in the vertical top view image and an oblique top view position parameter of each target object in the oblique top view image from the vertical top view image and the oblique top view image;
constructing a sample mapping set of the oblique overlooking position parameter of each target object and the spatial position parameter and the orientation parameter of each target object respectively;
and training the sample mapping set to obtain a calibration model, wherein the calibration model is used for a physical space to be calibrated, and the spatial position parameter and the orientation parameter of each object to be calibrated are determined according to the oblique overlooking position parameter of each object to be calibrated.
2. The method for calibrating visual space according to claim 1, wherein before obtaining the vertical overhead image and the oblique overhead image of the calibration training space acquired at the same time, the method comprises:
respectively acquiring a vertical overlook image and an oblique overlook image of the calibration training space at the same time through a vertical overlook camera and an oblique overlook camera which are arranged in the calibration training space; wherein, the calibration training space is provided with a neck cangue with the height of 1.5 meters and the length of 8 meters; the deployment parameters of the vertical overlook camera are as follows: vertically overlooking the ground plane of the calibration training space, wherein the height of the ground plane is 5 meters, and the horizontal distance between the ground plane and the neck cangue is 1.5 meters; the deployment parameters of the oblique overlooking camera are as follows: the included angle between the horizontal angle and the horizontal line is 48-55 degrees, the horizontal line faces the oblique lower part of the neck cangue, the height of the horizontal line is 4 meters, and the horizontal distance from the horizontal line to the neck cangue is 6 meters.
3. The visual space calibration method according to claim 1, further comprising:
acquiring a monitoring image from a monitoring camera in a physical space to be calibrated; the monitoring camera is set according to the deployment parameters of the oblique overlooking camera;
identifying the oblique overlooking position parameters of each object to be calibrated in the monitoring image from the monitoring image;
and inputting the oblique overlooking position parameters of each object to be calibrated into the calibration model to obtain the spatial position parameters and the orientation parameters of each object to be calibrated in the physical space to be calibrated.
4. The method for calibrating visual space according to claim 1, wherein said identifying spatial position and orientation parameters of each target object in said vertical top-view image and said oblique top-view position parameters of each target object in said oblique top-view image from said vertical top-view image and said oblique top-view image comprises:
respectively identifying a first identification area of each target object in the vertical overlooking image and a second identification area of each target object in the oblique overlooking image from the vertical overlooking image and the oblique overlooking image by adopting a preset convolutional neural network;
according to the first identification area of each target object, determining a spatial position parameter and an orientation parameter of each target object in the vertical overlook image;
determining an oblique overlooking position parameter of each target object in the oblique overlooking image according to the second identification area of each target object;
the constructing of the sample mapping set of the oblique overlook position parameter of each target object and the spatial position parameter and the orientation parameter of each target object includes:
establishing a corresponding relation between a first identification area and a second identification area of each target object;
and according to the corresponding relation, constructing a sample mapping set of the oblique overlooking position parameter of each target object and the spatial position parameter and the orientation parameter of each target object respectively.
5. The visual space calibration method according to claim 4, wherein the orientation parameter of each target object in the vertical overhead image is identified by the following steps:
identifying the head of each target object from the first identification area of each target object, and obtaining a third identification area of the head of each target object;
acquiring a maximum circumscribed ellipse surrounding each target object;
determining orientation parameters of each target object in the vertical overlook image according to the maximum circumscribed ellipse of each target object and the third identification area of each target object; the first identification area, the second identification area and the third identification area are identification frames; the spatial position parameter and the oblique overlooking position parameter are composed of an origin coordinate, a width and a height of the identification frame.
6. The visual space calibration method according to claim 5, wherein the determining of the orientation parameter of each target object in the vertical overhead view image according to the maximum circumscribed ellipse of each target object and the third identification area of each target object further comprises:
and according to a preset orientation approaching principle, carrying out normalization processing on orientation parameters of each target object in the vertical overlooking image.
7. The visual space calibration method according to claim 4, wherein the establishing of the correspondence relationship between the first identification area and the second identification area of each target object comprises:
determining a first center distance of each target object according to the distance between the center of the first identification area of each target object and the origin of the vertical overhead image on the vertical overhead image;
determining a second center distance of each target object according to the distance between the center of the second identification area of each target object and the origin of the oblique overhead view image on the oblique overhead view image;
respectively sequencing the first center distance and the second center distance of each target object according to the distance;
determining the difference value of the first center distance and the second center distance of the same sequencing position;
if each difference value is smaller than a preset threshold value, establishing a corresponding relation between a first identification area and a second identification area of each target object according to the sequence; otherwise, the vertical overlooking image and the oblique overlooking image of the calibration training space acquired at the same time are obtained again.
8. The visual space calibration method according to any one of claims 1-7, wherein the training of the sample mapping set to obtain a calibration model comprises:
establishing a BP and SVM mixed model, establishing oblique overlook position parameter and spatial position parameter and orientation parameter mapping by using the BP model, establishing oblique overlook position parameter and orientation parameter mapping by using the SVM model, and optimizing the BP model and the searched SVM model parameter by using the minimum total error of the mixed model as an optimization object through a PSO algorithm to obtain the calibration model.
9. A visual space calibration apparatus comprising a memory, a processor and a computer program stored on the memory and executable on the processor;
the computer program realizing the steps of the visual space calibration method according to any one of claims 1-8 when being executed by the processor.
10. A computer-readable storage medium, having stored thereon a visual space calibration program, which when executed by a processor, performs the steps of the visual space calibration method according to any one of claims 1-8.
CN202111071931.7A 2021-09-14 2021-09-14 Visual space calibration method, device and storage medium Active CN113744352B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111071931.7A CN113744352B (en) 2021-09-14 2021-09-14 Visual space calibration method, device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111071931.7A CN113744352B (en) 2021-09-14 2021-09-14 Visual space calibration method, device and storage medium

Publications (2)

Publication Number Publication Date
CN113744352A true CN113744352A (en) 2021-12-03
CN113744352B CN113744352B (en) 2022-07-29

Family

ID=78738452

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111071931.7A Active CN113744352B (en) 2021-09-14 2021-09-14 Visual space calibration method, device and storage medium

Country Status (1)

Country Link
CN (1) CN113744352B (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102645173A (en) * 2011-02-16 2012-08-22 张文杰 Multi-vision-based bridge three-dimensional deformation monitoring method
CN107860316A (en) * 2017-10-30 2018-03-30 重庆师范大学 Corn kernel three-dimensional parameter measurement apparatus and its measuring method
CN109272572A (en) * 2018-08-30 2019-01-25 中国农业大学 A kind of modeling method and device based on double Kinect cameras
CN110298320A (en) * 2019-07-01 2019-10-01 北京百度网讯科技有限公司 A kind of vision positioning method, device and storage medium
CN110335312A (en) * 2019-06-17 2019-10-15 武汉大学 A kind of object space localization method neural network based and device
CN111225558A (en) * 2017-08-07 2020-06-02 杰克逊实验室 Long-term and continuous animal behavior monitoring

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102645173A (en) * 2011-02-16 2012-08-22 张文杰 Multi-vision-based bridge three-dimensional deformation monitoring method
CN111225558A (en) * 2017-08-07 2020-06-02 杰克逊实验室 Long-term and continuous animal behavior monitoring
CN107860316A (en) * 2017-10-30 2018-03-30 重庆师范大学 Corn kernel three-dimensional parameter measurement apparatus and its measuring method
CN109272572A (en) * 2018-08-30 2019-01-25 中国农业大学 A kind of modeling method and device based on double Kinect cameras
CN110335312A (en) * 2019-06-17 2019-10-15 武汉大学 A kind of object space localization method neural network based and device
CN110298320A (en) * 2019-07-01 2019-10-01 北京百度网讯科技有限公司 A kind of vision positioning method, device and storage medium

Also Published As

Publication number Publication date
CN113744352B (en) 2022-07-29

Similar Documents

Publication Publication Date Title
CN111353413B (en) Low-missing-report-rate defect identification method for power transmission equipment
CN113537106B (en) Fish ingestion behavior identification method based on YOLOv5
WO2022188379A1 (en) Artificial intelligence system and method serving electric power robot
CN110570454B (en) Method and device for detecting foreign matter invasion
CN110490936B (en) Calibration method, device and equipment of vehicle camera and readable storage medium
CN108921057B (en) Convolutional neural network-based prawn form measuring method, medium, terminal equipment and device
CN112287896A (en) Unmanned aerial vehicle aerial image target detection method and system based on deep learning
CN112884782A (en) Biological object segmentation method, apparatus, computer device and storage medium
CN116543386A (en) Agricultural pest image identification method based on convolutional neural network
CN115810133A (en) Welding control method based on image processing and point cloud processing and related equipment
CN113744195B (en) Automatic hRPE cell microtubule detection method based on deep learning
CN110599514A (en) Image segmentation method and device, electronic equipment and storage medium
CN114037895A (en) Unmanned aerial vehicle pole tower inspection image identification method
JP7334083B2 (en) Classification reliability calculation device, region dividing device, learning device, classification reliability calculation method, learning method, classification reliability calculation program, and learning program
CN113744352B (en) Visual space calibration method, device and storage medium
CN115830302B (en) Multi-scale feature extraction fusion power distribution network equipment positioning identification method
CN115079393B (en) Chromosome karyotype analysis microscopic shooting device and method based on 10-fold objective lens
CN112115737B (en) Vehicle orientation determining method and device and vehicle-mounted terminal
CN110969156A (en) Convolutional neural network model for detecting milk cow nipple and construction method thereof
CN115861768A (en) Honeysuckle target detection and picking point positioning method based on improved YOLOv5
CN117036721A (en) Focusing method, apparatus, device, medium and program product for space station microscope
CN112069997B (en) Unmanned aerial vehicle autonomous landing target extraction method and device based on DenseHR-Net
CN114973384A (en) Electronic face photo collection method based on key point and visual salient target detection
CN114387308A (en) Machine vision characteristic tracking system
CN114155485A (en) Intelligent community intelligent security monitoring management system based on 3D vision

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant