CN111462194B - Training method, device and storage medium of object tracking model - Google Patents
Training method, device and storage medium of object tracking model Download PDFInfo
- Publication number
- CN111462194B CN111462194B CN202010236365.XA CN202010236365A CN111462194B CN 111462194 B CN111462194 B CN 111462194B CN 202010236365 A CN202010236365 A CN 202010236365A CN 111462194 B CN111462194 B CN 111462194B
- Authority
- CN
- China
- Prior art keywords
- image
- training
- pixel coordinates
- image acquisition
- tracking
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/292—Multi-camera tracking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Studio Devices (AREA)
Abstract
The application relates to a training method, a device and a storage medium of an object tracking model, belonging to the technical field of computers, wherein the method comprises the following steps: controlling a first image acquisition component to acquire images of a training image set to obtain sample pixel coordinates of a training object of each training image in the panoramic image; for each training image in the training image set, acquiring an expected rotation angle of the second image acquisition component relative to the training image; model training is carried out by using sample pixel coordinates and corresponding expected rotation angles of each training image, and an object tracking model is obtained; the problem that the tracking precision of tracking shooting objects is low due to the fact that random errors exist inevitably in the production process of the same model object tracking equipment can be solved; the object tracking model obtained through personalized training can be used for object tracking, so that the object tracking precision is improved. And because the training process is fully automatic, the model training efficiency can be improved.
Description
Technical Field
The application relates to a training method and device of an object tracking model and a storage medium, and belongs to the technical field of computers.
Background
In video conferences, tracking shooting is generally required to be performed with a speaker as a main body, which requires that an intelligent tracking camera be capable of performing accurate tracking and positioning.
In a typical object tracking method, an image is acquired by a tracking camera, and if a target object appears in the image, the target object is tracked and positioned.
However, since the tracking camera has random errors in the production process, there is caused a problem that the tracking accuracy of the tracking target object is low.
Disclosure of Invention
The application provides a training method, a training device and a storage medium of an object tracking model, which can automatically and quickly correct model parameters of produced object tracking equipment, and effectively solve the problems that the tracking accuracy of tracking a photographed object is low and personalized correction is difficult due to random errors inevitably existing in the production process of the same model of object tracking equipment. After personalized correction, the application also provides a scheme for automatically evaluating the correction effect, and only the corrected model meets the requirements, the correction is calculated, each device is ensured to meet the use requirements, and the application provides the following technical scheme:
In a first aspect, there is provided a method of training an object tracking model, the method comprising:
controlling a first image acquisition component to acquire images of a training image set to obtain sample pixel coordinates of a training object of each training image in the panoramic image; each training image in the training image set is positioned in the acquisition range of the first image acquisition component;
for each training image in the training image set, acquiring an expected rotation angle of a second image acquisition component relative to the training image, wherein the expected rotation angle enables a training object in the training image to be located in an expected image area of a tracking image acquired by the second image acquisition component;
model training is carried out by using sample pixel coordinates and corresponding expected rotation angles of each training image, and an object tracking model is obtained; the object tracking model is used for determining the rotation angle of the second image acquisition component according to the pixel coordinates of the target object in the panoramic image in the object tracking process, so that the second image acquisition component tracks and shoots the target object.
Optionally, the acquiring, for each training image in the training image set, a desired rotation angle of the second image acquisition component relative to the training image includes:
Acquiring a preset angle calculation formula;
inputting sample pixel coordinates of the training image into the angle calculation formula to obtain an initial rotation angle of the second image acquisition assembly;
controlling the second image acquisition assembly to rotate to the initial rotation angle;
acquiring pixel coordinates of a training object in the training image in a tracking image acquired by the second image acquisition component;
calculating an offset angle of the training object in a second image acquisition component by using pixel coordinates in the tracking image;
the desired rotation angle is determined based on the initial rotation angle and the offset angle.
Optionally, after the controlling the second image acquisition assembly to rotate to the initial rotation angle, the method further includes:
performing image recognition on the tracking image acquired by the second image acquisition component;
when the training object is identified, determining whether the identified training object is identical to the training object in the training image;
and triggering and executing the step of acquiring pixel coordinates of the training object in the training image in the tracking image acquired by the second image acquisition component when the identified training object is the same as the training object in the training image.
Optionally, the calculating the offset angle of the training object in the second image acquisition component using the pixel coordinates in the tracking image includes:
obtaining a scaling parameter of the second image acquisition component;
determining a focal length of the second image acquisition component according to the scaling parameter;
and inputting the focal length of the second image acquisition component and pixel coordinates in the tracking image into the angle calculation formula to obtain the offset angle.
Optionally, the model training is performed by using the sample pixel coordinates and the corresponding expected rotation angles of each training image, and after obtaining the object tracking model, the method further includes:
controlling the first image acquisition component to acquire images of a test image set to obtain test pixel coordinates of a test object of each test image in the panoramic image; the position of the test image set relative to the first image acquisition component is different from the position of the training image set relative to the first image acquisition component, and each test image in the test image set is positioned in the acquisition range of the first image acquisition component;
inputting the test pixel coordinates of the test image into the object tracking model for each test image in the test image set to obtain a test rotation angle of the second image acquisition assembly;
Controlling the second image acquisition assembly to rotate to the test rotation angle;
acquiring pixel coordinates of a test object in the test image in a tracking image acquired by a second image acquisition assembly after the angle rotation;
the accuracy of the object tracking model is determined based on differences between pixel coordinates in the tracking image and the desired image region.
Optionally, the desired image area is a pixel center point of a tracking image, and the determining the accuracy of the object tracking model based on a difference between pixel coordinates in the tracking image and the desired image area includes:
and when the difference value between the pixel coordinates in the tracking image and the pixel center point does not meet the correction condition, determining that the object tracking model is inaccurate, updating and correcting the object tracking model again until the difference value between the pixel coordinates in the tracking image and the pixel center point meets the correction condition.
Optionally, the correction condition includes:
for pixel coordinates of a test object in a plurality of test images in a tracking image, an average value of differences between horizontal pixel coordinates of the plurality of pixel coordinates and horizontal pixel coordinates of the pixel center point is smaller than a first preset threshold, and a maximum value of differences between horizontal pixel coordinates of the plurality of pixel coordinates and horizontal pixel coordinates of the pixel center point is smaller than a second preset threshold;
The difference between the vertical pixel coordinates of the plurality of pixel coordinates and the vertical pixel coordinates of the pixel center point is greater than a third preset threshold and less than a fourth preset threshold.
In a second aspect, there is provided a training apparatus for an object tracking model, the apparatus comprising:
the coordinate acquisition module is used for controlling the first image acquisition component to acquire images of the training image set, so as to obtain sample pixel coordinates of a training object of each training image in the panoramic image; each training image in the training image set is positioned in the acquisition range of the first image acquisition component;
the angle acquisition module is used for acquiring an expected rotation angle of the second image acquisition component relative to the training image for each training image in the training image set, wherein the expected rotation angle enables a training object in the training image to be located in an expected image area of the tracking image acquired by the second image acquisition component;
the model training module is used for carrying out model training by using sample pixel coordinates and corresponding expected rotation angles of each training image to obtain an object tracking model; the object tracking model is used for determining the rotation angle of the second image acquisition component according to the pixel coordinates of the target object in the panoramic image in the object tracking process, so that the second image acquisition component tracks and shoots the target object.
In a third aspect, there is provided a training apparatus for an object tracking model, the apparatus comprising a processor and a memory; the memory stores a program that is loaded and executed by the processor to implement the training method of the object tracking model of the first aspect.
In a fourth aspect, a computer readable storage medium is provided, in which a program is stored, the program being loaded and executed by the processor to implement the training method of the object tracking model according to the first aspect.
The application has the beneficial effects that: image acquisition is carried out on the training image set by controlling the first image acquisition component, so that sample pixel coordinates of a training object of each training image in the training image set in the panoramic image are obtained; for each training image in the training image set, acquiring an expected rotation angle of the second image acquisition component relative to the training image; model training is carried out by using sample pixel coordinates and corresponding expected rotation angles of each training image, and an object tracking model is obtained; the problem that the tracking accuracy of tracking the shot object is low due to the fact that random errors exist inevitably in the production process of the same type of object tracking equipment can be solved; because the object tracking model obtained by personalized training can be used for object tracking, the object tracking precision can be improved. Meanwhile, the training process is fully automatic, so that the model training efficiency can be improved.
The foregoing description is only an overview of the present application, and is intended to provide a better understanding of the present application, as it is embodied in the following description, with reference to the preferred embodiments of the present application and the accompanying drawings.
Drawings
FIG. 1 is a schematic diagram of a training system for an object tracking model according to an embodiment of the present application;
FIG. 2 is a schematic diagram of the distribution of target objects in a training image set according to one embodiment of the present application;
FIG. 3 is a flow chart of a method of training an object tracking model provided by one embodiment of the present application;
FIG. 4 is a schematic diagram of determining an angle of a face relative to a first image acquisition component, provided in one embodiment of the present application;
FIG. 5 is a flow chart of a training method for an object tracking model according to another embodiment of the present application;
FIG. 6 is a block diagram of a training apparatus for an object tracking model provided by one embodiment of the present application;
FIG. 7 is a block diagram of a training apparatus for an object tracking model provided by one embodiment of the present application.
Detailed Description
The following describes in further detail the embodiments of the present application with reference to the drawings and examples. The following examples are illustrative of the application and are not intended to limit the scope of the application.
FIG. 1 is a schematic structural diagram of a training system for an object tracking model according to an embodiment of the present application, where, as shown in FIG. 1, the system at least includes: a first image acquisition component 110, a second image acquisition component 120, and a control component 130.
The first image acquisition component 110 is used to acquire panoramic images. The first image capturing component 110 may also be referred to as a panoramic video camera, a panoramic camera, etc., and the present embodiment does not limit the name of the first image capturing component.
The first image acquisition assembly 110 is communicatively coupled to the control assembly 130. The first image acquisition component 110 transmits the acquired panoramic image to the control component 130 through a communication connection with the control component 130; alternatively, the first image capturing component 110 may identify the target object, and send the pixel coordinates of the identified target object to the control component 130.
Alternatively, the control component 130 may be a mobile phone, a tablet computer, a cradle head, or the like, which is not limited to the implementation of the control component 130 in this embodiment.
In this embodiment, the control component 130 is configured to: controlling the first image acquisition component 110 to acquire images of the training image set, so as to obtain sample pixel coordinates of a training object of each training image in the panoramic image; for each training image in the training image set, acquiring a desired rotation angle of the second image acquisition component 120 relative to the training image; model training is carried out by using sample pixel coordinates and corresponding expected rotation angles of each training image, and an object tracking model is obtained.
The object tracking model is used for determining a rotation angle of the second image acquisition component 120 according to pixel coordinates of the target object in the panoramic image in the object tracking process, so that the second image acquisition component 120 tracks and shoots the target object.
The training image set includes a plurality of training images. The training image is an image including a training object. In the present application, the training object and the target object refer to the object tracked by the second image acquisition component 120, and the training object and the target object may be a human face, a vehicle, an animal, etc.; the type of the training object is the same as or different from the type of the target object; the present embodiment does not limit the types of training objects and target objects.
Each training image in the training image set is located within the acquisition range of the first image acquisition component. The training images may be posted on a wall panel or displayed by a display device (e.g., a television, a display screen, etc.), and the embodiment does not limit the manner in which the training images are arranged. The positions of the plurality of training images in the training image set relative to the first acquisition assembly are distributed in different directions, and the acquisition range of the first image acquisition assembly is covered as far as possible. Referring to the arrangement of the training images 21 shown in fig. 2, as can be seen from fig. 2, a plurality of training images 21 are distributed in the horizontal direction and the vertical direction, respectively.
In this embodiment, by distributing a plurality of training images in different directions, the training images can be distributed as uniformly as possible within the acquisition range of the first image acquisition assembly, so that the training result is more reliable. And when the coverage area of the training image set is large enough, a plurality of first image acquisition assemblies and second image acquisition assemblies can be trained at the same time, so that the training efficiency is improved. In addition, the first image acquisition component and the second image acquisition component are trained through the training image set, simulation training of changing different positions by personnel is not needed, the training time of the object tracking model is shortened, and the training efficiency and accuracy of the object tracking model are optimized.
The sample pixel coordinates of the training object in the panoramic image are determined based on a coordinate system established by taking the center point of the panoramic image as an origin, the horizontal axis as an x axis and the vertical axis as a y axis; or, the sample pixel coordinates are determined based on a coordinate system established by taking the lower left vertex of the panoramic image as an origin, the horizontal bottom edge as an x-axis and the vertical left side edge as a y-axis, however, the sample pixel coordinates may also be determined based on other coordinate systems, and the determination manner of the coordinate system is not limited in this embodiment.
The desired angle of rotation is such that the training object in the training image is located within the desired image area of the tracking image acquired by the second image acquisition component 120. Wherein the desired image area is determined based on the pixel center point of the tracking image. Optionally, the desired rotation angle is such that the center point of the training object is located within a desired image area of the tracking image.
The control component 130 is also communicatively coupled to the second image acquisition component 120. The second image acquisition component 120 is configured to track images of an acquisition object (including a training object, a target object, and a test object hereinafter) to obtain a tracked image. The photographing angle of the second image capturing assembly 120 may be rotated, for example: 360 degrees in the horizontal direction and 180 degrees in the vertical direction. Optionally, the focal length of the second acquisition assembly is variable. In practical implementation, the second image capturing component 120 may be a camera with a cradle head, which is an electronic cradle head, that is, may drive the camera to rotate according to a control instruction of the control component 130.
Optionally, the acquisition range of the second image acquisition component 120 is smaller than the acquisition range of the first image acquisition component 110. Optionally, after the second image acquisition component 120 acquires the tracking image, the acquired tracking image data may be sent to the control component 130.
In one example, the second image acquisition assembly 120 is co-axially located and positioned adjacent to the first image acquisition assembly 110. Such as: the second image acquisition assembly 120 is located directly below the first image acquisition assembly 110; or directly above.
Of course, in other embodiments, the second image acquisition assembly 120 may not be co-located with the first image acquisition assembly 110.
Alternatively, the first image capturing assembly 110 rotates in synchronization with the second image capturing assembly 120; alternatively, the first image acquisition assembly does not rotate in synchronization with the second image acquisition assembly 120.
In this embodiment, only the number of the first image capturing units 120 and the second image capturing units 120 is taken as an example to describe an example, and in actual implementation, the control unit 130 may be communicatively connected to the plurality of first image capturing units 120 and the plurality of second image capturing units 120, and the number of the first image capturing units 120 and the number of the second image capturing units 120 are not limited in this embodiment.
Alternatively, the control component 130 may be integrated in the second image acquisition component 120, i.e. the control component 130 is implemented as the same device as the second image acquisition component 120.
Fig. 3 is a flowchart of a training method of an object tracking model according to an embodiment of the present application, where the method is applied to the training system of the object tracking model shown in fig. 1, and the execution subject of each step is illustrated as a control component 130 in the system. The method at least comprises the following steps:
Step 301, controlling a first image acquisition component to acquire images of a training image set, and obtaining sample pixel coordinates of a training object of each training image in the panoramic image.
Wherein each training image in the training image set is located within the acquisition range of the first image acquisition component.
Optionally, the sample pixel coordinates of the training object in the training image in the panoramic image may be sent by the first image acquisition component (i.e., obtained after the first image acquisition component recognizes the target object in the panoramic image); or the panoramic image is obtained by identifying the panoramic image sent by the first image acquisition component by the control component.
It should be noted that, if the target object is identified by the first image capturing component, the panoramic image may be an image displayed in the viewfinder by the first image capturing component, where the image is inaccessible; alternatively, the first image acquisition component may acquire an accessible image.
Because the first image acquisition component or the control component is used for identifying the training object in the training image, in the application, the sample pixel coordinates of the training image refer to: sample pixel coordinates in a training object panoramic image in a training image.
Optionally, the pixel coordinates of the training object in the panoramic image are determined based on a coordinate system established with the center point of the panoramic image as an origin, the horizontal axis as an x-axis, and the vertical axis as a y-axis; or, the pixel coordinates are determined based on a coordinate system established by taking the lower left vertex of the panoramic image as an origin, the horizontal bottom edge as an x-axis and the vertical left side edge as a y-axis, however, the pixel coordinates may be determined based on other coordinate systems, and the determination manner of the coordinate system is not limited in this embodiment.
Optionally, the sample pixel coordinates of the training object in the panoramic image are: training pixel coordinates of a center point of the object (such as a face center point); alternatively, an average of pixel coordinates of each point of the subject is trained.
Step 302, for each training image in the training image set, obtaining a desired rotation angle of the second image acquisition component relative to the training image.
The desired rotation angle is such that the training object in the training image is located within a desired image area of the tracking image acquired by the second image acquisition component. Wherein the desired image area is determined based on the pixel center point of the tracking image.
The second image acquisition component is used for tracking the target object to shoot and obtain a tracking image. The photographing angle of the second image capturing assembly may be rotated.
The desired rotation angle of the second image acquisition assembly includes a rotation angle in a horizontal direction and a rotation angle in a vertical direction.
Optionally, the control component sends a control instruction to the second image acquisition component, wherein the control instruction carries the rotation angle; and after receiving the control instruction, the second image acquisition component rotates based on the rotation angle in the control instruction. Or the control component controls the equipment body (such as a cradle head) to rotate according to the rotation angle, and the equipment body drives the second image acquisition component to rotate.
In one example, for each training image in the training image set, obtaining a desired angle of rotation of the second image acquisition component relative to the training image comprises: acquiring a preset angle calculation formula; inputting sample pixel coordinates of the training image into an angle calculation formula to obtain an initial rotation angle of the second image acquisition assembly; controlling the second image acquisition assembly to rotate to an initial rotation angle; acquiring pixel coordinates of a training object in a training image in a tracking image acquired by a second image acquisition component; calculating an offset angle of the training object in the second image acquisition component by using pixel coordinates in the tracking image; the desired rotation angle is determined based on the initial rotation angle and the offset angle.
In one example, the angle calculation formula is represented by: tanα=fh/F.
Wherein, alpha is the angle (horizontal angle or vertical angle) of the training object relative to the first image acquisition component, fh is the distance (horizontal distance or vertical distance) between the pixel coordinate of the training object and the center of the screen of the first image acquisition component, and F is the focal length of the first image acquisition component. In this embodiment, the first image capturing component and the second image capturing component are integrated, and the two components are very close to each other, and at this time, the angle of the training object relative to the first image capturing component approximates to the initial rotation angle of the training object relative to the second image capturing component.
Referring to the schematic cross-sectional view of the training object and the first image acquisition component shown in fig. 4, fig. 4 illustrates an example in which the training object is a face and the sample pixel coordinates of the training object are the pixel coordinates of the center point of the face. As can be seen from fig. 4, the face center point P is in a trigonometric function relationship between the imaging position (i.e. pixel coordinates) in the first image acquisition component and the lens focal length F, and the angle calculation formula can be obtained based on the trigonometric function relationship.
In another example, to improve the accuracy of the angle calculation, the angle calculation formula is expressed by: tanα=fh/f+b.
Where b represents the bias term. Initializing the value of b to be a preset value.
Because the training images acquired by the first image acquisition component are multiple, the one-to-one correspondence between the sample pixel coordinates of each training image and the expected rotation angle is ensured. After the second image acquisition assembly is controlled to rotate to an initial rotation angle, image recognition is required to be carried out on the tracking image acquired by the second image acquisition assembly; when the training object is identified, determining whether the identified training object is identical to the training object in the training image; and when the recognized training object is the same as the training object in the training image, executing the step of acquiring the pixel coordinates of the training object in the training image in the tracking image acquired by the second image acquisition component.
In one example, the second image acquisition assembly is a tracking camera, the focal length of which is varied, so that it is necessary to acquire the focal length of the second image acquisition assembly first and then calculate the offset angle from the current focal length. Specifically, calculating an offset angle of the training object in the second image acquisition component using pixel coordinates in the tracking image includes: obtaining a scaling parameter of the second image acquisition component; determining a focal length of the second image acquisition assembly according to the scaling parameter; and inputting the focal length of the second image acquisition component and pixel coordinates in the tracking image into an angle calculation formula to obtain an offset angle.
The focal length of the second image acquisition assembly is determined according to the scaling parameters and is expressed by the following formula (other types of devices may have different expression formulas, and the embodiment is not limited to the specific expression formulas):
1/F’＝[tan(a×zoom+b)]/c
wherein F' is the focal length of the second image acquisition assembly, and zoom is a scaling parameter. a, b and c are specific values obtained by fitting related parameters of the second image acquisition assembly, and the values of a, b and c corresponding to the image acquisition assemblies of different models are different. At this time, fh in the angle calculation formula is the distance (horizontal distance or vertical distance) between the pixel coordinates of the target object in the tracking image and the center of the screen of the second acquisition component; f is the focal length F' of the second image acquisition assembly. In one embodiment, for example, the focal length fitting formula for a certain model of camera is:
1/F’＝[tan(-0.0001×zoom+0.785398)]/1350
after the offset angle is obtained, the initial rotation angle and the offset angle are added to obtain the desired rotation angle.
And 303, performing model training by using the sample pixel coordinates and the corresponding expected rotation angles of each training image to obtain an object tracking model.
The object tracking model is used for determining the rotation angle of the second image acquisition component according to the pixel coordinates of the target object in the panoramic image so that the second image acquisition component can track and shoot the target object.
Optionally, performing model training using the sample pixel coordinates and the desired rotation angle to obtain an object tracking model, including: and training parameters in the angle calculation formula by using sample pixel coordinates and corresponding expected rotation angles of each training image to obtain an object tracking model. Such as: and training the values of 1/F and b in the angle calculation formula to obtain an object tracking model.
In summary, in the training method of the object tracking model provided in this embodiment, the first image acquisition component is controlled to perform image acquisition on the training image set, so as to obtain the sample pixel coordinates of the training object of each training image in the panoramic image in the training image set; for each training image in the training image set, acquiring an expected rotation angle of the second image acquisition component relative to the training image; model training is carried out by using sample pixel coordinates and corresponding expected rotation angles of each training image, and an object tracking model is obtained; the problem that the tracking accuracy of tracking the shot object is low due to the fact that random errors exist inevitably in the production process of the same type of object tracking equipment can be solved; because the object tracking model obtained by personalized training can be used for object tracking, the object tracking precision can be improved. Meanwhile, the training process is fully automatic, so that the model training efficiency can be improved.
In addition, when the expected rotation angle corresponding to each sample pixel coordinate is obtained, determining an initial rotation angle of the second image acquisition assembly according to the sample pixel coordinate, and controlling the second image acquisition assembly to rotate to the initial rotation angle; then, calculating an offset angle of the second image acquisition assembly, and fine-adjusting the initial rotation angle of the second image acquisition assembly so that the training object is positioned in an expected image area of the tracking image; since the second image acquisition assembly is rotated to the initial rotation angle, the training image is typically located within the tracking image; the problem that when the second image acquisition component randomly rotates to determine the expected rotation angle, the training image is not normally positioned in the tracking image, and the efficiency of determining the expected rotation angle is low can be avoided; therefore, the efficiency of determining the desired rotation angle can be improved.
Optionally, after the object tracking model is obtained through the above embodiment, the object tracking model needs to be evaluated to determine the accuracy of the tracking model. At this time, referring to fig. 5, after step 303, at least the following steps are further included:
step 501, controlling a first image acquisition component to acquire images of a test image set, and obtaining test pixel coordinates of a test object of each test image in the panoramic image.
The position of the test image set relative to the first image acquisition assembly is different from the position of the training image set relative to the first image acquisition assembly, and each test image in the test image set is located in the acquisition range of the first image acquisition assembly.
Alternatively, the test object may be the same type of object as the training object and the target object; or a different type of object.
Optionally, the test image set is the same as or different from the training image set.
The relevant description of this step is detailed in step 301, except that the training image set is replaced with the test image set, the training image is replaced with the test image, and the training object is replaced with the test object.
Step 502, for each test image in the test image set, inputting the test pixel coordinates of the test image into the object tracking model to obtain the test rotation angle of the second image acquisition assembly.
Because the object tracking model reflects the mapping relation between the pixel coordinates of the target object in the panoramic image and the rotation angle of the second image acquisition assembly, after the test pixel coordinates are input into the object tracking model, the test rotation angle corresponding to the test pixel coordinates can be obtained.
In step 503, the second image capturing component is controlled to rotate to the test rotation angle.
Optionally, the control component sends a control instruction to the second image acquisition component, wherein the control instruction carries a test rotation angle; and after the second image acquisition component receives the control instruction, rotating based on the test rotation angle in the control instruction. Or the control component controls the equipment body (such as a cradle head) to rotate according to the test rotation angle, and the equipment body drives the second image acquisition component to rotate.
Step 504, acquiring pixel coordinates of the test object in the test image in the tracking image acquired by the second image acquisition component after the angle rotation.
Alternatively, the pixel coordinates of the test object in the tracking image may be sent by the second image acquisition component (i.e., obtained after the second image acquisition component recognizes the target object in the tracking image); or the control component is obtained by identifying the tracking image sent by the second image acquisition component.
In step 505, the accuracy of the object tracking model is determined based on the difference between the pixel coordinates in the tracking image and the desired image area.
Optionally, the desired image area is a pixel center point of the tracking image, and determining accuracy of the object tracking model based on a difference between pixel coordinates in the tracking image and the desired image area includes: when the difference between the pixel coordinates and the pixel center point in the tracking image does not meet the correction condition, determining that the object tracking model is inaccurate, and updating and correcting the object tracking model again (namely, executing steps 301-303 again) until the difference between the pixel coordinates and the pixel center point in the tracking image meets the correction condition.
In one example, the correction conditions include: for pixel coordinates of a test object in the plurality of test images in the tracking image, an average value of differences between horizontal pixel coordinates of the plurality of pixel coordinates and horizontal pixel coordinates of the pixel center point is smaller than a first preset threshold, and a maximum value of differences between horizontal pixel coordinates of the plurality of pixel coordinates and horizontal pixel coordinates of the pixel center point is smaller than a second preset threshold; the difference between the vertical pixel coordinates of the plurality of pixel coordinates and the vertical pixel coordinates of the pixel center point is greater than a third preset threshold and less than a fourth preset threshold.
The correction condition is expressed by the following formula:
c<y i <d
x i is the difference between the i-th horizontal pixel coordinate of the plurality of horizontal pixel coordinates and the horizontal pixel coordinate of the pixel center point. a is a first preset threshold; b is a second preset threshold. y is i Is the difference between the i-th vertical pixel coordinate of the plurality of vertical pixel coordinates and the vertical pixel coordinate of the pixel center point. c is a third preset threshold; d is a fourth preset threshold.
Of course, the correction conditions may also be other ways, such as: for the vertical pixel coordinate of each pixel coordinate in the plurality of pixel coordinates, the average value of the differences between the plurality of vertical pixel coordinates and the vertical pixel coordinate of the pixel center point is smaller than the first preset threshold, and the maximum value of the differences between the plurality of vertical pixel coordinates and the vertical pixel coordinate of the pixel center point is smaller than the second preset threshold.
In summary, according to the training method of the object tracking model provided by the embodiment, the accuracy of the object tracking model is verified, and when the object tracking model meets the correction condition, the object tracking model is corrected again until the object tracking model does not meet the correction condition, and the object tracking model is stopped; the accuracy of tracking and positioning by using the tracking and positioning model can be improved.
Fig. 6 is a block diagram of an object tracking model training apparatus according to an embodiment of the present application, and this embodiment is described by taking a control unit 130 in a training system of the object tracking model shown in fig. 1 as an example. The device at least comprises the following modules: a coordinate acquisition module 610, an angle acquisition module 620, and a model training module 630.
The coordinate acquisition module 610 is configured to control the first image acquisition component to perform image acquisition on a training image set, so as to obtain sample pixel coordinates of a training object of each training image in the training image set in the panoramic image; each training image in the training image set is located within an acquisition range of the first image acquisition component.
The angle acquisition module 620 is configured to acquire, for each training image in the training image set, a desired rotation angle of the second image acquisition component relative to the training image, where the desired rotation angle enables a training object in the training image to be located in a desired image area of the tracking image acquired by the second image acquisition component.
The model training module 630 is configured to perform model training by using the sample pixel coordinates and the corresponding expected rotation angles of each training image, so as to obtain an object tracking model; the object tracking model is used for determining the rotation angle of the second image acquisition component according to the pixel coordinates of the target object in the panoramic image in the object tracking process, so that the second image acquisition component tracks and shoots the target object.
Optionally, the angle acquisition module 620 includes: an angle formula acquisition unit 621, an initial angle calculation unit 622, a rotation angle control unit 623, a pixel coordinate acquisition unit 624, an offset angle acquisition unit 625, and a rotation angle determination unit 626.
An angle formula acquisition unit 621 for acquiring a preset angle calculation formula;
an initial angle calculation unit 622, configured to input the sample pixel coordinates of the training image into the angle calculation formula, to obtain an initial rotation angle of the second image acquisition component;
a rotation angle control unit 623 for controlling the rotation of the second image capturing component to the initial rotation angle;
a pixel coordinate acquiring unit 624, configured to acquire pixel coordinates of a training object in the training image in the tracking image acquired by the second image acquisition component;
An offset angle obtaining unit 625, configured to calculate an offset angle of the training object in the second image acquisition component using pixel coordinates in the tracking image;
a rotation angle determining unit 626 for determining the desired rotation angle based on the initial rotation angle and the offset angle.
Optionally, the angle acquisition module 620 further includes: the object recognition unit 627 is trained. The object identifying unit 627 is configured to: after the second image acquisition assembly is controlled to rotate to the initial rotation angle, carrying out image recognition on the tracking image acquired by the second image acquisition assembly; when the training object is identified, determining whether the identified training object is identical to the training object in the training image; and triggering the pixel coordinate acquisition unit 624 to execute the step of acquiring the pixel coordinates of the training object in the training image in the tracking image acquired by the second image acquisition component when the identified training object is the same as the training object in the training image.
Optionally, the offset angle acquiring unit 625 is configured to:
obtaining a scaling parameter of the second image acquisition component;
Determining a focal length of the second image acquisition component according to the scaling parameter;
and inputting the focal length of the second image acquisition component and pixel coordinates in the tracking image into the angle calculation formula to obtain the offset angle.
Optionally, the apparatus further comprises: a test coordinate acquisition module 640, a test angle acquisition module 650, a rotation angle control module 660, a tracking coordinate acquisition module 670, and a tracking model verification module 680.
The test coordinate acquisition module 640 is configured to perform model training by using the sample pixel coordinates and the corresponding expected rotation angles of each training image, and after obtaining an object tracking model, control the first image acquisition component to perform image acquisition on a test image set, so as to obtain test pixel coordinates of a test object of each test image in the test image set in a panoramic image; the position of the test image set relative to the first image acquisition component is different from the position of the training image set relative to the first image acquisition component, and each test image in the test image set is positioned in the acquisition range of the first image acquisition component;
a test angle acquisition module 650, configured to input, for each test image in the set of test images, a test pixel coordinate of the test image into the object tracking model to obtain a test rotation angle of the second image acquisition component;
A rotation angle control module 660 for controlling the second image acquisition assembly to rotate to the test rotation angle;
a tracking coordinate acquiring module 670, configured to acquire pixel coordinates in a tracking image acquired by the second image acquisition component after the test object in the test image rotates at an angle;
a tracking model verification module 680 for determining the accuracy of the object tracking model based on the differences between pixel coordinates in the tracking image and the desired image area.
Optionally, the desired image area is a pixel center point of the tracking image, and the tracking model verification module 680 is configured to:
and when the difference value between the pixel coordinates in the tracking image and the pixel center point does not meet the correction condition, determining that the object tracking model is inaccurate, updating and correcting the object tracking model again until the difference value between the pixel coordinates in the tracking image and the pixel center point meets the correction condition.
Optionally, the correction condition includes:
for pixel coordinates of a test object in a plurality of test images in a tracking image, an average value of differences between horizontal pixel coordinates of the plurality of pixel coordinates and horizontal pixel coordinates of the pixel center point is smaller than a first preset threshold, and a maximum value of differences between horizontal pixel coordinates of the plurality of pixel coordinates and horizontal pixel coordinates of the pixel center point is smaller than a second preset threshold;
The difference between the vertical pixel coordinates of the plurality of pixel coordinates and the vertical pixel coordinates of the pixel center point is greater than a third preset threshold and less than a fourth preset threshold.
For relevant details reference is made to the method embodiments described above.
It should be noted that: in the training device for an object tracking model provided in the above embodiment, only the division of the above functional modules is used for illustration, and in practical application, the above functional allocation may be performed by different functional modules according to needs, that is, the internal structure of the training device for an object tracking model is divided into different functional modules, so as to complete all or part of the functions described above. In addition, the training device of the object tracking model provided in the above embodiment and the training method embodiment of the object tracking model belong to the same concept, and detailed implementation processes of the training device and the training method embodiment of the object tracking model are detailed in the method embodiment, and are not repeated here.
FIG. 7 is a block diagram of an object tracking model training apparatus, which may be an apparatus including the control component 130 shown in FIG. 1, provided in accordance with one embodiment of the present application. The apparatus comprises at least a processor 701 and a memory 702.
The processor 701 may include one or more processing cores, such as: 4 core processors, 8 core processors, etc. The processor 701 may employ a DSP (Digital Signal Processing ), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA
(Programmable Logic Array) the programmable logic array is implemented in at least one hardware form. The processor 701 may also include a main processor, which is a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit ); a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 701 may integrate a GPU (Graphics Processing Unit, image processor) for rendering and drawing of content required to be displayed by the display screen. In some embodiments, the processor 701 may also include an AI (Artificial Intelligence ) processor for processing computing operations related to machine learning.
Memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in memory 702 is used to store at least one instruction for execution by processor 701 to implement the training method of the object tracking model provided by the method embodiments of the present application.
In some embodiments, the training device of the object tracking model may further optionally include: a peripheral interface and at least one peripheral. The processor 701, the memory 702, and the peripheral interfaces may be connected by buses or signal lines. The individual peripheral devices may be connected to the peripheral device interface via buses, signal lines or circuit boards. Illustratively, peripheral devices include, but are not limited to: radio frequency circuitry, display screen, audio circuitry, and power supply, among others.
Of course, the training apparatus of the object tracking model may also include fewer or more components, which is not limited in this embodiment.
Optionally, the present application further provides a computer readable storage medium, where a program is stored, where the program is loaded and executed by a processor to implement the training method of the object tracking model of the above method embodiment.
Optionally, the present application further provides a computer product, where the computer product includes a computer readable storage medium, where a program is stored, where the program is loaded and executed by a processor to implement the training method of the object tracking model of the above method embodiment.
The technical features of the above-described embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above-described embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples illustrate only a few embodiments of the application, which are described in detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.
Claims (9)
1. A method of training an object tracking model, the method comprising:
controlling a first image acquisition component to acquire images of a training image set to obtain sample pixel coordinates of a training object of each training image in the panoramic image; each training image in the training image set is positioned in the acquisition range of the first image acquisition component;
For each training image in the training image set, acquiring an expected rotation angle of a second image acquisition component relative to the training image, wherein the expected rotation angle enables a training object in the training image to be located in an expected image area of a tracking image acquired by the second image acquisition component;
model training is carried out by using sample pixel coordinates and corresponding expected rotation angles of each training image, and an object tracking model is obtained; the object tracking model is used for determining the rotation angle of the second image acquisition component according to the pixel coordinates of the target object in the panoramic image in the object tracking process so that the second image acquisition component tracks and shoots the target object;
the second image acquisition assembly is adjacent to the first image acquisition assembly in position;
the acquiring, for each training image in the training image set, a desired rotation angle of the second image acquisition component relative to the training image includes:
acquiring a preset angle calculation formula;
inputting sample pixel coordinates of the training image into the angle calculation formula to obtain an initial rotation angle of the second image acquisition assembly;
Controlling the second image acquisition assembly to rotate to the initial rotation angle;
acquiring pixel coordinates of a training object in the training image in a tracking image acquired by the second image acquisition component;
calculating an offset angle of the training object in a second image acquisition component by using pixel coordinates in the tracking image;
the desired rotation angle is determined based on the initial rotation angle and the offset angle.
2. The method of claim 1, wherein said controlling said second image acquisition assembly to rotate to said initial rotation angle further comprises:
performing image recognition on the tracking image acquired by the second image acquisition component;
when the training object is identified, determining whether the identified training object is identical to the training object in the training image;
and triggering and executing the step of acquiring pixel coordinates of the training object in the training image in the tracking image acquired by the second image acquisition component when the identified training object is the same as the training object in the training image.
3. The method of claim 1, wherein the calculating an offset angle of the training object in a second image acquisition component using pixel coordinates in the tracking image comprises:
Obtaining a scaling parameter of the second image acquisition component;
determining a focal length of the second image acquisition component according to the scaling parameter;
and inputting the focal length of the second image acquisition component and pixel coordinates in the tracking image into the angle calculation formula to obtain the offset angle.
4. A method according to any one of claims 1 to 3, wherein the model training using the sample pixel coordinates and the corresponding desired rotation angle of each training image, after obtaining the object tracking model, further comprises:
controlling the first image acquisition component to acquire images of a test image set to obtain test pixel coordinates of a test object of each test image in the panoramic image; the position of the test image set relative to the first image acquisition component is different from the position of the training image set relative to the first image acquisition component, and each test image in the test image set is positioned in the acquisition range of the first image acquisition component;
inputting the test pixel coordinates of the test image into the object tracking model for each test image in the test image set to obtain a test rotation angle of the second image acquisition assembly;
Controlling the second image acquisition assembly to rotate to the test rotation angle;
acquiring pixel coordinates of a test object in the test image in a tracking image acquired by a second image acquisition assembly after the angle rotation;
the accuracy of the object tracking model is determined based on differences between pixel coordinates in the tracking image and the desired image region.
5. The method of claim 4, wherein the desired image region is a pixel center point of a tracking image, the determining the accuracy of the object tracking model based on differences between pixel coordinates in the tracking image and the desired image region comprising:
and when the difference value between the pixel coordinates in the tracking image and the pixel center point does not meet the correction condition, determining that the object tracking model is inaccurate, updating and correcting the object tracking model again until the difference value between the pixel coordinates in the tracking image and the pixel center point meets the correction condition.
6. The method of claim 5, wherein the correction conditions comprise:
for pixel coordinates of a test object in a plurality of test images in a tracking image, an average value of differences between horizontal pixel coordinates of the plurality of pixel coordinates and horizontal pixel coordinates of the pixel center point is smaller than a first preset threshold, and a maximum value of differences between horizontal pixel coordinates of the plurality of pixel coordinates and horizontal pixel coordinates of the pixel center point is smaller than a second preset threshold;
The difference between the vertical pixel coordinates of the plurality of pixel coordinates and the vertical pixel coordinates of the pixel center point is greater than a third preset threshold and less than a fourth preset threshold.
7. A training apparatus for an object tracking model, the apparatus comprising:
the coordinate acquisition module is used for controlling the first image acquisition component to acquire images of the training image set, so as to obtain sample pixel coordinates of a training object of each training image in the panoramic image; each training image in the training image set is positioned in the acquisition range of the first image acquisition component;
the angle acquisition module is used for acquiring an expected rotation angle of the second image acquisition component relative to the training image for each training image in the training image set, wherein the expected rotation angle enables a training object in the training image to be located in an expected image area of the tracking image acquired by the second image acquisition component; the second image acquisition assembly is adjacent to the first image acquisition assembly in position; the acquiring, for each training image in the training image set, a desired rotation angle of the second image acquisition component relative to the training image includes: acquiring a preset angle calculation formula; inputting sample pixel coordinates of the training image into the angle calculation formula to obtain an initial rotation angle of the second image acquisition assembly; controlling the second image acquisition assembly to rotate to the initial rotation angle; acquiring pixel coordinates of a training object in the training image in a tracking image acquired by the second image acquisition component; calculating an offset angle of the training object in a second image acquisition component by using pixel coordinates in the tracking image; determining the desired rotation angle based on the initial rotation angle and the offset angle;
The model training module is used for carrying out model training by using sample pixel coordinates and corresponding expected rotation angles of each training image to obtain an object tracking model; the object tracking model is used for determining the rotation angle of the second image acquisition component according to the pixel coordinates of the target object in the panoramic image in the object tracking process, so that the second image acquisition component tracks and shoots the target object.
8. A training device for an object tracking model, the device comprising a processor and a memory; stored in the memory is a program that is loaded and executed by the processor to implement the training method of the object tracking model according to any one of claims 1 to 6.
9. A computer-readable storage medium, in which a program is stored which, when being executed by a processor, is adapted to carry out a training method of an object tracking model according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010236365.XA CN111462194B (en) | 2020-03-30 | 2020-03-30 | Training method, device and storage medium of object tracking model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010236365.XA CN111462194B (en) | 2020-03-30 | 2020-03-30 | Training method, device and storage medium of object tracking model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111462194A CN111462194A (en) | 2020-07-28 |
| CN111462194B true CN111462194B (en) | 2023-08-11 |
Family
ID=71680242
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010236365.XA Active CN111462194B (en) | 2020-03-30 | 2020-03-30 | Training method, device and storage medium of object tracking model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111462194B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107679455A (en) * | 2017-08-29 | 2018-02-09 | 平安科技（深圳）有限公司 | Target tracker, method and computer-readable recording medium |
| CN109492506A (en) * | 2017-09-13 | 2019-03-19 | 华为技术有限公司 | Image processing method, device and system |
| CN110232706A (en) * | 2019-06-12 | 2019-09-13 | 睿魔智能科技（深圳）有限公司 | More people are with shooting method, device, equipment and storage medium |
-
2020
- 2020-03-30 CN CN202010236365.XA patent/CN111462194B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107679455A (en) * | 2017-08-29 | 2018-02-09 | 平安科技（深圳）有限公司 | Target tracker, method and computer-readable recording medium |
| CN109492506A (en) * | 2017-09-13 | 2019-03-19 | 华为技术有限公司 | Image processing method, device and system |
| CN110232706A (en) * | 2019-06-12 | 2019-09-13 | 睿魔智能科技（深圳）有限公司 | More people are with shooting method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111462194A (en) | 2020-07-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111325798B (en) | Camera model correction method, device, AR implementation equipment and readable storage medium | |
| CN113194263B (en) | Gun and ball linkage control method and device, computer equipment and storage medium | |
| CN112714926A (en) | Method and device for generating a photo-realistic three-dimensional model of a recording environment | |
| KR20180022539A (en) | Method and apparatus of stitching for minimizing parallax using control points in overlapping region | |
| CN110136207B (en) | Fisheye camera calibration system, fisheye camera calibration method, fisheye camera calibration device, electronic equipment and storage medium | |
| CN111028205A (en) | Eye pupil positioning method and device based on binocular ranging | |
| CN113021328A (en) | Hand-eye calibration method, device, equipment and medium | |
| US20210407132A1 (en) | Fisheye camera calibration system, method and electronic device | |
| CN108648141B (en) | Image splicing method and device | |
| CN108717704A (en) | Method for tracking target, computer installation based on fish eye images and computer readable storage medium | |
| CN111815715A (en) | Method and device for calibrating zoom pan-tilt camera and storage medium | |
| CN114782646A (en) | House model modeling method and device, electronic equipment and readable storage medium | |
| CN110111364A (en) | Method for testing motion, device, electronic equipment and storage medium | |
| US20210156697A1 (en) | Method and device for image processing and mobile apparatus | |
| CN111462194B (en) | Training method, device and storage medium of object tracking model | |
| CN113112553A (en) | Parameter calibration method and device for binocular camera, electronic equipment and storage medium | |
| CN111353945B (en) | Fisheye image correction method, device and storage medium | |
| US11166005B2 (en) | Three-dimensional information acquisition system using pitching practice, and method for calculating camera parameters | |
| CN111460972B (en) | Object tracking method, device and storage medium | |
| CN113177988B (en) | Spherical screen camera and laser calibration method, device, equipment and storage medium | |
| CN111223139B (en) | Target positioning method and terminal equipment | |
| CN115842960A (en) | Method for adjusting angle of device lens, terminal device and storage medium | |
| CN114640800B (en) | Camera arrangement method and system | |
| CN117252914A (en) | Training method and device of depth estimation network, electronic equipment and storage medium | |
| CN113920083A (en) | Image-based size measurement method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |