CN114862960A - Multi-camera calibrated image ground leveling method and device, electronic equipment and medium - Google Patents

Abstract

The present application discloses an image ground leveling method, device, electronic device and medium for multi-camera calibration. The method includes: establishing a mapping relationship between image coordinates and world coordinates in a multi-camera calibration process; acquiring human body image information and performing face key point detection in the human body image information to obtain human eye key point information and head posture auxiliary key points information; 3D key points in the world coordinate system are calculated based on the mapping relationship, human eye key point information and head posture auxiliary key point information; the human eye or monocular key points with the same posture are obtained to construct a fitting plane; the world coordinate system is calculated The rotation matrix from the xoy plane below to the fitting plane, the rotation matrix multiplied by the mapping relationship is the mapping relationship between the image coordinates and the leveled world coordinates. The method does not use additional calibration objects and manual markers, and performs automatic ground leveling after camera calibration through the detection of human key points, which is simple and easy to implement, and is suitable for various operating environments.

Description

Image ground leveling method, device, electronic device and medium for multi-camera calibration

technical field

Embodiments of the present disclosure relate to the technical field of visual measurement, and in particular, to a multi-camera calibration image ground leveling method, device, electronic device, and medium.

Background technique

With the development of computer vision technology, its application is more and more extensive. Generally speaking, in computer vision applications, it is necessary to determine the relationship between the three-dimensional geometric position of a point in space and its corresponding point in the image, that is, to establish a mapping relationship between image coordinates and spatial position coordinates. The solution process of this mapping relationship is camera calibration. In the application of computer vision, the calibration of camera parameters is a very critical link, and the accuracy of the calibration results and the stability of the algorithm directly affect the accuracy of the results generated by computer vision. Therefore, doing a good job in camera calibration is the premise of doing follow-up work. At present, the traditional camera calibration methods include direct linear transformation calibration method, Zhang Zhengyou calibration method and so on. The traditional camera calibration needs to use a calibration object with a known size. By establishing the correspondence between the known coordinate points on the calibration object and the image points, the internal and external parameters of the camera model are obtained by algorithm.

The traditional camera calibration method can establish the mapping relationship between pixel coordinates and world coordinates, but because the world coordinate system is selected in different ways, the world coordinates and the spatial position coordinates may not be consistent. A typical problem is that if the xoy plane of the world coordinate system is not parallel to the ground, the z coordinate calculated in computer vision applications is not height information. In some practical applications, such as judging whether the standing and sitting posture is straight in human posture estimation, it will lead to inaccurate judgment. Specifically, when using a three-dimensional calibration object, such as a three-dimensional calibration frame, generally the plane where the bottom four calibration spheres are located is used as the xoy plane of the world coordinate system. If the manufacturing process is slightly inaccurate, or the frame is deformed due to long use, it will lead to deviations in the xoy plane. When using a plane calibration object, the world coordinate system is based on the calibration plate, so it is necessary to calibrate the calibration plate parallel or perpendicular to the ground for the first time. It is difficult to guarantee the verticality. Larger tilt angles in the image are also difficult to detect.

In addition, to obtain the plane where the ground is located, it is necessary to find at least 3 positional coordinate points on the ground. In the process of camera calibration, it is necessary to manually mark or detect at least 3 feature points in the multi-camera image. This first requires additional calibration objects on the ground for manual annotation or algorithm detection, and the feature points in the multi-camera image are unique and easy to match. Manual labeling in this way is time-consuming and labor-intensive with large errors. Algorithmic detection is greatly affected by the environment and perspective, and requires special detection and matching algorithms.

SUMMARY OF THE INVENTION

In view of this, the embodiments of the present disclosure provide an image ground leveling method, apparatus, electronic device and medium for multi-camera calibration, so as to solve the error comparison in the prior art image ground leveling method using additional calibration objects for multi-camera calibration. It is a technical problem that is large, has low precision, is easily affected by factors such as environment and viewing angle, and requires specialized detection and matching algorithms.

In view of the above-mentioned defects in the prior art, the inventive concept of the technical solution of the present application is that in the absence of an external calibration object, the human body itself can be used as the calibration object. Because the ground leveling does not require the feature point size to be known, but only needs to be parallel to the ground. The key points of the human body are unique, and the algorithm is mature and easy to detect. It is only necessary to design the rules to select the key points parallel to the ground. With the development of deep learning technology, the detection technology of human key points has become more and more mature. For some key points, such as face key points, the detection accuracy is not inferior to that of humans. The difficulty of detecting different key points of the human body is different. The detection of key points such as waist and legs is obviously more difficult than the detection of face key points, and such key points themselves have a wide area, which is very difficult in multi-camera. It is difficult to determine that the same feature point is detected, and it is difficult to ensure the detection accuracy. Therefore, the inventors selected the human eye as the feature point. Because the human eye area is small and has obvious features, it is easy to detect and has high detection accuracy. When a person stands upright, the height of the two eyes from the ground is basically the same; considering that the posture of the person's head may change, the head posture is more consistent in multiple detections, and the two eyes or one eye are used as feature points. Points fit a plane, which can be approximately considered parallel to the ground. After that, the rotation matrix from the xoy plane to the fitting plane can be obtained to transform the world coordinate system.

To achieve the above object, a first aspect of the embodiments of the present disclosure provides an image ground leveling method for multi-camera calibration, the method comprising: establishing a mapping relationship between image coordinates and world coordinates during the multi-camera calibration process;

Obtaining human body image information and performing face key point detection in the human body image information to obtain human eye key point information and head posture auxiliary key point information;

3D key points in the world coordinate system are obtained by calculating based on the mapping relationship, the human eye key point information and the head posture auxiliary key point information;

Obtain the key points of both eyes or single eyes of the human body with the same posture to construct a fitting plane;

Calculate the rotation matrix from the xoy plane in the world coordinate system to the fitting plane, and multiply the rotation matrix by the mapping relationship to obtain the mapping relationship between the image coordinates and the leveled world coordinates.

A second aspect of the embodiments of the present disclosure provides a multi-camera calibration image ground leveling device, including:

an establishment unit, configured to establish a mapping relationship between image coordinates and world coordinates during the multi-camera calibration process;

a first acquiring unit, configured to acquire human body image information and perform face key point detection in the human body image information to obtain human eye key point information and head posture auxiliary key point information;

a first computing unit, configured to calculate and obtain 3D key points in the world coordinate system based on the mapping relationship, the human eye key point information and the head posture auxiliary key point information;

The second acquisition unit is configured to acquire key points of both eyes or monocular of the human body with the same posture to construct a fitting plane;

The second calculation unit is configured to calculate a rotation matrix from the xoy plane in the world coordinate system to the fitting plane, where the rotation matrix is multiplied by the mapping relationship to obtain a mapping relationship between image coordinates and leveled world coordinates .

In a third aspect of the embodiments of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the above method when the processor executes the computer program.

In a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the foregoing method are implemented.

One of the above-mentioned embodiments of the present disclosure has the following beneficial effects: the image ground leveling method for multi-camera calibration of the present application is applicable to various camera calibration methods. The world coordinate system is consistent with the ground, which is convenient for subsequent evaluations. No additional calibration objects are used, no manual marking is required, and automatic ground leveling after camera calibration is performed through the detection of human key points, which is simple and easy to operate, and is suitable for various operating environments.

Description of drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent when taken in conjunction with the accompanying drawings and with reference to the following detailed description. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and elements are not necessarily drawn to scale.

1 is a schematic flowchart of some embodiments of a multi-camera calibration image ground leveling method according to the present disclosure;

2 is a schematic structural diagram of some embodiments of a multi-camera calibration image ground leveling method device according to the present disclosure;

3 is a schematic structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure.

Detailed ways

In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and techniques are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present disclosure with unnecessary detail.

The following will describe in detail a method for leveling the ground of an image calibrated by multiple cameras according to an embodiment of the present disclosure with reference to the accompanying drawings.

FIG. 1 is a schematic flowchart of an image ground leveling method for multi-camera calibration provided by an embodiment of the present disclosure. As shown in Figure 1, the multi-camera calibration image ground leveling method includes the following steps:

Step S101, establishing a mapping relationship between image coordinates and world coordinates in the multi-camera calibration process;

Step S102, acquiring human body image information and performing face key point detection in the human body image information to obtain human eye key point information and head posture auxiliary key point information.

In some embodiments, the acquiring of the human body image information is synchronizing and timed photographing by multiple cameras within a preset interval time period. The human body image information includes image information of the human body at different positions and in multiple different time periods; the number of the multiple different time periods is at least three. The head pose auxiliary key point information includes auxiliary key point information for head pose estimation in all preset time periods, such as nose key points.

Step S103: Calculate and obtain 3D key points in the world coordinate system based on the mapping relationship, the human eye key point information, and the head posture auxiliary key point information.

Step S104, obtaining the key points of both eyes or one eye of the human body with the same posture to construct a fitting plane;

In some embodiments, the posture consistency includes time points in which the directions of the plane normal vectors constructed by the human eye key points and the auxiliary key points are relatively consistent within all preset time periods.

Step S105: Calculate a rotation matrix from the xoy plane in the world coordinate system to the fitting plane, and multiply the rotation matrix by the mapping relationship to obtain a mapping relationship between image coordinates and leveled world coordinates.

As an example, in some embodiments, multi-camera calibration is performed (the number of cameras=N), and a mapping relationship M: 2D ₁ . . . 2D _N →3D is established from image coordinates to world coordinates. The multi-cameras are set to take pictures synchronously at intervals, and take pictures It _,n of people standing upright in multiple different positions, where t is the time and n is the camera number. Perform face key point detection in the image _It,n to obtain human eye key points and auxiliary key points for head pose estimation, such as LEYE_2D _t,n , REYE_2D _t,n and NOSE_2D _t,n . According to the camera calibration mapping relationship M and 2D _{t, 1} ... 2D _{t, N} , 3D key points in the world coordinate system are obtained, such as LEYE_3D _t , REYE_3D _t and NOSE_3D _t . The head pose is estimated according to the facial key points, and HEADPOSE _t is obtained, and k ones with the same pose are taken from all the moments, k>=3, denoted as t1...tk. Take binocular or monocular key points with the same posture, such as LEYE_3D _t1 ... LEYE_3D _tk , and fit the plane LEYE_SURFACE. Calculate the rotation matrix R from the xoy plane to the eye fitting plane LEYE_SURFACE in the original world coordinate system, then R⊙M is the mapping relationship between the image coordinates and the leveled world coordinates.

All the above-mentioned optional technical solutions can be combined arbitrarily to form optional embodiments of the present application, which will not be repeated here.

The following are the apparatus embodiments of the present disclosure, which can be used to execute the method embodiments of the present disclosure. For details not disclosed in the apparatus embodiments of the present disclosure, please refer to the method embodiments of the present disclosure.

FIG. 2 is a schematic diagram of a multi-camera calibration image ground leveling device provided by an embodiment of the present disclosure. As shown in FIG. 2 , the multi-camera calibration image ground leveling device includes: a establishing unit 201 , a first obtaining unit 202 , a first calculating unit 203 , a second obtaining unit 204 and a second calculating unit 205 . The establishment unit 201 is configured to establish a mapping relationship between image coordinates and world coordinates in the multi-camera calibration process; the first acquisition unit 202 is configured to obtain human body image information and perform facial key point detection in the human body image information Obtaining human eye key point information and head posture auxiliary key point information; the first calculation unit 203 is configured to calculate based on the mapping relationship, the human eye key point information and the head posture auxiliary key point information Obtain the 3D key points under the world coordinate system; the second acquisition unit 204 is configured to obtain the key points of both eyes or monocular of the human body with the same posture to construct a fitting plane; the second calculation unit 205 is configured to calculate the world coordinate system. The rotation matrix of the xoy plane to the fitting plane, the rotation matrix multiplied by the mapping relationship is the mapping relationship between the image coordinates and the leveled world coordinates. It should be understood that the size of the sequence numbers of the steps in the above embodiments does not mean the sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present disclosure.

FIG. 3 is a schematic diagram of a computer device 3 provided by an embodiment of the present disclosure. As shown in FIG. 3 , the computer device 3 of this embodiment includes a processor 301 , a memory 302 , and a computer program 303 stored in the memory 302 and executable on the processor 301 . When the processor 301 executes the computer program 303, the steps in each of the foregoing method embodiments are implemented. Alternatively, when the processor 301 executes the computer program 303, the functions of the modules/units in the foregoing device embodiments are implemented.

Illustratively, the computer program 303 may be divided into one or more modules/units, which are stored in the memory 302 and executed by the processor 301 to complete the present disclosure. One or more modules/units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 303 in the computer device 3 .

The computer device 3 may be a desktop computer, a notebook computer, a palmtop computer, and a cloud server and other computer devices. Computer device 3 may include, but is not limited to, processor 301 and memory 302 . Those skilled in the art can understand that FIG. 3 is only an example of the computer device 3, and does not constitute a limitation to the computer device 3, and may include more or less components than the one shown, or combine some components, or different components For example, a computer device may also include input and output devices, network access devices, buses, and the like.

The processor 301 may be a central processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processors, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field-available processors Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 302 may be an internal storage unit of the computer device 3 , eg, a hard disk or a memory of the computer device 3 . The memory 302 can also be an external storage device of the computer device 3, for example, a pluggable hard disk, a smart memory card (Smart Media Card, SMC), a Secure Digital (Secure Digital, SD) card, a flash memory card ( Flash Card), etc. Further, the memory 302 may also include both an internal storage unit of the computer device 3 and an external storage device. The memory 302 is used to store computer programs and other programs and data required by the computer device. Storage 302 may also be used to temporarily store data that has been or will be exported.

Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. Module completion means dividing the internal structure of the device into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment may be integrated in one processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit, and the above-mentioned integrated units may adopt hardware. It can also be realized in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing from each other, and are not used to limit the protection scope of the present application. For the specific working processes of the units and modules in the above-mentioned system, reference may be made to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In the foregoing embodiments, the description of each embodiment has its own emphasis. For parts that are not described or described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this disclosure.

In the embodiments provided in the present disclosure, it should be understood that the disclosed apparatus/computer device and method may be implemented in other manners. For example, the apparatus/computer equipment embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. Multiple units or components may be Incorporation may either be integrated into another system, or some features may be omitted, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

Units described as separate components may or may not be physically separated, and components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated modules/units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer-readable storage medium. Based on this understanding, the present disclosure realizes all or part of the processes in the methods of the above embodiments, and can also be completed by instructing relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program is processed when the When the device is executed, the steps of the foregoing method embodiments may be implemented. A computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form, and the like. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, removable hard disk, magnetic disk, optical disk, computer memory, Read-Only Memory (ROM), random access memory Memory (Random Access Memory, RAM), electric carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the content contained in computer-readable media may be modified as appropriate in accordance with the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not be Including electrical carrier signals and telecommunication signals.

The above embodiments are only used to illustrate the technical solutions of the present disclosure, but not to limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The recorded technical solutions are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the present disclosure. within the scope of protection.

Claims (8)

1. an image ground leveling method of multi-camera calibration, is characterized in that, comprises: Establish the mapping relationship between image coordinates and world coordinates in the multi-camera calibration process; Obtaining human body image information and performing face key point detection in the human body image information to obtain human eye key point information and head posture auxiliary key point information; 3D key points in the world coordinate system are obtained by calculating based on the mapping relationship, the human eye key point information and the head posture auxiliary key point information; Obtain the key points of both eyes or single eyes of the human body with the same posture to construct a fitting plane; Calculate the rotation matrix from the xoy plane in the world coordinate system to the fitting plane, and multiply the rotation matrix by the mapping relationship to obtain the mapping relationship between the image coordinates and the leveled world coordinates. 2 . The image ground leveling method for multi-camera calibration according to claim 1 , wherein the acquisition of the human body image information is synchronously taking pictures at a predetermined interval by a plurality of cameras. 3 . 3. The multi-camera calibration image ground leveling method according to claim 1, wherein the human body image information comprises image information of the human body at different positions and multiple different time periods. 4 . The image ground leveling method for multi-camera calibration according to claim 1 , wherein the head posture auxiliary key point information includes the head posture auxiliary key points that are consistent with the human body posture in all preset time periods. 5 . information. 5 . The image ground leveling method for multi-camera calibration according to claim 3 , wherein the number of the plurality of different time periods is at least three. 6 . 6. A multi-camera calibration image ground leveling device, characterized in that, comprising: an establishment unit, configured to establish a mapping relationship between image coordinates and world coordinates during the multi-camera calibration process; a first acquiring unit, configured to acquire human body image information and perform face key point detection in the human body image information to obtain human eye key point information and head posture auxiliary key point information; a first computing unit, configured to calculate and obtain 3D key points in the world coordinate system based on the mapping relationship, the human eye key point information and the head posture auxiliary key point information; The second acquisition unit is configured to acquire key points of both eyes or monocular of the human body with the same posture to construct a fitting plane; The second calculation unit is configured to calculate a rotation matrix from the xoy plane in the world coordinate system to the fitting plane, where the rotation matrix is multiplied by the mapping relationship to obtain a mapping relationship between image coordinates and leveled world coordinates . 7. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the computer program as claimed in the claims The steps of any one of 1 to 6 of the method. 8. A computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented. step.

Images